hospitalresearch

what it is

in clemson's watt ai program, i work with a research team on a system that lets hospital staff ask plain-english questions across large sets of safety reports. it looks for recurring patterns and cites the exact records behind each answer.

1. ask
2. search
3. answer with citations
4. evaluate

why it matters

safety reports hold real lessons, but nobody has time to read them all. and in a hospital, an answer you can't trace back to its records isn't one you should act on.

what i built

my part is the evaluation: how we find out whether the system deserves that trust.

• relevance labels: a way for people to grade every report any setup retrieved for a question, so scores come from human judgment.
• retrieval metrics: hit rate, precision, recall and ndcg, plus a paired measure of whether rewriting the question actually helps.
• citation checks: a check that every record an answer cites is one the model actually saw.
• cost tracing: time, model calls and tokens for each stage of every query.

my favorite part

a report nobody has graded yet doesn't count as irrelevant. it counts as unknown. so instead of one number that looks final, the score comes back as a range until the labels are in.

where it stands

tested

the metric code and citation check pass their tests on synthetic data.

not measured yet

human grading hasn’t happened, so there are no real relevance scores. whether each claim is supported by its records hasn’t been evaluated either.

next

agreeing on a set of questions with my research lead, grading the results, and measuring faithfulness.

built with

• python
• fastapi
• chromadb
• bm25
• hybrid retrieval