The problem
Clinical researchers spend a great deal of time on work that is not research. Before a dataset can answer anything, someone has to find out what is actually in it: which columns mean what, which rows are unusable, which values are impossible, and which are merely surprising.
Then, separately, they have to read. The literature around any given question is larger than the time available to read it.
ClinicalPilot AI was my Master of Computer Applications major project, and it targets both halves of that.
Constraints
- The output has to be checkable. In a clinical context, a confident answer with no provenance is not useful, it is a liability. Anything the system says has to point at where it came from.
- The inputs are genuinely messy. Real clinical datasets carry inconsistent coding, missing values that mean several different things, and units that change halfway through a column.
- The user is a researcher, not an engineer. No query language, and no schema knowledge assumed.
What I built
Validation before anything else
The pipeline refuses to analyse a dataset it has not first profiled. Automated validation and cleaning runs on ingest: type inference with the inferences surfaced rather than hidden, range and plausibility checks, missing-value profiling that distinguishes “not recorded” from “recorded as absent”, and a report the researcher reads before any analysis is offered.
Pandas does the work. The important part is not the library, it is that the cleaning is a visible, reviewable step rather than something that happens silently on the way to a chart.
Retrieval, not recall
Paper summarisation and dataset question answering both run through a retrieval pipeline rather than asking a model to answer from memory. The model sees retrieved passages and is asked to answer from them, and the answer carries citations back to the source.
This is the design decision the whole project rests on. A model answering from its weights produces fluent text that cannot be verified. A model answering from retrieved context produces text a researcher can check in the time it takes to click a link. For clinical work, only the second one is acceptable.
Building it changed how I think about model output generally. It is the same category of thing as an incoming API request: untrusted input that gets validated before it is allowed to matter.
Cohort discovery without a query language
Interactive dashboards for exploring cohorts, so that narrowing a population is a matter of adjusting filters and watching the count move, rather than writing SQL and hoping the joins are right.
Stack and why
FastAPI and Pydantic on the backend, for the same reason as always: typed boundaries and generated documentation. PostgreSQL for the structured data. React for the dashboards. Pandas for the profiling and cleaning work.
What I would do differently
I would build the evaluation harness first. I assessed retrieval quality by reading outputs, which is fine for finding obvious failures and useless for detecting slow drift. A held-out set of questions with known good answers, scored automatically on every change, would have told me whether a tweak to the chunking strategy actually helped or only felt like it did.