The problem
Clinical researchers spend a great deal of time on work that is not research. Before a dataset can answer anything, someone has to find out what is actually in it: which columns mean what, which rows are unusable, which values are impossible, and which are merely surprising.
Then, separately, they have to read. The literature around any given question is larger than the time available to read it.
ClinicalPilot AI was my Master of Computer Applications major project, and it targets both halves of that.
Constraints
- The output has to be checkable. In a clinical context, a confident answer with no provenance is not useful, it is a liability. Anything the system says has to point at where it came from.
- The inputs are genuinely messy. Real clinical datasets carry inconsistent coding, missing values that mean several different things, and units that change halfway through a column.
- The user is a researcher, not an engineer. No query language, and no schema knowledge assumed.
What I built
Validation before anything else
The pipeline refuses to analyse a dataset it has not first profiled. Automated validation and cleaning runs on ingest: type inference with the inferences surfaced rather than hidden, range and plausibility checks, missing-value profiling that distinguishes “not recorded” from “recorded as absent”, and a report the researcher reads before any analysis is offered.
Pandas does the work. The important part is not the library, it is that the cleaning is a visible, reviewable step rather than something that happens silently on the way to a chart.
Which is also why the tests are built from broken data rather than clean data: columns with mixed types, dates that do not parse, values outside any plausible range, and rows that are absent in one file and present in another. A validation layer is only worth having if you know what it does when it is given the worst case.
Speaking the formats the data actually arrives in
Clinical records do not arrive as a tidy CSV. They arrive as FHIR R4 resources or C-CDA documents, which are standards with enough room in them that two conforming systems can still disagree about where a value lives. The ingest layer reads both, maps them onto one internal model, and records what it could not place rather than dropping it quietly.
That mapping is the least glamorous part of the project and the part that makes the rest possible. Everything above it works on one shape.
Retrieval, not recall
Paper summarisation and dataset question answering both run through a retrieval pipeline rather than asking a model to answer from memory. Embeddings live in PostgreSQL through pgvector, so the search sits in the same database as the data it is answering about. The model sees retrieved passages and is asked to answer from them, and the answer carries citations back to the source.
This is the design decision the whole project rests on. A model answering from its weights produces fluent text that cannot be verified. A model answering from retrieved context produces text a researcher can check in the time it takes to click a link. For clinical work, only the second one is acceptable.
Building it changed how I think about model output generally. It is the same category of thing as an incoming API request: untrusted input that gets validated before it is allowed to matter.
An agent with three tools, and a way to tell whether it is any good
The question box is a streaming agent rather than a single call. It has three tools: search the literature, query the dataset, and pull a patient record. It picks between them, and the answer streams back as it works, so a researcher sees progress instead of a spinner.
Giving a model tools is easy. Knowing whether a change made it better is the hard part, and the reason for the twenty-question evaluation harness: a fixed set of questions with known good answers, scored on every change. Without it, tuning a prompt or a chunking strategy is guesswork that feels like progress. With it, a change that sounds better and scores worse gets reverted.
The suite runs to 159 tests across ingest, retrieval and the agent.
Cohort discovery without a query language
Interactive dashboards for exploring cohorts, so that narrowing a population is a matter of adjusting filters and watching the count move, rather than writing SQL and hoping the joins are right.
Stack and why
FastAPI and Pydantic on the backend, for the same reason as always: typed boundaries and generated documentation. PostgreSQL for the structured data, with pgvector so the embeddings live beside it rather than in a second system to keep in step. React and TypeScript for the dashboards. Pandas for the profiling and cleaning work.
The code is public, so none of the above has to be taken on trust.
What I would do differently
I would build the evaluation harness first rather than last. It exists now, and it changed how the work felt: before it, I judged retrieval quality by reading outputs, which finds obvious failures and misses slow drift entirely. Every tuning decision made before that point was taken with less evidence than it deserved, and a few of them turned out to be wrong when there was finally a number attached.