Pre-Production Validation with Brim

When working with AI, making an answer is easy, but validating it and scaling it reliably is harder. Brim's Validation Tool is designed to help you do this for any project, starting pre-production with designing your abstraction, iterating to meet your metrics, supporting evaluation in preparation for production, and enabling ongoing monitoring for clinical workflows.

This article is designed to help you use Brim's Validation Tool from iteration through Production.

Core Principles

1. Measure what matters.

Make sure your evaluation lines up with the real-world priorities of your workflow. This starts with prioritizing the most important variables in the validation. For example, if you are identifying patients for a clinical trial, you might want to start with agreement for the top level variable (clinical trial eligibility), and work from there.

This also means considering the types of errors that are most consequential. For example, if you are identifying high-risk patients for rescheduling, recall (catching every possible high risk patient) might matter more than precision (avoiding false positives).

2. Understand your ground truth.

No dataset is perfect. If you are building your comparison dataset yourself, consider building your comparison dataset carefully: have two people abstract the same records, resolve disagreements with an expert, and keep definitions crystal clear. You can even do this in Brim, which helps you more efficiently find evidence in the notes.

If you can, get a measure of the inter-rater agreement; this represents the level of clarity between human reviewers, and is a good goal to shoot for when doing AI-guided abstraction.

3. Allow “unknown.”

Sometimes the evidence just isn’t in the chart. When Brim doesn't have evidence, it will give a default value you specify, usually "No evidence." This is meant to prevent guessing and improve reliability.

How to Validate with Brim

There are two main applications of the Validation Tool pre-production for Brim:

1. Create a Validation Dataset in Brim. Brim can be used to create a validation dataset quickly supporting downstream workflows.
2. Use a Validation Dataset while Building Variables. You can use a validation dataset while iterating on variables to calculate agreement, understand which variables need more tuning, and inform optimization of variables.

1. Create a Validation Dataset in Brim

Goal: Establish a baseline quickly.

Once you know the variables you want to abstract, you can create a Validation Dataset in Brim that you can use during iteration.

1. Upload data and create variables.
2. Generate for a small batch. We recommend 10-20 patients.
3. Upload a Training Dataset OR
4. Manually label Training Examples using Label Review. On the first run, the Brim generated values likely won't be perfect. We recommend supplementing the evidence Brim pulls with reviewing the charts and:
   1. Accepting, Editing, or Removing each Brim value in Label Review.
   2. Adding manual labels where appropriate, including the evidence in the text that drove the conclusion.

2. Use your Validation Dataset to Improve Variables

Goal: Learn quickly and refine.

1. Upload a Holdout Dataset. This will allow you to measure agreement on a set of patients without those patients being directly used to improve variables.
2. Look at your Overall Agreement and Agreement by variable. In Settings > Validation Data, you'll be able to see the overall agreement and agreement by variable. This will can help you focus your iteration efforts.
3. Improve variables automatically with the Optimizer
   1. Start by clicking "Optimize" on variables that are lagging in agreement. The Variable Optimizer will apply Variable Best Practices and leverage training examples to try to improve agreement.
4. Improve variables manually.
   1. In "See all Comparison Values", you can filter by variable and sort by Agreement value to see which values aren't agreeing.
   2. If you can see patterns, try to address them in the variable prompt. Common error types include formatting mismatches, missed information, and ambiguous definitions.
   3. You can also define a Validation Match Rule for your variable if the values don't need to match validation values exactly. For example:
      1. "The values must be within 2"
      2. "The values must be on the same day"
      3. "The values must be within 30 minutes of eachother"
5. Stop iterating when performance is close to or better than human agreement for your most important variables.

Brim’s tools are designed to make your validation & evaluation process straightforward. Reach out if you have more suggestions for how we can make this easier!