Validating Ongoing Workflows in Brim

Validating a chart abstraction pipeline for an ongoing clinical workflow is not a one-time job; you need tools to help your pipelines stay accurate as data and models change. Brim gives you two easy ways to check your workflows over time:

1. Monitor Distributions of Key Variables

• What it is: See how your variable outputs are spread across categories or values.

  Why it matters: Shifts in these distributions can signal silent errors or data drift.

  How to do it:

  1. There are instructions for accessing this view for a variable here, and a walkthrough below.

If you stay familiar with this data over time, you should be able to quickly spot shifts that might indicate drift. Example: If “Unknown” responses rise from 5% to 20%, that could be a sign something changed upstream.

2. Label and Check Agreement

• What it is: Review a small sample of records, label them manually, and compare to pipeline output.

  Why it matters: Provides ground truth and catches errors distribution checks may miss.

  How to do it:

  1. Select a random batch of patients. Industry standard is 5% of the total, but even 10–20 reviewed records can surface systematic issues.
  2. Review the patients, in Brim or separately.
  3. Create a validation dataset, either manually or by exporting a detailed export from Brim.
  4. Upload as a Validation dataset and track agreement.

If the agreement is lower than your standard for this pipeline, we recommend digging deeper into the data. You may need to revise your data upload or iterate on your Brim variables to improve agreement.

Tip: Not all shifts are bad — sometimes distributions change because your underlying population changes. Use thresholds and clinical context to decide when to act.

You can read more about Best Practices on our blog.