Using Brim's Validation Tool
Brim’s Validation Tool makes it easy to evaluate your abstraction results against a comparison dataset—helping you assess accuracy, track progress, and guard against drift as you iterate on your variable set.
What It Does
The Validation Tool compares each generated data point to a corresponding value in your uploaded comparison dataset. Brim automatically calculates agreement for every generation, so you can:
- Instantly understand how your current abstractions perform against a trusted reference set.
- Track performance over time as you update your variable definitions.
- Catch regressions or data drift before exporting.
How to Use the Validation Tool
1. Navigate to the Validation Tool
You can access the tool in two places:
- Project Setup (from the left navigation bar)
- Settings (dropdown in the top-right corner of your project)
Look for the section labeled “Validation Data”.
2. Upload Your Comparison Dataset
- Click "Choose File" and select a file to upload.
The Brim detailed export file is already in the correct format for upload.
You can also create your own comparison dataset file. Each row of the CSV should be a value, and it should have the following fields:
| Field Name | Field description | Notes |
| Name | The variable name | This must exactly match the variable name you'd like it compared to in Brim. |
| Scope | The scope of the variable: One Label Per Patient One Label Per Note Many Labels Per Note |
This must match the variable scope in Brim. Brim will ignore any Many Labels Per Note variables because matching values is imprecise. |
| Patient_id | Alphanumeric patient ID. | This must exactly match the patient ID you'd like it compared to in Brim. |
| Document_id | Alphanumeric document ID OR Blank for One Label Per Patient values |
This must exactly match the document ID you'd like it compared to in Brim. |
| Value | This must exactly match the value in Brim in order to have Agreement = TRUE. |
- Click "Upload". Brim will validate the structure and immediately compare the uploaded data to any generated results.
- If you don't have generated results yet, go to Label Generation > Start New Generation to generate values.
Checking the "Include only reviewed results" checkbox will ignore any rows where the "Reviewed" column does not contain "TRUE". If your document does not include a "Reviewed" column, checking this box will exclude all rows from the comparison dataset.
3. Review Validation Results
Once uploaded, Brim automatically calculates agreement scores between each generation and your comparison dataset.
To view results:
- Go to Settings > Validation Data.
- On the right, you'll see a count of "Matching Variables".
- Variables can only be matched if they have exactly matching name and scope, and their scope is One Label Per Patient or One Label Per Note
- In the middle, you'll see a count of "Values Compared."
- Values can only be compared if they are for Matching Variables AND
- They have exactly matching patient IDs and Document IDs for values in Brim.
- On the left, you'll see an Agreement Percentage.
- This represents the % of Values Compared that have an exact match between Brim and the comparison dataset.
Investigate Specific Values
If you click "See all Compared Values", you can see a list of all Values Compared, along with the Comparison Value, the Brim Value, and the evidence used in Brim's determination. This helps you understand where agreement is strongest and weakest.
Best Practices
- Start small: Try validation on a handful of patients before scaling up.
- Iterate: Use the tool to refine unclear instructions or edge cases.
- Protect against drift: Run a quick validation on every new batch of documents.
FAQ
Q: Can I replace my comparison dataset?
Yes. Just upload a new CSV and it will replace the previous one.
Q: Can I export the validation results?
Yes. Doing a Detailed Export from the Export Data screen also exports the validation value and agreement.