Using Brim's Validation Tool
Brim’s Validation Tool makes it easy to evaluate your abstraction results against a comparison dataset—helping you assess accuracy, track progress, and guard against drift as you iterate on your variable set.
What It Does
The Validation Tool compares each generated data point to a corresponding value in your uploaded comparison dataset. Brim automatically calculates agreement for every generation, so you can:
- Instantly understand how your current abstractions perform against a trusted reference set.
- Track performance over time as you update your variable definitions.
- Catch regressions or data drift before exporting.
How to Use the Validation Tool
1. Navigate to the Validation Tool
You can access the tool in two places:
- Project Setup (from the left navigation bar)
- Settings (dropdown in the top-right corner of your project)
Look for the section labeled “Validation Data”.
2. Upload Your Comparison Dataset
- Click "Choose File" and select a file to upload.
The Brim detailed export file is already in the correct format for upload.
You can also create your own comparison dataset file. Each row of the CSV should be a value, and it should have the following fields:
| Field Name | Field description | Notes |
| Name | The variable name | This must exactly match the variable name you'd like it compared to in Brim. |
| Scope | The scope of the variable: One Label Per Patient One Label Per Note Many Labels Per Note Patient-Level Dependent Variable |
This must match the variable scope in Brim. Brim will ignore any Many Labels Per Note variables because matching values is imprecise. |
| Patient_id | Alphanumeric patient ID. | This must exactly match the patient ID you'd like it compared to in Brim. |
| Document_id | Alphanumeric document ID OR Blank for One Label Per Patient and Patient-Level Dependent Variable values |
This must exactly match the document ID you'd like it compared to in Brim. |
| Value | This must exactly match the value in Brim in order to have Agreement = TRUE. |
- Click "Upload". Brim will validate the structure and immediately compare the uploaded data to any generated results.
- If you don't have generated results yet, go to Label Generation > Start New Generation to generate values.
Checking the "Include only reviewed results" checkbox will ignore any rows where the "Reviewed" column does not contain "TRUE". If your document does not include a "Reviewed" column, checking this box will exclude all rows from the comparison dataset.
3. Review Validation Results
Once uploaded, Brim automatically calculates agreement scores between each generation and your comparison dataset.
To view results:
- Go to Settings > Validation Data.
- On the right, you'll see a count of "Matching Variables".
- Variables can only be matched if they have exactly matching name and scope, and their scope is One Label Per Patient or One Label Per Note
- In the middle, you'll see a count of "Values Compared."
- Values can only be compared if they are for Matching Variables AND
- They have exactly matching patient IDs and Document IDs for values in Brim.
- On the left, you'll see an Agreement Percentage.
- This represents the % of Values Compared that have an exact match between Brim and the comparison dataset.
4. Investigate Specific Values
If you click "See all Compared Values", you can see a list of all Values Compared, along with the Comparison Value, the Brim Value, and the evidence used in Brim's determination. This helps you understand where agreement is strongest and weakest.
Unmatched Values
Brim can't always match a comparison dataset value to a value in Brim. Understanding the details of why can help you tweak your Brim setup, comparison dataset, or both to get the results you want.
View upload details
After you upload validation data (either in Project Setup or the Validation Data section in Settings), you'll see a short summary of how many variables and values were matched to the generated values in Brim.
If you click "More info", you'll see a breakdown of the dataset by number of rows:
- Total rows: this is the number of rows in the comparison dataset, excluding the header.
- Accepted: this is the total number of rows in the comparison dataset that Brim could match to a Brim value.
- Skipped: this is the total number of rows in the comparison dataset that Brim skipped without comparing to a value in Brim. This is further broken down by reason:
- Duplicate: There are multiple duplicate rows in the comparison dataset. In this case, Brim accepts one row and skips the others.
- Empty Value: If a row has a blank value, Brim skips it.
- Not Reviewed: If you selected "Only load reviewed labels." when uploading your comparison dataset, Brim will skip any rows that don't have a "TRUE" value in a "Reviewed" column.
- Missing Required Column: The required columns are listed above; if a row is missing a required column, Brim will skip it.
- Missing Document ID: All variables with a One Label Per Note scope must have a corresponding document ID so that Brim knows which document to match them to. If they do not, they are skipped.
- Unsupported/Unknown Scope: This counts rows that are skipped because their scope is not listed as either One Label Per Patient or One Label Per Note
See details on unmatched values
In Settings > Validation Data > See all Compared Values, you'll see a tabbar at the top with "Matched" and "Unmatched" options.
If you select "Unmatched", you'll see a list of all of the values that were Skipped during upload, along with a value-specific reason why.
Best Practices
- Start small: Try validation on a handful of patients before scaling up.
- Iterate: Use the tool to refine unclear instructions or edge cases.
- Protect against drift: Run a quick validation on every new batch of documents.
FAQ
Q: Can I replace my comparison dataset?
Yes. Just upload a new CSV and it will replace the previous one.
Q: Can I export the validation results?
Yes. Doing a Detailed Export from the Export Data screen also exports the validation value and agreement.