Optimizing Variable Instructions

When defining a variable, the instructions are the primary communication mechanism between you, Brim, the Large Language Models that draft labels, and the humans that review labels. The Optimizer helps you improve those instructions without hand-tuning prompts yourself. It analyzes the disagreement between your current Training labels in validation and existing Brim labels, applies our Variable Best Practices to propose targeted changes, then tests whether these changes improve agreement on a sample of patients. You can run it iteratively to improve agreement as you change labels.

We recommend running the Optimizer after you import or create your variables, and again after you've generated and reviewed some patient labels or uploaded training data, so it can learn from your edits.

Starting the Optimizer

1. Go to "Project Setup" and find the variable you want to optimize in the Variables section.
2. In the Actions column, click "Optimize" for that variable. This opens the "New Optimization" page, which shows your current instructions for the variable.
3. Edit the current instructions if you'd like before you begin. You can also tell the Optimizer how you want the instructions improved. This step is optional.
4. Click "Start Optimization".

You can also access the Optimizer in a few other locations in the workflow:

1. While Editing a variable, you can click the "Optimize Instruction" link above instructions.
2. While reviewing labels in label review, if you click the question mark icon to view reasoning, and then the wrench icon, you'll be brought directly to the Optimizer.
3. If you have uploaded or labelled validation data for the variable, you can click the "Optimize" button next to the variable at the bottom of the Validation Overview page.

Reviewing the proposed changes

Once you start it, the Optimizer reviews your existing training examples and scorecard feedback, then suggests updated instructions and aggregation instructions.

When it finishes, the page shows your current instructions alongside the proposed instructions so you can compare them.

From here you can:

• Run a Optimization sample generation to test the changes against a sample set of patients. This is only available if you have validation data uploaded.
  • Once you run a sample generation, you'll be able to see the change in agreement across the sample in training and holdout data. At this point, you'll be give the choices to Apply or Discard as described below.
• Apply changes. This saves the changes to your variable.
• Discard the changes and try another optimization.

Validating with Training Data and Holdout data

If you accept and run a Optimization generation, Brim generates labels for the training data of patients and calculates the agreement for you. Your validation results show:

• The number of patients in the sample.
• Your training agreement percentages before and after the optimization (for the sample).
• Your holdout agreement percentage before and after the optimization (for the sample).

These changes are only tested during this generation, so nothing is applied to the variable yet.

If your agreement improves, click "Apply" to save the optimized instructions to your variable. If you don't want to save the changes, click "Discard".

Reviewing Optimization History

Each optimization is saved in your optimization history, where you can revisit the proposed changes and easily access Variable History for Applied Changes.