Optimizing Variable Instructions

When defining a variable, the instructions are the primary communication mechanism between you, Brim, the Large Language Models that draft labels, and the humans that review labels. The Optimizer helps you improve those instructions without hand-tuning prompts yourself. It analyzes the disagreement between your current Training labels in validation and existing Brim labels, applies our Variable Best Practices to propose targeted changes, then tests whether these changes improve agreement on a sample of patients. You can run it iteratively to improve agreement as you change labels.

We recommend running the Optimizer after you import or create your variables, and again after you've generated and reviewed some patient labels or uploaded training data, so it can learn from your edits.

Starting the Optimizer

1. Go to "Project Setup" and find the variable you want to optimize in the Variables section.
2. In the Actions column, click "Optimize" for that variable. This opens the "New Optimization" page, which shows your current instructions for the variable.
3. Edit the current instructions if you'd like before you begin. You can also tell the Optimizer how you want the instructions improved. This step is optional.
4. Select whether you'd like Brim to Generate Improvement Suggestions. This will give you some ideas for how to improve the instructions, which you can choose to apply or reject.
5. Click "Generate Improvement Suggestions" or "Start Optimization" to get started.

You can also access the Optimizer in a few other locations in the workflow:

1. While Editing a variable, you can click the "Optimize Instruction" link above instructions.
2. While reviewing labels in label review, if you click the question mark icon to view reasoning, and then the wrench icon, you'll be brought directly to the Optimizer.
3. If you have uploaded or labelled validation data for the variable, you can click the "Optimize" button next to the variable at the bottom of the Validation Overview page.

Improvement Suggestions

If you select "Generate Improvement Suggestions", the Optimizer will start by creating a list of ideas to improve your variable. These can include rule and edge case clarifications, helpful definitions to add, and ways to simplify your variable.

Once you receive the list of improvement suggestions, you can uncheck any that you wish to be ignored in the optimization itself. Then click "Start Optimization" to apply the changes you've selected.

Reviewing the proposed changes

Once you start it, the Optimizer reviews your feedback, the existing training examples and guidance from the Variable Scorecard, then suggests updated instructions and aggregation instructions.

When it finishes, the page shows the proposed changes for your review.

From here you can:

• Run a Optimization sample generation to test the changes against a sample set of patients. This is only available if you have validation data uploaded.
  • Once you run a sample generation, you'll be able to see the change in agreement across the sample in training and holdout data. At this point, you'll be give the choices to Apply or Discard as described below.
• Apply changes. This saves the changes to your variable.
• Discard the changes and try another optimization.

Validating with Training Data and Holdout data

If you accept and run a Optimization generation, Brim generates labels for a sample of patients in training and holdout and calculates the agreement for you. Your validation results show:

• The number of patients in the sample.
• Your training agreement percentages before and after the optimization (for the sample).
• Your holdout agreement percentage before and after the optimization (for the sample).

These changes are only tested during this generation, so nothing is applied to the variable yet.

If your agreement improves, click "Apply" to save the optimized instructions to your variable. If you don't want to save the changes, click "Discard".

Reviewing Optimization History

Each optimization is saved in your optimization history, where you can revisit the proposed changes and easily access Variable History for Applied Changes.