Table of Contents
In order to expedite development, satisfy client expectations, and meet project goals without introducing timeline creep, it’s critical that any systems that we deploy can be benchmarked in a reliable and replicable way. This is especially true given that the majority of our projects involve Large Language Models, which are notoriously difficult to evaluate. This framework will discuss best practices around testing as well as outline some real testing scenarios.
In order to know where we’re headed, and what we’re testing for, we leverage the project definition and deliverables to build our success criteria. We follow the below best practices when building a list of criteria.
<aside> ✅ Create measurable criteria Criteria should be numeric whenever possible. This keep the ship pointed in the right direction and avoid the common pitfall of only optimizing over the same few examples (and then wondering why the model generalization isn’t improving)
</aside>
<aside> ✅ Create automatable criteria Criteria should, when possible, not require a human in the loop to determine the validity of each model output. This is often not possible with LLMs given the wide input and output space, but can be helpful to include as part of the pipeline
</aside>
<aside> ✅ Reference the deliverables document Make sure the criteria we’re proposing measure the full breadth of the value we’ve agreed to deliver to the client
</aside>
<aside> ✅ Align with the client Make sure the client is bought in, and understands the importance of each of the values we are measuring. The client should understand what each value means and how performance in each category will impact the final product.
</aside>
<aside> ⛔ Don’t promise specific targets We want to avoid promising any specific numeric target. Especially in the case of LLMs, but also with ML in general. We don’t want to set up the client to be let down. What’s important is that the numbers are improving, not that we hit some arbitrary number. All AI problems are different, and it’s often not possible to know what the “upper bound” of performance will be until all options are exhausted.
</aside>
<aside> ⛔ Don’t choose too many criteria It’s OK to have more criteria that you’re evaluating against as a developer, but have a limited set of the most important ones that you plan to communicate to the client on a regular basis
</aside>
<aside> ✅ Create an evaluation timeline Figure out, based on the project timeline (or better yet concurrently with development of project timeline) when a) the initial testing harness will be set up and usable b) how often we expect to have new results for the clients. A good rule of thumb is trying to present new metrics at least every 2 weeks while modeling improvement is ongoing.
</aside>
<aside> ✅ Tie evaluation to deployment Whatever evaluation is automatable should be run as part of the deployment process. Any evaluation that isn’t automatable should be at least partially automated to the extent that the model is run and the data is ready for manual review.
</aside>
<aside> ✅ Keep record of progress Each time a deployment is run, an artifact of the evaluation results should be saved to a uniformly accessible location. These artifacts can be used to track progress over time, and build visuals.
</aside>
<aside> ✅ Test frequently! You should be running the automated evaluations when any material changes are made to the code
This applies to the manual portion of evaluation too in a slightly different way. Identify the “target portion” of the test suite you are trying to improve and manually evaluate the outputs of just that portion as you develop. Once you’re satisfied with the improvements there, manually evaluate the broader set to ensure no regression before release.
</aside>
<aside> ⛔ Don’t manually evaluate the full test suite every time
<aside> ✅ Create visuals of the results To the extent that it’s possible, put in a little extra time to create visuals communicating the progress. This can be as simple as charting accuracy over time, or more complex like analyzing different error cases. These visuals should be automatically generated whenever possible.
</aside>
<aside> ✅ Attribute improvements to specific efforts “This week we tried x,y, and z. Y ended up improving ____ behaviour, and resulted in a 5% increase to ____ criteria”
</aside>