Table of Contents

Data Exploration

It’s important to explore and analyze the data before committing to a solution. In this step, we can improve our understanding of the problem, and communicate any learnings to the client as early as possible.

LLM Data Exploration Jumpstart

Phase 1: Proof of Concept

It’s now time to use your learnings from the data exploration step to build a proof of concept for the theorized approach. This will ensure our method is viable before implementing a full production pipeline. Traditionally, this is a “quick and dirty” approach in a Jupyter Notebook where you’re targeting a few things:

Does the proposed approach produce coherent results for all of the question categories or tasks?

<aside> ✅ Yes! If we are able to produce coherent results, that is a great sign for the project! It means that we’re on the right track in our way of thinking about solving the problem.

</aside>

<aside> ➡️ Not Yet Do we think this can be fixed through improving the sophistication of the retrieval methods?

<aside> ✅ Yes! Then it’s likely alright to move to the implementation phase of the project

</aside>

<aside> ➡️ Not Yet We should now consider introducing additional data sources or fine tuning on industry specific terminology. This may result in a scope extension and should be communicated to all of the relevant parties of the project

</aside>

</aside>

Does the approach meet speed requirements?

<aside> ✅ Yes! If we are able to meet the speed requirements, we’re on track to begin implementation!

</aside>

<aside> ➡️ Not Yet We need to create a concrete plan for improving speed before we begin implementation that answers the following questions:

1. Can any prompts or actions be executed in parallel?
2. Is it possible to stream back responses?
3. Is it possible to improve the responsiveness of the UI to make the system seem faster? </aside>

Phase 2: Implementation

Now that we’ve decided to move forward with our approach refer, back to the LLM Development Bible for guides on various aspects of implementation.

Step-by-Step Guide to Custom Language Models

Phase 3: Evaluate and Improve

After the initial pipeline implementation, we’ll begin sharing results with the client and improving the model based on their feedback. However, in order to do this, we’ll need a means to test and evaluate the model in order to quantify our improvements over time.

LLM Test and Evaluation Best Practices

Next Phase

Sometimes, it’s not going to be possible to meet all of the client needs in the initial phase of work. We can always refer to the deliverables document, and point out our ability to include certain out of scope improvements in the next phase of work.