Table of Contents

<aside> 🌲 This is a living document!

This document will continue to evolve as we undergo more analysis and determine additional key questions to ask!

</aside>

Motivation

It’s critical to understand the data before jumping into a solution. This guide outlines some threads you might want to pull during your data analysis and exploration.

Key Questions

At PressW, we have a set of key questions that we ask at the beginning of the data exploration phase so that we can create a reliable system to deploy for our clients.

What are the main “buckets” of questions the client expects to answer?

Are you able to manually answer those questions?

<aside> ✅ Yes! We’re in a good place to have the LLM start to answer them as well!

</aside>

<aside> 🚨 Not Manually, no This means that the LLM won’t be able to either. We’ll need to augment the LLM somehow to fit your use case.

</aside>

<aside> 👉🏼 Important! As you’re doing this exercise, note down what documents you’re using to answer these questions. These documents will become the basis of how we create intuition in the LLM prompt or help by pre-filtering queries in your document retrieval step.

</aside>

Do we need to pull in supplementary data sources to accurately answer questions?

We can augment LLMs with data sources and tools in order to improve accuracy to our questions. Below are two examples of ways we can add data into an LLM!

<aside> 💽 Custom Datasets Custom datasets (client documents, databases, and even internet pages) is a great way to improve LLM accuracy within custom domains of knowledge

</aside>

<aside> 🌐 The Internet to the Rescue If the problem statement makes sense and security allows for it, we can incorporate a SERP (Search Engine Results Page) retriever to the system. This allows us to make standard internet queries through search engines to augment the LLM’s knowledge!

</aside>

Do we have all of the data that the client promised?

Often clients don’t know exactly what data they have. This is the stage where you should surface any issues or inconsistencies with the data they thought they have vs. what they actually have.

How quickly do we need to surface responses to the user?

The response to this question will affect decisions we make about the needs of how the backend is setup.

Ex: Number of Prompt Steps

Adding to the length of prompt steps directly correlates with how long the system will take to return back to the user and can affect the time-to-first token for the application. We need to be mindful as application developers to keep this at a reasonable level.

Ex: Maximum Tokens in Prompt

The number of tokens within the prompt affect the latency as there are more tokens that the LLM has to process.

Ex: Number of Retrieval Sources

Each source of knowledge that the system needs to pull from to augment the LLM’s knowledge increases the time to respond to the user.

What document formats will we be adding into the system, if any?

How will this affect our chunking strategy?

We might need to use any third party libraries to load specific document types.

Do we need to process images?

This might require us to upgrade to a multimodal model, or preprocess documents through an OCR pipeline.

How are the documents related to one another?

Document relationships are an important understanding when beginning a project. If there are inter-document relationships, we may need to consider a hierarchical retrieval approach where we retrieve documents not only via embeddings, but by graphing out which documents will help us understand others.

Do the documents require filtering on any specific criteria?

We need to determine if the system needs to perform any forms of access rights management, data segmentation by user, etc.