Table of Contents
Choosing chunking settings will be largely specific to the dataset and thus will require experimentation. It will be important to employ a standardized method of evaluating the changes we make to the chunking hyperparameters, see here:
You should first think a bit about the shape of the data before just slapping 1000 characters down as your chunk size and moving on (don’t worry, we’ve all done it).
Basically, ask these questions:
For a couple examples of each, look for the context you as a human would need to answer the question. Ideally, the full context should be captured in the chunk. This will mean that the retriever will be able provide the LLM with everything it needs as the first result.
In the case of the hard questions, it might be that multiple chunks of info are needed to create the whole picture. Regardless, the same principle applies. Break the harder questions down into some number of simpler questions, then apply the same principle as you did before. Important note: don’t break it down into any more questions than the “top k” parameter of your retriever. If you have some really hard questions that need to be decomposed to such a degree, increase your retriever’s “top k.”
Take caution though, don’t just crank the chunk size up. Each chunk should encompass no more than a hand full of, preferably related, concepts. This will ensure the information doesn’t get washed out when it’s embedded and the retriever can do its job.
The purpose of overlap is to minimize the chance that you accidentally split a concept in half and make it unable to be linked to its neighbors in the Once the chunks are embedded and put into the database, they lose all sense of order within the document.
Since in language, sentence are naturally a decent model of a thought, a good place to start is looking at sentence length. In the average case, a sentence will be split down the middle if we don’t use any overlap. We can assume a uniform distribution to model the question: “at what word of the chunk’s last sentence will the split be?” This way, we can try to ensure that in most cases the neighboring chunks share a complete thought that links them.
The logical next step would be choosing the average sentence length as the overlap to mitigate that problem. However, we’d then still have the problem in 50% of the cases. If we’re getting extremely analytical, we’d want to plot a distribution of all of the sentence sizes in the document and choose a length that captures the bulk of the distribution without including too many outliers. If that’s a little heavy handed, try starting with the mean sentence length + 2 * the standard deviation and you’ll likely capture around 97.5% of the data.