Late Chunking: Improving RAG Performance with Context-Aware Embeddings

On this page
Late chunking is a chunking method for Retrieval Augmented Generation (RAG) that embeds a whole document first and splits it into chunks afterwards. A long-context embedding model creates one embedding per token for the full text, and only then are those token embeddings averaged into chunk embeddings. Every chunk embedding therefore carries context from the rest of the document.
In our example below, a chunk that starts with "It was originally developed..." scores 0.81 similarity with the search term "TensorFlow" using late chunking, against 0.54 with traditional chunking. This guide explains the chunking problem, how late chunking works, and how to implement it in Python.
The problem it solves is the usual chunk size trade-off. Chunks that are too small lose context. Chunks that are too large make it harder for the embedding model to capture a precise meaning. Either way, retrieval accuracy suffers.
NOTE: Late chunking was first introduced by the people at jina ai. The code samples in this post are derived from their work, and we use their jina-embeddings-v3 model for our examples. So shout-out to Jina AI!
A quick introduction to Retrieval Augmented Generation
Retrieval Augmented Generation (RAG) lets an LLM answer questions using your own documents. For a detailed introduction, read our detailed post here.
RAG has two main phases:
-
The indexing phase
-
The retrieval phase
During the indexing phase, the documents that form the knowledge base are transformed into embeddings, which are high-dimensional vectors.
Embeddings are numeric representations of texts that roughly capture their semantic meaning. The important property: texts with similar meaning have similar embeddings. "My car" and "My automotive" have similar embeddings, whereas "My car" and "My cow" have rather different ones.
This property makes semantic search possible. Users (or an AI) search in natural language and find documents with a similar meaning, not just matching keywords. That's the core of RAG.
In the retrieval phase, a user asks a question that an LLM should answer. Using the embeddings from the indexing phase, we search for documents that are semantically similar to the user's question and therefore most likely to answer it.
Finally, we send the user's question and the retrieved documents to an LLM, which uses them to write a tailored answer.
A typical, simplified RAG pipeline is shown below:
Typical RAG pipeline
The problem with chunking in RAG applications
Chunking is where RAG pipelines often lose information. To see why, start with why we chunk at all. There are three reasons:
-
LLMs don't have unlimited context windows, so we need to limit the amount of data we send. In an ideal world, we could send the user's question together with the whole knowledge base and wait for the answer. Context limits make that impossible.
-
LLM context costs money. Even as context windows grow, every token we send is billed, so we want to keep the input small.
-
LLMs have trouble finding the right information in large amounts of data. If we pre-filter the relevant information, the answer gets better.
How does document chunking work? There are various methods, but they all boil down to dividing documents into chunks of roughly x words. We can split by word count or at semantically meaningful boundaries, like the end of a paragraph or chapter.
Either way, splitting a document cuts connections between its parts. Take this document as an example:
"TensorFlow is one of the most popular open-source machine learning frameworks.
It was originally developed by researchers and engineers from Google Brain.
The framework provides a flexible ecosystem of tools, libraries, and community resources. These resources help developers build and deploy ML-powered applications efficiently.
The framework excels at numerical computation and large-scale machine learning. Its architecture allows for easy deployment of computation across various platforms."
Let's assume we chunk the document after each paragraph. Here the paragraphs are mostly single sentences, but the problem exists no matter where you cut.
Now we ask: What framework was developed by Google Brain? We'd most likely find the second paragraph, but that chunk no
longer says which framework Google Brain created. "It" refers to
"TensorFlow", and paragraph-based chunking loses that connection.
How does late chunking work?
Late chunking keeps that connection by changing the order of operations. We'll explain it in two parts: a summary and a more detailed walkthrough.
Summary: Instead of first splitting a document into chunks and then embedding each chunk, late chunking applies the embedding model to the full document and creates token-level embeddings (one embedding per token). These token-level embeddings are then combined into chunk-level embeddings (one embedding per chunk). Because the token embeddings were computed over the whole text, each of them carries meaning from the whole text, not just from its own token or chunk. When we combine them into chunk embeddings, each chunk knows about itself and about the surrounding chunks.
(The chart below was kindly taken from jina ai)
Late chunking: token embeddings from a long document pooled into chunk embeddings
Detailed description: Here is what happens step by step.
First, the text (the whole document, if it fits the model's context) is tokenized, meaning it is divided into individual tokens:
Second, these tokens are passed through the model:
The token embeddings are the model's last hidden state, the last layer before the output. So we can access them directly:
Put simply, after passing the tokens through the model, we read the
individual token embeddings from last_hidden_state.
For the example text above, this gives us something like the following. The
representation is simplified: it shows one embedding array per word, while in
reality there is one per token.
All that's left is to divide (chunk) this array of embeddings. For example, we can split it according to the text the tokens represent. If we chunk by sentence, we take the embeddings of all tokens in the first sentence and average them. That average is the embedding for this chunk. Repeat for every sentence. The result is a list of chunk embeddings, each derived from context-aware token embeddings instead of from a chunk processed in isolation.
This is the main difference from traditional chunking: each token's embedding was created with the full context available. When "It" was embedded, the model already knew it referred to "TensorFlow", because it processed the whole text at once.
Why are late chunking embeddings context-aware?
Late chunking embeddings are more context-aware than traditionally chunked embeddings because of the attention mechanism in the transformer architecture.
When the model processes "It", it doesn't look at that token in isolation. Through its self-attention layers, each token can "attend to" (look at) all other tokens in the input text.
Here's a simplified explanation of how self-attention works:
-
For each token, the model calculates attention scores with every other token in the sequence. These scores represent how much attention should be paid to each other token when creating the final representation.
-
For example, when processing the word "It", the attention scores for the other words are:
-
The model learns these attention patterns during training. It learns that pronouns like "It" should pay strong attention to the nouns they refer to.
In code, this happens inside the transformer model when it processes the full input:
Compare this with traditional chunking:
Traditional:
Late chunking:
Interested in building high-quality AI agent systems?
We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:
- Understanding the 14 systematic failure modes in multi-agent systems
- Evidence-based best practices for agent design
- Structured communication protocols and verification mechanisms
Hands-on: how to implement late chunking in Python
Let's build an end-to-end example with the sample text from above.
Next, we split the text into sentence chunks. The traditional method needs these chunks directly. Late chunking needs them to find the token positions where each chunk starts and ends, because we split the array of embeddings only after embedding the whole text.
Last but not least, the late chunking function itself:
That's it. Now we can create embeddings with both methods:
To compare the two methods, let's run a small 'benchmark' that prints the similarity between the word "TensorFlow" and each chunk:
These are the results in our case:
Late chunking vs. traditional chunking: the results
The similarity scores (higher is better) are impressive. Late chunking scores a bit lower on the first chunk (0.82 vs. 0.88), but much higher on every other chunk. The second chunk, "It was originally developed...", goes from 0.54 to 0.81.
The small drop on the first chunk is expected. The traditional method already does very well when the search term appears directly in the chunk.
With traditional chunking, the other chunks lose their semantic connection to the first chunk, the one that names TensorFlow. Late chunking keeps that connection, which, depending on your use case, can boost your RAG search performance noticeably. Better chunk embeddings also combine well with other retrieval improvements, such as reranking with ColBERT or hypothetical document embeddings (HyDE). If you would rather configure chunking than code it, RAGFlow lets you pick a chunking method per knowledge base.
Further reading
- How to test your RAG pipeline?
- Advanced RAG: Recursive Retrieval with llamaindex
- DSPy: Automatic prompt tuning and creating
- Embeddings and vector search with Google BigQuery
- Shorter embeddings with OpenAI's v3 embedding models
- Building a RAG system with Azure AI Search
- Free and open source ChatGPT alternative: LibreChat



