Late Chunking: Improving RAG Performance with Context-Aware Embeddings

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published 7 min read
Late Chunking: Improving RAG Performance with Context-Aware Embeddings
On this page

Late chunking is a chunking method for Retrieval Augmented Generation (RAG) that embeds a whole document first and splits it into chunks afterwards. A long-context embedding model creates one embedding per token for the full text, and only then are those token embeddings averaged into chunk embeddings. Every chunk embedding therefore carries context from the rest of the document.

In our example below, a chunk that starts with "It was originally developed..." scores 0.81 similarity with the search term "TensorFlow" using late chunking, against 0.54 with traditional chunking. This guide explains the chunking problem, how late chunking works, and how to implement it in Python.

The problem it solves is the usual chunk size trade-off. Chunks that are too small lose context. Chunks that are too large make it harder for the embedding model to capture a precise meaning. Either way, retrieval accuracy suffers.

NOTE: Late chunking was first introduced by the people at jina ai. The code samples in this post are derived from their work, and we use their jina-embeddings-v3 model for our examples. So shout-out to Jina AI!

A quick introduction to Retrieval Augmented Generation

Retrieval Augmented Generation (RAG) lets an LLM answer questions using your own documents. For a detailed introduction, read our detailed post here.

RAG has two main phases:

  1. The indexing phase

  2. The retrieval phase

During the indexing phase, the documents that form the knowledge base are transformed into embeddings, which are high-dimensional vectors.

Embeddings are numeric representations of texts that roughly capture their semantic meaning. The important property: texts with similar meaning have similar embeddings. "My car" and "My automotive" have similar embeddings, whereas "My car" and "My cow" have rather different ones.

This property makes semantic search possible. Users (or an AI) search in natural language and find documents with a similar meaning, not just matching keywords. That's the core of RAG.

In the retrieval phase, a user asks a question that an LLM should answer. Using the embeddings from the indexing phase, we search for documents that are semantically similar to the user's question and therefore most likely to answer it.

Finally, we send the user's question and the retrieved documents to an LLM, which uses them to write a tailored answer.

A typical, simplified RAG pipeline is shown below:

Typical RAG pipeline

The problem with chunking in RAG applications

Chunking is where RAG pipelines often lose information. To see why, start with why we chunk at all. There are three reasons:

  1. LLMs don't have unlimited context windows, so we need to limit the amount of data we send. In an ideal world, we could send the user's question together with the whole knowledge base and wait for the answer. Context limits make that impossible.

  2. LLM context costs money. Even as context windows grow, every token we send is billed, so we want to keep the input small.

  3. LLMs have trouble finding the right information in large amounts of data. If we pre-filter the relevant information, the answer gets better.

How does document chunking work? There are various methods, but they all boil down to dividing documents into chunks of roughly x words. We can split by word count or at semantically meaningful boundaries, like the end of a paragraph or chapter.

Either way, splitting a document cuts connections between its parts. Take this document as an example:

"TensorFlow is one of the most popular open-source machine learning frameworks.

It was originally developed by researchers and engineers from Google Brain.

The framework provides a flexible ecosystem of tools, libraries, and community resources. These resources help developers build and deploy ML-powered applications efficiently.

The framework excels at numerical computation and large-scale machine learning. Its architecture allows for easy deployment of computation across various platforms."

Let's assume we chunk the document after each paragraph. Here the paragraphs are mostly single sentences, but the problem exists no matter where you cut.

Now we ask: What framework was developed by Google Brain? We'd most likely find the second paragraph, but that chunk no longer says which framework Google Brain created. "It" refers to "TensorFlow", and paragraph-based chunking loses that connection.

How does late chunking work?

Late chunking keeps that connection by changing the order of operations. We'll explain it in two parts: a summary and a more detailed walkthrough.

Summary: Instead of first splitting a document into chunks and then embedding each chunk, late chunking applies the embedding model to the full document and creates token-level embeddings (one embedding per token). These token-level embeddings are then combined into chunk-level embeddings (one embedding per chunk). Because the token embeddings were computed over the whole text, each of them carries meaning from the whole text, not just from its own token or chunk. When we combine them into chunk embeddings, each chunk knows about itself and about the surrounding chunks.

(The chart below was kindly taken from jina ai)

Late chunking: token embeddings from a long document pooled into chunk embeddings

Detailed description: Here is what happens step by step.

First, the text (the whole document, if it fits the model's context) is tokenized, meaning it is divided into individual tokens:

1tokens = tokenizer(input_text, return_tensors='pt')

Second, these tokens are passed through the model:

1model_output = model(**tokens)

The token embeddings are the model's last hidden state, the last layer before the output. So we can access them directly:

1token_embeddings = model_output.last_hidden_state[0]

Put simply, after passing the tokens through the model, we read the individual token embeddings from last_hidden_state. For the example text above, this gives us something like the following. The representation is simplified: it shows one embedding array per word, while in reality there is one per token.

1[
2 [0.1, 0.2, ...], # embedding for "TensorFlow"
3 [0.3, 0.4, ...], # embedding for "is"
4 [0.2, 0.1, ...], # embedding for "one"
5 [0.4, 0.3, ...], # embedding for "of"
6 [0.1, 0.1, ...], # embedding for "the"
7 [0.2, 0.3, ...], # embedding for "most"
8 [0.3, 0.2, ...], # embedding for "popular"
9 ...
10]

All that's left is to divide (chunk) this array of embeddings. For example, we can split it according to the text the tokens represent. If we chunk by sentence, we take the embeddings of all tokens in the first sentence and average them. That average is the embedding for this chunk. Repeat for every sentence. The result is a list of chunk embeddings, each derived from context-aware token embeddings instead of from a chunk processed in isolation.

This is the main difference from traditional chunking: each token's embedding was created with the full context available. When "It" was embedded, the model already knew it referred to "TensorFlow", because it processed the whole text at once.

Why are late chunking embeddings context-aware?

Late chunking embeddings are more context-aware than traditionally chunked embeddings because of the attention mechanism in the transformer architecture.

When the model processes "It", it doesn't look at that token in isolation. Through its self-attention layers, each token can "attend to" (look at) all other tokens in the input text.

Here's a simplified explanation of how self-attention works:

  1. For each token, the model calculates attention scores with every other token in the sequence. These scores represent how much attention should be paid to each other token when creating the final representation.

  2. For example, when processing the word "It", the attention scores for the other words are:

    1TensorFlow: 0.7 (high attention score because it's the reference)
    2is: 0.1
    3one: 0.1
    4...
    5framework: 0.4
    6...
  3. The model learns these attention patterns during training. It learns that pronouns like "It" should pay strong attention to the nouns they refer to.

In code, this happens inside the transformer model when it processes the full input:

1inputs = tokenizer(input_text, return_tensors='pt')
2model_output = model(**inputs)

Compare this with traditional chunking:

Traditional:

1chunk1 = "TensorFlow is the most popular framework..."
2chunk2 = "It was originally developed..."
3# When processing chunk2, "It" has no access to "TensorFlow"

Late chunking:

1full_text = "TensorFlow is the most popular framework [...]. It was originally developed [...]"
2# When processing, "It" can attend to "TensorFlow" through self-attention
3model_output = model(**inputs)
4# Only after attention is applied do we average the token embeddings into chunks
5embeddings = late_chunking(model_output, [span_annotations])[0]

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms

Hands-on: how to implement late chunking in Python

Let's build an end-to-end example with the sample text from above.

1pip install transformers torch einops
1from transformers import AutoModel
2from transformers import AutoTokenizer
3
4tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v3', trust_remote_code=True)
5model = AutoModel.from_pretrained('jinaai/jina-embeddings-v3', trust_remote_code=True)
6
7input_text = "TensorFlow is one of the most popular open-source machine learning frameworks. It was originally developed by researchers and engineers from Google Brain. The framework provides a flexible ecosystem of tools, libraries, and community resources. These resources help developers build and deploy ML-powered applications efficiently. The framework excels at numerical computation and large-scale machine learning. Its architecture allows for easy deployment of computation across various platforms."

Next, we split the text into sentence chunks. The traditional method needs these chunks directly. Late chunking needs them to find the token positions where each chunk starts and ends, because we split the array of embeddings only after embedding the whole text.

1def chunk_by_sentences(input_text: str, tokenizer: callable):
2 inputs = tokenizer(input_text, return_tensors='pt', return_offsets_mapping=True)
3 punctuation_mark_id = tokenizer.convert_tokens_to_ids('.')
4 sep_id = tokenizer.convert_tokens_to_ids('[SEP]')
5 token_offsets = inputs['offset_mapping'][0]
6 token_ids = inputs['input_ids'][0]
7 chunk_positions = [
8 (i, int(start + 1))
9 for i, (token_id, (start, end)) in enumerate(zip(token_ids, token_offsets))
10 if token_id == punctuation_mark_id
11 and (
12 token_offsets[i + 1][0] - token_offsets[i][1] > 0
13 or token_ids[i + 1] == sep_id
14 )
15 ]
16 chunks = [
17 input_text[x[1] : y[1]]
18 for x, y in zip([(1, 0)] + chunk_positions[:-1], chunk_positions)
19 ]
20 # The span_annotations are the start and end positions of our
21 # text-chunks - in terms of tokens. So the first and last position
22 # of the tokens of our individual chunks
23 span_annotations = [
24 (x[0], y[0]) for (x, y) in zip([(1, 0)] + chunk_positions[:-1], chunk_positions)
25 ]
26 return chunks, span_annotations
27chunks, span_annotations = chunk_by_sentences(input_text, tokenizer)

Last but not least, the late chunking function itself:

1def late_chunking(
2 model_output, span_annotation: list, max_length=None
3):
4 token_embeddings = model_output.last_hidden_state
5 outputs = []
6 for embeddings, annotations in zip(token_embeddings, span_annotation):
7 if (
8 max_length is not None
9 ): # remove annotations which go beyond the max-length of the model
10 annotations = [
11 (start, min(end, max_length - 1))
12 for (start, end) in annotations
13 if start < (max_length - 1)
14 ]
15 pooled_embeddings = [
16 embeddings[start:end].sum(dim=0) / (end - start)
17 for start, end in annotations
18 if (end - start) >= 1
19 ]
20 pooled_embeddings = [
21 embedding.detach().cpu().numpy() for embedding in pooled_embeddings
22 ]
23 outputs.append(pooled_embeddings)
24
25 return outputs

That's it. Now we can create embeddings with both methods:

1# Traditional chunking method
2embeddings_traditional_chunking = model.encode(chunks)
3
4# Late chunking methods
5inputs = tokenizer(input_text, return_tensors='pt')
6model_output = model(**inputs)
7embeddings = late_chunking(model_output, [span_annotations])[0]

To compare the two methods, let's run a small 'benchmark' that prints the similarity between the word "TensorFlow" and each chunk:

1import numpy as np
2
3# Method to calculate the cosine similarity
4cos_sim = lambda x, y: np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))
5
6# Embed our search term
7embedding = model.encode('TensorFlow')
8
9for chunk, new_embedding, trad_embeddings in zip(chunks, embeddings, embeddings_traditional_chunking):
10 print(chunk)
11 print(f' - late chunking:', cos_sim(embedding, new_embedding))
12 print(f' - traditional:', cos_sim(embedding, trad_embeddings))

These are the results in our case:

1TensorFlow is one of the most popular open-source machine learning frameworks.
2 - late chunking: 0.8232424
3 - traditional: 0.88007504
4 It was originally developed by researchers and engineers from Google Brain.
5 - late chunking: 0.80999833
6 - traditional: 0.535794
7 The framework provides a flexible ecosystem of tools, libraries, and community resources.
8 - late chunking: 0.7612619
9 - traditional: 0.44915122
10 These resources help developers build and deploy ML-powered applications efficiently.
11 - late chunking: 0.77815264
12 - traditional: 0.48452184
13 The framework excels at numerical computation and large-scale machine learning.
14 - late chunking: 0.8006218
15 - traditional: 0.6145765

Late chunking vs. traditional chunking: the results

The similarity scores (higher is better) are impressive. Late chunking scores a bit lower on the first chunk (0.82 vs. 0.88), but much higher on every other chunk. The second chunk, "It was originally developed...", goes from 0.54 to 0.81.

The small drop on the first chunk is expected. The traditional method already does very well when the search term appears directly in the chunk.

With traditional chunking, the other chunks lose their semantic connection to the first chunk, the one that names TensorFlow. Late chunking keeps that connection, which, depending on your use case, can boost your RAG search performance noticeably. Better chunk embeddings also combine well with other retrieval improvements, such as reranking with ColBERT or hypothetical document embeddings (HyDE). If you would rather configure chunking than code it, RAGFlow lets you pick a chunking method per knowledge base.

Further reading

Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.