Neo4j LLM Knowledge Graph Builder: How to Create Knowledge Graphs for RAG

On this page

To create a knowledge graph for RAG without writing extraction code, you can use the Neo4j LLM Knowledge Graph Builder. You connect it to a Neo4j database, give it a document or a web page, and an LLM extracts the entities and relationships into a graph. You can then chat with that graph using GraphRAG, vector search or Text2Cypher, right in the same app.

This starter guide walks through the whole process. Whether you're new to graph databases or already build RAG systems, you'll learn how to:

• Set up the Neo4j LLM Knowledge Graph Builder
• Create your first knowledge graph for RAG
• Visualize the created graph and explore it

We start with what knowledge graphs are and how they compare to vector-based RAG, then move on to the hands-on part.

What are knowledge graphs in the context of RAG?

Knowledge graphs are structured representations of information that show the relationships between entities. They consist of nodes (concepts or entities) and edges (the relationships between them). In short, they organize data roughly the way people connect pieces of information in their heads.

In Retrieval Augmented Generation (RAG), knowledge graphs are an advanced way to store and retrieve information. RAG gives a language model relevant external knowledge while it generates an answer. Knowledge graphs add to traditional, purely vector-based RAG in three ways. They structure data so it is easy to query and navigate. They represent the relationships between pieces of information explicitly. And they show how different facts or concepts relate to each other, which gives the model broader context.

Simple knowledge graph

Knowledge graphs promise to improve RAG systems in several ways:

• Accuracy: structured, relevant information helps RAG systems generate more accurate and contextually appropriate responses.
• Reasoning: the relational structure lets RAG systems connect separate pieces of information and handle more complex questions.
• Flexibility: graphs are easy to update and extend, so a RAG system can take in new information over time.
• Explainability: explicit relationships make it easier to trace how a RAG system arrived at an answer.
• Fewer hallucinations: grounding the language model in a structured knowledge base can reduce false or inconsistent answers.

There are two types of knowledge graphs to distinguish:

• Lexical graphs connect the structural elements of your data. Think of document -> chapter -> section -> paragraph -> chunk.
• Semantic graphs capture the "meaning" of your documents. For example, they might connect "Paris" with "France" and "Eiffel Tower", or information about "onboarding" with "employee" and "training".

Most of the time you want both in your RAG system, because you need to find relationships in document structure as well as in content meaning.

Lexical and semantic graph

Vector-based RAG vs. knowledge graph-based RAG

Vector-based and knowledge graph-based RAG are not rivals. They complement each other, and you might use both in the same application.

Still, comparing them side by side helps to understand knowledge graphs, since most readers know the vector-based approach better.

1. Data representation

   • Vector-based RAG: Represents documents or chunks of text as high-dimensional vectors in an embedding space.
   • Knowledge graph RAG: Represents information as interconnected entities and relationships.

2. Information retrieval

   • Vector-based RAG: Uses similarity measures (like cosine similarity) to find relevant vectors.
   • Knowledge graph RAG: Uses graph traversal algorithms to find relevant information through relationships.

3. Context understanding

   • Vector-based RAG: Implicit context based on vector proximity.
   • Knowledge graph RAG: Explicit context through defined relationships between entities.

4. Handling of structured data

   • Vector-based RAG: Less effective with highly structured data.
   • Knowledge graph RAG: Excels at representing and querying structured data. Can represent complex relationships and link structured and unstructured data.

5. Scalability

   • Vector-based RAG: Can handle large amounts of unstructured text efficiently.
   • Knowledge graph RAG: Efficient for querying complex relationships, but can be more resource-intensive to build and maintain.

6. Updating information

   • Vector-based RAG: Adding new chunks of text is straightforward. Just add the new embeddings.
   • Knowledge graph RAG: Can be updated incrementally by adding new nodes and edges. Changing the schema is more complex.

7. Explainability

   • Vector-based RAG: Less transparent, as relationships are implicit in vector space.
   • Knowledge graph RAG: More transparent, with explicit relationships that can be traced.

8. Handling ambiguity

   • Vector-based RAG: May struggle to tell similar concepts apart.
   • Knowledge graph RAG: Can explicitly represent different meanings or contexts for similar terms.

9. Resource requirements

   • Vector-based RAG: Generally requires less upfront work to implement.
   • Knowledge graph RAG: Often requires more initial effort to build and structure the knowledge base.

10. Flexibility

    • Vector-based RAG: More flexible with unstructured or previously unseen text.
    • Knowledge graph RAG: More rigid, but strong in domains with well-defined relationships.

What is Neo4j?

Neo4j is the leading graph database management system, and it's open source. It stores, manages and queries highly connected data efficiently. Unlike relational databases, Neo4j uses a graph structure for semantic queries, which makes it a good fit for applications with complex relationships and interconnected data, such as knowledge graphs.

Main features of Neo4j

• Native graph storage: Neo4j stores data as nodes and relationships, which mirrors real-world connections more closely than tables do.
• Cypher: Neo4j's declarative graph query language for querying graph data efficiently.
• ACID compliance: transactions are atomic, consistent, isolated and durable, which protects data integrity.
• Scalability: Neo4j scales horizontally through its Causal Clustering architecture, for both reads and writes.
• Performance: Neo4j is optimized for traversing relationships, which makes it much faster than relational databases for some queries, especially those involving complex relationships.
• Flexibility: you can add new nodes, relationships and properties without breaking existing queries.
• Visualization: Neo4j has built-in tools for visualizing graph data. The Neo4j team also provides many good tools for working with LLMs.

What is the Neo4j LLM Knowledge Graph Builder?

The Neo4j LLM Knowledge Graph Builder is an application that turns unstructured text into a structured knowledge graph. It processes PDFs, documents, web pages and even YouTube video transcripts and stores the resulting graph in a Neo4j database.

Entity graph generated from a Neuralink article in the Graph Builder

The Knowledge Graph Builder's processing pipeline works in four steps.

1. Input processing: it ingests documents with LangChain Loaders, which cover PDFs, web pages and YouTube video transcripts, among other formats.
2. Text chunking: the content is split into chunks. These chunks become the base nodes of the graph, linked to their source documents and to each other.
3. Embedding generation: it computes an embedding for each chunk and stores it in the chunk node and in a vector index.
4. Entity and relationship extraction: an LLM such as OpenAI, Gemini or Llama3 extracts entities and relationships from the text, using modules like llm-graph-transformer or diffbot-graph-transformer.

Graph construction produces two main structures: a lexical graph of documents and chunks with embeddings, and an entity graph with the extracted entities and their relationships. For better connectivity, the Graph Builder also creates a k-nearest neighbors (kNN) graph by linking similar chunks with SIMILAR relationships.

For RAG folks, the most interesting part is that the Graph Builder supports several retrieval approaches:

• GraphRAG (the approach this post is about)
• Vector-based retrieval ("classical" RAG)
• Text2Cypher queries (a way to query the graph directly)

So you can combine these methods for potentially better results, and, even more useful, compare them against each other. To make such a comparison measurable, see our guide on evaluating RAG pipelines with RAGAS.

Setting up the Neo4j LLM Knowledge Graph Builder

The Neo4j LLM Knowledge Graph Builder needs a Neo4j database to work with, so we first need one.

There are two ways to get it:

1. Use the fully managed Neo4j AuraDB. This is the easiest way to get started.
2. Set up your own Neo4j database on your local machine or a server.

Setting up Neo4j AuraDB

To use AuraDB, go to Neo4j AuraDB and create an account. On first login, you'll see a username and password. There is also a "Download" button that downloads a file with all the required connection parameters.

Wait until your instance is created, and you're done.

A running free Neo4j AuraDB instance with its connection URI

Setting up a self-hosted Neo4j database

Running Neo4j with Docker is also simple.

1. Make sure you have Docker installed.

2. Run the following command (replace your_password with the password you want to use and /path/to/your/data with the path to your actual data location on your local host):

1docker run \
2    --restart always \
3    --publish=7474:7474 --publish=7687:7687 \
4    --env NEO4J_apoc_export_file_enabled=true \
5    --env NEO4J_apoc_import_file_enabled=true \
6    --env NEO4J_apoc_import_file_use__neo4j__config=true \
7    --env NEO4J_PLUGINS=\[\"apoc\"\] \
8    --env NEO4J_AUTH=neo4j/your_password \
9    --volume=/path/to/your/data:/data \
10    neo4j:5.21.0

Running the Neo4j LLM Knowledge Graph Builder

With the database running, we can start the Knowledge Graph Builder.

Again, there are two ways to do this:

1. Use the Neo4j LLM Knowledge Graph Builder web application
2. Use the provided Docker Compose file

Using the LLM Knowledge Graph Builder web application

Go to the Neo4j LLM Knowledge Graph Builder web app and enter your connection information.

If you used AuraDB, take the values from the credentials file you downloaded earlier.

Neo4j LLM Knowledge Graph Builder connection settings

Running the Neo4j LLM Knowledge Graph Builder locally with Docker Compose

Again, make sure Docker is installed.

1. Create a .env file in your project's root folder with the following variables:

   1
   2 OPENAI_API_KEY="your-openai-key"
   3 DIFFBOT_API_KEY="your-diffbot-key"
   

   Get your OpenAI key from the OpenAI Platform and your Diffbot key from the Diffbot application.

2. Run the following command:

   1 docker-compose up --build
   

See the Neo4j LLM Knowledge Graph Builder deployment documentation for more configuration options and instructions.

Creating your first knowledge graph for RAG

You should now have access to the web application.

Graph Builder start screen with data sources, file list and chat

Let's create a first knowledge graph to see how it all works.

1. Click "Web Sources" in the left menu bar.
2. Click "Website Link" (the third icon) and enter a website you want to get data from. For example, you could use our blog post about using LLMs to chat with BigQuery.
3. Click "Submit".

The first time you submit a file, you are asked to create a schema for your knowledge graph. Designing a schema is a topic for another post. If you leave the settings empty, the LLM creates the schema for you, which is actually not a bad idea.

Graph schema settings

After a moment, the Graph Builder shows the new file in the main area of the app. Select it with the checkbox on the left and click "Generate Graph". This takes a while, depending on the size of the document.

Once the graph is generated, click "Preview Graph". You should see a graph visualization of your document, similar to the one below.

graph visualization with LLM-built entities

Use the dropdown in the top right to switch between the graph types (lexical, entity, kNN) and explore the relationships.

Trying RAG with the new knowledge graph

The graph looks nice, but is it useful? Go back to the main area and use the chat area on the right-hand side.

Ask a question like "What is the system message in the llm api call?" This is a tricky question, because the answer sits in a code section of the blog post and might be hard to find.

As you can see below, the system finds this information in our blog post without trouble.

Chat with LLM-built knowledge graph

And there you have it: we built our first knowledge graph and used it in a small RAG example.

Where is the knowledge graph stored?

The Neo4j LLM Knowledge Graph Builder adds the graph data to the Neo4j database instance you provided during setup.

Conclusion

The Neo4j LLM Knowledge Graph Builder is a practical way to create knowledge graphs for RAG without writing extraction code. You connect it to a Neo4j database (AuraDB or a self-hosted Docker container with APOC), add a document or web page, and click "Generate Graph".

The Graph Builder creates a lexical graph of documents and chunks and an entity graph of the concepts in them, so it captures both structural and semantic relationships. In our example, even with no schema and minimal setup, the chat answered a question about a code snippet in a blog post correctly.

This guide is a starting point. Good next steps are designing your own schema instead of letting the LLM choose one, checking the quality of the extracted entities, and comparing GraphRAG, vector retrieval and Text2Cypher on your own documents. Since the graph lives in your Neo4j database, you can also query and update it directly with Cypher.

If you would rather not run a separate graph database, the open-source RAG engine RAGFlow can also extract a knowledge graph from your documents. A similar idea exists for relational data: NVIDIA Auto Ontology generates business terms and join paths from database metadata for a text-to-SQL agent.

Further reading

• BitNet: LLM Quantization at its Extreme
• Use LLMs to extract data from documents
• Using AI directly from your PostgreSQL database
• Advanced RAG: Recursive Retrieval with LlamaIndex
• RAG, embeddings and vector search with Google BigQuery

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

• Understanding the 14 systematic failure modes in multi-agent systems
• Evidence-based best practices for agent design
• Structured communication protocols and verification mechanisms

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.

Talk to Andreas about your projectFollow on LinkedIn