Introduction to RAGFlow: Open-Source RAG Engine with Deep Document Understanding

Retrieval Augmented Generation (RAG) has become one of the most promising and effective applications of LLMs. Despite many people claiming RAG is dead (mainly for social media engagement reasons), it is the application with the most value created, when talking about LLMs.

RAG, in short, is a technique which allows to provide LLMs with up-to-date and relevant data without needing to train or fine-tune a model. Therefore, it's the most important technique for most chatbots and internal knowledge base applications.

At it's core, RAG requires you to create a search index, based on vector embeddings. These embeddings approximately represent the meaning of a chunk of text. When a user asks a question, the RAG system can use this semantic representation to find the most relevant chunks of text in the index (for answering the users question). These chunks are then passed to the LLM, which generates an answer based on the retrieved information.

By far the most relevant step in this whole process is extracting (parsing) the source documents (which are often pdfs or Microsoft Office documents), creating the index and having a powerful search on top of this index. What sounds simple at first, is actually quite complex to get right.

While we as a AI consulting company obviously advocate for creating your own RAG systems when you have specific requirements, we also understand that this can be quite costly and time consuming. That's where today's article comes in: We want to introduce you to RAGFlow, an open-source RAG engine that provides truthful question-answering capabilities with well-founded citations from complex formatted data through deep document understanding. From what we can see, it's one of the most advanced RAG platforms available today, and has the best document extraction and indexing capabilities we have seen so far.

What is RAGFlow?

Ragflow markets itself as an open-source RAG engine with main focus on "deep document understanding". So it tries to have specific focus on the document parsing and indexing part of RAg. Arguably the most important parts of RAG.

In more detail, RAGFlow consists of the following components:

• A web application, providing a user interface for document upload, search, chat and more. It even provides a low-code agent builder interface - more on that later.

• An API server, providing API endpoints for the required functionality (document upload, search, chat, etc.). The API server also integrates the LLMs/models and serves the data storage.

  What's especially interesting, the API server not only serves the backend for the web application, but can be utilized with external tooling. So you might integrate RAGFlow with Open WebUI (where RAGFlow would be your powerful, high-quality RAG backend and Open WebUI your general purpose Chat UI).

• Task Executor, an asynchronous task executor, which is used to process the document uploads and indexing in the background.

RAGFlow Architecture

Running RAGFlow locally using Docker

To get started with RAGFlow, let's use Docker to run the application. Before you start, you need to make some decisions:

1. Where to you want to host the uploaded files. Self-hosted MinIO? S3? It just needs to be something S3 compatible.

2. Which metadata storage database do you want to use? The default option is MySQL, but you can also use PostgreSQL.

3. Which LLM provider do you want to use? RAGFlow supports many model providers. We need to make this decision as we need to set the environment variables for the LLM provider before starting the Docker container.

4. Which vector database do you want to use? (Well, it's not only a vector datbase - bascially, which database do you want to use to store your texts and vectors?). By default, RAGFlow uses Elasticsearch, but you can also use Infinity. Infinity is a new player on the field, marketing itself as "AI native database". And it has some impressive performance benchmarks.

For this guide, we'll use the following setup:

• MinIO for file storage
• PostgreSQL for metadata storage. Because we like PostgreSQL. And it's boring in the best way possible.
• OpenAI for LLM provider
• Infinity for vector database - not because Elasticsearch is bad (quite the contrary, it's great). But infinity is cool (as we went with the boring solution for the other components, we wanted to try something exciting here).

To get started, make sure docker is installed. Then get yourself all the files in the RAGFlow repositories docker folder.

1git clone https://github.com/infiniflow/ragflow.git
2cd ragflow/docker
3
4# Check out the latest stable version
5git checkout -f v0.19.0

Next, we need to define our docker compose file as well as the environment variables, for the docker compose file. Additionally, there is some configuration to change. As this is quite a lot, let's make a plan:

1. Then we need to adjust the default .env file for our needs.

2. We need to adjust the docker-compose.yml and docker-compose-base.yml to include the services we want to use.

3. As we use infinity, we need to adjust the infinity_conf.toml (or at least have a look).

4. Finally, there is a service_conf.yaml.template file which defines some of the RAGFlow specific configuration.

Let's start with the .env file. This sets the environment variables used in the docker compose file (not the containers). Make the following changes to the default .env file:

1DOC_ENGINE=infinity
2
3# The maximum amount of the memory, in bytes, that a specific Docker container
4# can use while running.
5# Update it according to the available memory in the host machine.
6MEM_LIMIT=8073741824
7
8# Set this to the timezone of your server.
9TIMEZONE=Europe/Vienna
10
11# There is no default setting for this variable, so add it.
12DB_TYPE=postgres
13
14# Add these settings for the PostgreSQL database.
15
16POSTGRES_USER=rag_flow
17POSTGRES_PASSWORD=your_password_here
18POSTGRES_DBNAME=ragflow
19POSTGRES_PORT=5432
20POSTGRES_HOST=postgres

Next, open the docker-compose-base.yml file (this contains all the dependencies for RAGFlow).

1. Delete the services with the names es01 and mysql. Also delete the volumes mysqldata and esdata01.

2. Add the following service for running postgres. Omit this step if you want to use an external PostgreSQL database.

   1postgres:
   2  image: postgres:16
   3  container_name: ragflow-postgres
   4  env_file: .env
   5  environment:
   6    - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
   7    - POSTGRES_DB=${POSTGRES_DBNAME}
   8    - POSTGRES_USER=${POSTGRES_USER}
   9    - TZ=${TIMEZONE}
   10  ports:
   11    - 5434:5432 # Note, make sure to not use 5432 for the host port, as this
   12      # would otherwise conflict with infinities PSQL_PORT.
   13  volumes:
   14    - ./ragflow-postgres_data:/var/lib/postgresql/data
   15  networks:
   16    - ragflow
   17  healthcheck:
   18    test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
   19    interval: 10s
   20    timeout: 10s
   21    retries: 5
   22  restart: on-failure
   

3. Replace ../history_data_agent with ./history_data_agent (except if you want to have the history data agent in a different location than the other RAGFlow service data).

Let's move on the docker-compose.yml file. This file defines the main RAGFlow service. Note down the local volume mounts. And change the depends_on from mysql to postgres (if you use a new PostgreSQL database).

Enabling GPU support for RAGFlow

If you have a GPU available, you can enable GPU support for RAGFlow. Simply add the GPU support to the docker compose file, as follows:

1ragflow-server:
2  image: ...
3  ...
4  deploy:
5    resources:
6      reservations:
7        devices:
8          - driver: nvidia
9            count: all
10            capabilities: [gpu]

Almost done! The second-last thing, let's have a look at the infinity_conf.toml file. This file contains the configuration for the infinity database. We don't need to change anything here. Just have a look at the settings and change the ones which are obviously wrong for your setup (like memory).

Finally, let's adjust the service_conf.yaml.template file. This file contains the configuration for RAGFlow itself. Open it and change the following settings:

1. Uncomment the postgres section. The default values of the postgres section are already fine.

2. If you need to use a different blob storage or enable oauth2 login, you can do so here.

Let's create the local directories which are used to store the data:

1mkdir ./ragflow-logs
2mkdir ./history_data_agent
3mkdir ./ragflow-postgres_data
4mkdir ./infinity_data

That should be all. Let's start the thing with docker compose:

1docker compose up -d

After a few seconds, check the logs of the ragflow-server container:

1docker logs -f ragflow-server

You should see something along the lines of:

1* Running on all addresses (0.0.0.0)

Now you can navigate to http://localhost and you should see the RAGFlow web application. You'll be greeted with a login screen, where you can register a new user and then log in.

RAGFlow Login

After logging in, you can get started.

RAGFlow Dashboard

Setting up RAGFlow

Now that we have ourselves a running instance, let's set it up for basic RAG usage.

First, add the required AI models to RAGFlow.

1. Click on your profile icon in the top right corner, then select "Model providers".

2. From there, select the model provider you want to use (e.g., OpenAI), and add your API key. For this guide, we're using OpenAI.

   RAGFlow Model Providers

3. After adding a model provider, we need to set the default (system) models. This can be done in the same screen, by clicking "Set default models". Select the models you want to use for the different tasks (e.g., chat, embedding, etc). Make sure to set at least the chat and embedding models, as these are required for the basic RAG functionality.

   RAGFlow Set Default Models

After adding the model provider, we're ready to upload our first document.

1. Click on "Knowledge Base" in the top navigation bar and click "Create knowledge base".

2. Enter a name for your knowledge base and click "Create". This will create the knowledge base and redirect you to the knowledge base configuration page.

   ![Knowledge Base

   Creation](/blog/ragflow/knowledge-base-configuration.png)

3. Here you can enter a name, photo and description for the knowledge base. The more important settings though are the following:

   • PDF Parser: PDF is one of the most ridiculously complex file format you could imagine. (I'm getting angry just thinking about it). But it unfortunately is the most common file format for documents. RAGFlow therefore provides a custom PDF parser called 'deepdoc' which seems to be one of the better OCR solutions out there. We suggest using it for pdf parsing. Alternatively, you can use the vision capabilities of 'gpt-4.1' or 'gpt-4o' to parse the pdfs. This is a bit more expensive, but could also yield good results. We'd stick with the 'deepdoc' approach for now.

   • Embedding Model: This is the model used to create the embeddings for the documents. You can use any of the models you added before. We suggest using 'text-embedding-3-small' or 'text-embedding-3-large' from OpenAI, as these are the most advanced ones for now.

   • Chunking method: This defines how the documents are chunked into smaller pieces for indexing. (Because we can't use full, long documents due to costs, and context window of the LLMs). There are a multitude of methods available. It's best to scroll through them, as RAGFlow provides a very nice description of each of the methods at the right-hand-side of the screen.

     Chunking Methods

     Note that you can later on override this setting on a per-document level. So choose the one which you think is best for your knowledge base but don't worry too much about it. You can always change it later.

   • Page Rank: You can increase the page rank of a knowledge base, basically making this knowledge base more important than others.

   • Auto-Keyword: Set the number of keywords which should be automatically extracted from the documents. This is used for searching and filtering the documents later on. We highly recommend this setting, as this might increase the retrieval quality later on. Just keep in mind that this will increase the cost and time for initial indexing of the documents.

   • Auto-Question: This setting allows RAGFlow to automatically generate questions from the documents. This might help during retrieval, as potential questions might be semantically similar to real-world user questions - thus improving the retrieval quality. Works similar to HyDe.

   • RAPTOR: From the RAGFlow: RAPTOR (Recursive Abstractive Processing for Tree Organized Retrieval) is an enhanced document preprocessing technique introduced in a 2024 paper. Designed to tackle multi-hop question-answering issues, RAPTOR performs recursive clustering and summarization of document chunks to build a hierarchical tree structure. This enables more context-aware retrieval across lengthy documents. Again, this might be very good for longer documents, but will require many more tokens! See here for more information on how RAPTOR works.

     • Extract knowledge graph: This setting makes RAGFlow use a GraphRAG-like system to extract a knowledge graph from the documents, to enhance multi-hop question-answering involving nested logic. This can enhance the retrieval quality for complex documents, but will definitely increase the cost and time for indexing the documents manifold. Don't enable this by default - more often than not you'll just increase costs without any real benefit. On certain cases however it can be very beneficial. Trial and error is the way to go here.

   Click on "Save" to save the knowledge base configuration and continue to the "Dataset" section.

4. In the "Dataset" section, you can upload documents to the knowledge base. Click on "Add File" to get a file upload dialog. You can:

   • Upload a single or multiple files at once - up to 32 files and 1GB at a time.

   • Upload a folder with files.

   Add your file and then click "OK".

   RAGFlow Upload Documents

   After that, the file is available in the "Dataset" table - but it will not be indexed yet. Click on the green "Play" button the start the parsing process.

   Optionally, click on the "Settings" button at the file to change the Chunking Method or manually add some meta information.

   Change file chunking method

   Metadata are added as json file, and note that the whole metadata object is added to the LLM prompt when asking questions later on. So add everything here you want to be available for answering questions.

   Change file meta data

5. Wait for the indexing to finish. If you're interested what exactly RAGFlow was doing, you can hover over the "Success" badge after the indexing is done.

   RAGFlow Indexing Information

   To test the knowledge base, click on "Retrieval testing" in the left-side menu. Play around with the retrieval settings and see which chunks are retrieved and what the similarity scores are.

   RAGFlow Retrieval Testing

One particularly nice feature is to change the parsed and chunked documents later on (or even see what the parsing and chunking process did to the documents). In the "Dataset" tab, click on the document and an overlay will open, showing the chunks of the document, the extracted keywords and questions. It will even show the original document on the right-hand-side, so you can see how the document was chunked and parsed.

Double-click on a chunk to edit it.

RAGFlow Document Chunks

Using a RAGFlow Knowledge Base

We're getting there. The most important part is done, we have documents in our knowledge base and can retrieve them.

There are two integrated ways to use this knowledge base:

1. Using the "Chat" feature, which allows you to ask questions and get answers based on the knowledge base.

2. Using the "Search" feature, which allows you to search for document chunks and get a list of chunks matching your search query. It also gives a summarized answer to your search question.

The main difference is, that the "search" feature is intended as a single-shot Q&A with focus on retrieval quality. You'll not only get an answer, but also detailed information about which chunks were used.

RAGFlow Search

While the "chat" feature is more like your traditional chatbot with multiple turns and a conversation history. It will also show you in which document the answer was found, but it's less focused on this part. Furthermore, the chat can be granularly configured.

Configuring the RAGFlow Chat

In the "Chat" section, click on "Create an Assistant". This opens a dialog where you can configure the chat assistant. You can set the following settings:

• Name: The name of the assistant.
• Description: A description of the assistant, which will be shown to the user.
• Avatar: An avatar for the assistant, which will be shown in the chat.
• Empty Response: Define the response when no documents were found for answering a user question.
• Opening Greeting: The opening greeting of the assistant, which will be shown to the user when the chat is started.
• Show quote: Whether to show the original text/document as reference (highly recommended).
• Keyword analyses: Apply LLM to analyze user's questions, extract keywords which will be emphasize during the relevance computation.
• Knowledge Base: Select the knowledge base which should be used for this assistant. You might also select multiple knowledge bases.

RAGFlow Chat Configuration

But wait, there is more! On the "Prompt Engine" tab you can configure details about which prompt to use for the generation part of the chat, as well as details about retrieval. We're skipping this part for now, as all the settings are quite self-explanatory.

RAGFlow Prompt Engine

Once you click "Ok", you can create a new chat. The chat interface is basically what you'd expect form your typical AI chatbot. Nothing fancy, but also no major issues.

RAGFlow Chat Interface

Embedding a RAGFlow Chat in your application or homepage

The final feature we're going to look at is the ability to embed a RAGFlow chat in your application or homepage. This is done by creating an assistant as described above, then hovering over the assistant and clicking "Embed into webpage".

Embed RAGFlow Chat

But before that, you need to create an API key for your RAGFlow instance. Click on you profile icon in the top right corner, then select "API" and then "API Keys". Click on "Create API Key".

RAGFlow API Key

Now that you have a key, you create the embed as shown above. This will provide a code snippet which you can copy and paste into your application or homepage.

RAGFlow Embed Code

The chat widget will look like this:

RAGFlow Chat Widget

Conclusion

So, that's a wrap for our introduction to RAGFlow. We hope you found this article helpful and informative. In our next article we'll tackle more advanced RAGFlow topics, like the awesome "Agent Builder" feature, and how to integrate RAGFlow with Open WebUI to create a powerful, open-source RAG system with a great chat UI.

Interested in how to train your very own Large Language Model?

We prepared a well-researched guide for how to use the latest advancements in Open Source technology to fine-tune your own LLM. This has many advantages like:

• Cost control
• Data privacy
• Excellent performance - adjusted specifically for your intended use