Chat with your Confluence: A Step-by-Step Guide using Airbyte, Langchain and PGVector

On this page

This guide shows how to build a chatbot that answers questions from your Confluence pages. We load the pages with Airbyte's Python library PyAirbyte, through LangChain's AirbyteLoader, and clean the HTML with BeautifulSoup. Then we store OpenAI embeddings in PostgreSQL with the PGVector extension and answer questions with a LangChain RetrievalQA chain and GPT-4 Turbo. The whole pipeline fits in a few dozen lines of Python.

Many teams keep their documentation in Confluence, and finding the right page in a large space is slow. A chatbot driven by a Large Language Model (LLM) and connected to your Confluence data lets people ask questions in plain language instead.

We'll start with Airbyte, an open-source data integration platform, and use it to sync Confluence data. Next comes PGVector, a PostgreSQL extension for vector operations, which we use to search the data by meaning. Finally, LangChain glues the components together. If you're new to the underlying pattern, our introduction to Retrieval Augmented Generation (RAG) explains it.

What is an LLM and why use it for a chatbot?

A Large Language Model (LLM) like GPT-4 is an AI model that understands and generates human-like text. LLMs are trained on huge datasets, so they can follow context, answer questions and hold a conversation. For a chatbot, this means more natural, accurate and relevant answers to user questions.

Introduction to Airbyte, LangChain and PGVector

Airbyte is an open-source data integration tool that syncs data from various sources to databases, data lakes or data warehouses. In our case, Airbyte reads the data from Confluence.

PGVector is a PostgreSQL extension for vector operations. It stores embeddings and runs fast similarity searches over them, which makes it a good fit for our chatbot's backend. For a primer on what PGVector is and why it works so well for chatbots, have a look at this blog post about "What is PGVector"

LangChain is a framework for connecting LLMs with other tools. It lets you build a chain of components, where the output of one becomes the input of the next. In our case, LangChain connects Airbyte, PGVector and the LLM.

Prerequisites

To follow along with this guide, you'll need the following already set up:

• A Confluence account with access to the data you want to query.
• A PostgreSQL database. This is where your Confluence data will live for the chatbot to access.
• Docker and Docker Compose on your local machine. Follow the official Docker instructions

Setting up Airbyte and LangChain for Confluence data integration

Airbyte has long been a flexible, UI-based tool for data integration. Its more than 300 connectors cover a wide range of data sources and destinations. The catch was that you had to run the whole Airbyte platform on your system or use their managed cloud offering.

With PyAirbyte, Airbyte's Python library, you can use the Airbyte connectors from within a Python script without running Airbyte itself. Setting up Airbyte becomes as easy as an import.

Airbyte and LangChain are also well integrated. LangChain provides an AirbyteLoader that uses PyAirbyte to run any Airbyte connector and feed the data into the LangChain ecosystem.

I can't overstate how useful this combination of LangChain and Airbyte is. We finally have tools that connect virtually any data source with LLMs, and we can build chatbots on top of them.

Setting up Confluence as a source in the LangChain AirbyteLoader

With the theory out of the way, let's set up our chatbot.

1. Create a Confluence API token in your Confluence instance.

   • Navigate to your confluence api token screen
   • Click on "Create API Token" and give it a label, e.g. "Airbyte Integration"
   • Copy the generated token and store it in a safe place. You will need it later.

2. Install the LangChain AirbyteLoader

   1pip install langchain
   2pip install langchain-airbyte
   3pip install langchain-openai
   4pip install langchain-postgres
   5pip install beautifulsoup4
   6pip install psycopg
   

3. Import the AirbyteLoader and set up the Confluence source

   1from langchain_airbyte import AirbyteLoader
   2
   3loader = AirbyteLoader(
   4    source = "source-confluence",
   5    stream = "pages",
   6    config = {
   7        "domain_name": "devopsandmore.atlassian.net",
   8        "email": "andreas.nigg@devopsandmore.com",
   9        "api_token": "<your-confluence-api-key>",
   10    }
   11)
   

4. That's all we need to read the Confluence pages. Let's print the result.

1docs = loader.load()
2print(docs[0].page_content[:500])

The output is YAML and might look like this:

1_expandable:
2  ancestors: ''
3  childTypes: ''
4  children: /rest/api/content/65610/child
5  container: /rest/api/space/~625124fdf813eb00692f81e5
6  metadata: ''
7  operations: ''
8  schedulePublishDate: ''
9  schedulePublishInfo: ''
10  space: /rest/api/space/~625124fdf813eb00692f81e5
11_links:
12  editui: /pages/resumedraft.action?draftId=65610
13  self: https://devopsandmore.atlassian.net/wiki/rest/api/content/65610
14  tinyui: /x/SgAB
15  webui: /spaces/~625124fdf813eb00692f81e5/pages/65610/Sample+Pages
16body:
17  _expandable:
18    anonymous_export_view: ''
19    atlas_doc_format: ''
20    dynamic: ''
21    editor: ''
22    editor2: ''
23    export_view: ''
24    styled_view: ''
25  storage:
26    _expandable:
27      content: /rest/api/content/65610
28    embeddedContent: []
29    representation: storage
30    value: "\n<ac:structured-macro ac:name=\"info\" ac:schema-version=\"1\" ac:macro-id=\"\
31      e4a7c28f-0952-47e9-bf6c-a6edb999e9bf\"><ac:rich-text-body><p> We've created\
32      \ a few sample pages that you can use to get started.</p></ac:rich-text-body></ac:structured-macro>\n\
33      <p />\n<table data-layout=\"wide\">\n<tbody>\n    <tr>\n        <td><h3 style=\"\
34      text-align: center;\"><ac:emoticon ac:name=\"tick\" /> <a href=\"/wiki/spaces/~625124fdf813eb00692f81e5/pages/65625/Product+requirements\"\

That's a good start, but there are two problems to tackle:

1. The sync is quite slow. The loader.load() method reads ALL documents synchronously.
2. The output contains a lot of fields we don't need.

LangChain has us covered on both. First, we can replace the loader.load() call with:

1my_async_iterator = loader.alazy_load()
2async for doc in my_async_iterator:
3    print(doc.page_content)

This lazy-loading version of the AirbyteLoader only loads Confluence pages when they are needed. It also has an async interface, so we could run the operation concurrently.

Second, LangChain lets you choose which fields are returned, and in what form. You do this by passing a template to the AirbyteLoader constructor.

1from langchain_core.prompts import PromptTemplate
2
3loader = AirbyteLoader(
4    source = "source-confluence",
5    stream = "pages",
6    config = {
7        "domain_name": "devopsandmore.atlassian.net",
8        "email": "andreas.nigg@devopsandmore.com",
9        "api_token": "<your-confluence-api-key>",
10    },
11    template=PromptTemplate.from_template(
12        '{body[storage][value]}'
13    ),
14)

The template parameter is a template string that defines which fields to select from the response. We want the body->storage->value field, which the template string body[storage][value] selects.

One issue remains: the page content is full of HTML tags we don't need. The beautifulsoup4 package solves this. It parses HTML and turns it into plain text, which is what we want to send to the LLM later.

LangChain also extracts a lot of metadata we don't need, so we remove most of it. Our final code to pre-process the Confluence data looks like this:

1from bs4 import BeautifulSoup
2my_async_iterator = loader.alazy_load()
3docs = []
4async for doc in my_async_iterator:
5    doc.metadata = {"title": doc.metadata["title"],
6                    "createdAt": doc.metadata["history"]["createdDate"],
7                    "createdBy": doc.metadata["history"]["createdBy"]}
8
9    html_content = doc.page_content
10    soup = BeautifulSoup(html_content, 'html.parser')
11    clean_html = soup.get_text()
12
13    # Appending the title to the page content, as we want to use the title in
14    # our pgvector embedding similarity search
15    doc.page_content = doc.metadata["title"] + "\n" + clean_html
16    docs.append(doc)

That's it for preprocessing.

Storing the Confluence data in a PGVector vector store

With the data extracted, we can put it into our PGVector vector store.

If you haven't already, install PGVector in your Postgres database. On your Postgres database host, run

1cd /tmp
2git clone --branch v0.7.0 https://github.com/pgvector/pgvector.git
3cd pgvector
4make
5make install # may need sudo

Alternatively, you can run a PGVector-enabled Postgres directly from Docker with the image pgvector/pgvector:pg16.

Next, enable the PGVector extension. Connect to your Postgres database and run

1CREATE EXTENSION vector;

That's all PGVector needs. Now we can use LangChain to insert our data.

Creating embeddings from texts and inserting them into PGVector

At a high level, we have two tasks:

1. Create vector embeddings from the Confluence texts
2. Insert these embeddings, alongside the texts, into our PGVector vector database

We could do this manually, but LangChain's abstractions reduce it to a few lines of code.

1from langchain_openai import OpenAIEmbeddings
2from langchain_core.documents import Document
3from langchain_postgres import PGVector
4from langchain_postgres.vectorstores import PGVector
5
6# See docker command above to launch a postgres instance with pgvector enabled.
7connection = "postgresql+psycopg://<user>:<password>@<host>:<port>/<database>"  # Change to your postgres instance
8collection_name = "blog_docs"
9embeddings = OpenAIEmbeddings(api_key="<your-openai-api-key>")
10
11vectorstore = PGVector(
12    embeddings=embeddings,
13    collection_name=collection_name,
14    connection=connection,
15    use_jsonb=True,
16)
17
18vectorstore.add_documents(docs)

LangChain uses an OpenAI embedding model to create the vector embeddings, so you need a working OpenAI API key.

If you'd rather create embeddings inside the database itself, our post on using AI directly from PostgreSQL with pgai shows how.

Chatting with the Confluence data using LangChain RetrievalQA

With the Confluence data stored and prepared, we can finally use our LLM, GPT-4 Turbo, to answer questions based on it.

1from langchain.chains import RetrievalQA
2from langchain_openai import ChatOpenAI
3
4qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(temperature=0, model="gpt-4-turbo", api_key="<your-openai-api-key>", streaming=True),
5    chain_type="stuff",
6    retriever=vectorstore.as_retriever())

To run a prompt, call

1qa.run("How to create invoices with SevDesk?")

That's all we need. Under the hood, LangChain does the following:

1. Create vector embeddings from the search query.
2. Use PGVector's vector similarity search to find documents in the vector store that are semantically similar to the search query.
3. Send the question plus the similar documents to the GPT-4 Turbo large language model.
4. Return the answer from the model.

Conclusion

This guide built an AI chatbot on top of Confluence data: Airbyte for very easy data extraction, PGVector for storing text embeddings, and LangChain for gluing the components together.

Key points:

• Large Language Models (LLMs) like GPT-4 let chatbots give more natural, accurate and relevant answers, because they understand and generate human-like text.

• Airbyte syncs data from sources like Confluence to databases. PyAirbyte, Airbyte's Python library, runs Airbyte connectors directly from Python without the full platform.

• PGVector is a PostgreSQL extension for vector operations, which makes it a convenient store for the embeddings an LLM chatbot searches.

• LangChain chains tools together. Here it connects Airbyte, PGVector and the LLM.

• LangChain's AirbyteLoader uses PyAirbyte to load Confluence data. We pre-process the data to keep only the relevant fields and strip the HTML.

• PGVector stores the cleaned Confluence data together with vector embeddings created by OpenAI's embedding model via LangChain.

• LangChain's RetrievalQA chain combines the PGVector store and the GPT-4 model to answer user questions from the stored Confluence knowledge.

From here, the obvious next step is to add concurrency to the loading step and run the sync on a schedule, so the chatbot stays current with your Confluence space. Before people rely on the answers, build a small test set and score the pipeline, as shown in our guide on evaluating RAG pipelines with Ragas. For a different take on the same problem, read how we moved our company documents from Confluence to a self-maintaining LLM wiki.

If you want a Confluence chatbot like this running in production, our AI consulting team builds custom RAG systems on company knowledge bases.

Further reading

• RAG with Azure AI Search: building enterprise retrieval on Azure
• Using SharePoint lists as knowledge sources in Copilot Studio agents

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

• Understanding the 14 systematic failure modes in multi-agent systems
• Evidence-based best practices for agent design
• Structured communication protocols and verification mechanisms

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.

Talk to Andreas about your projectFollow on LinkedIn