In today’s data-driven world, getting accurate and timely answers from vast repositories of information is a significant challenge. While Large Language Models (LLMs) are powerful tools, they frequently have issues such as an inability to access real-time or private data and a tendency to “hallucinate,” which means generating confident-sounding but incorrect information.
This is where Retrieval-Augmented Generation (RAG) becomes useful. RAG is a powerful technique that merges the best aspects of both a robust search engine’s precision and an LLM’s vast generative power. By creating a RAG system, you can ground an LLM’s responses in your own private data, which ensures accuracy, relevance, and full control.
This article details how to build such a system using Elasticsearch for vector search and a local LLM for generation. The provided code examples, which are built with Python, LangChain, and LangGraph, show a practical, production-ready architecture that can be adapted for a wide range of business use cases.
Architecture of my solution
What is RAG and why does it matter?
At its core, RAG is a three-step process: Retrieve, Augment, and Generate.
- Retrieve: When a user asks a question, the system first retrieves the most relevant documents or “chunks” of information from your private knowledge base. A powerful search engine like Elasticsearch is especially effective for this crucial step.
- Augment: The retrieved information is then added to the user’s original question, which creates a rich “context” that is passed to the LLM.
- Generate: The LLM, which is now equipped with the specific and relevant context, generates a precise and accurate answer that is grounded in your data.
This process significantly reduces hallucinations and guarantees that the LLM’s response is both trustworthy and up-to-date.
Why use Elasticsearch and a local LLM?
This specific combination offers significant advantages:
- Vector Search with Elasticsearch: Unlike traditional keyword-based search, vector search understands the meaning and intent behind a query. Elasticsearch is a scalable and flexible solution for vector databases that converts documents into numerical representations called embeddings. This allows it to find the most conceptually relevant information for a given query, even if the exact keywords are not present.
- Privacy and Control with Local LLMs: By using a local LLM, such as one run with Ollama, your sensitive data never leaves your company’s infrastructure. This is crucial for industries with strict privacy regulations. It also gives you full control over the model’s behaviour, performance, and costs without being dependent on third-party APIs.
Step 1: Building the RAG System – The Ingestion Flow
The first half of the architecture diagram shows how your private data, such as a PDF, is prepared and stored.
1 – Ingestion Code (Python): A Python script reads the PDF document, cleans the text to remove formatting errors, and then divides it into smaller text chunks. This is done to help the system handle information in manageable pieces, as an entire PDF document can be too large to be fed into an embedding model. Attempting to embed an entire document would likely result in it being truncated or the model failing. Chunking breaks the document into smaller, more focused pieces, which allows the embedding model to create a more accurate and meaningful vector for each chunk. This ensures that when a user asks a specific question, the vector search can find a highly relevant chunk instead of a single, ambiguous vector for the entire document.
##### Loading PDF #####
loader = PyPDFLoader(pdf_path)
docs = loader.load()
##### Chunking Strategy #####
print("Chunking document with RecursiveCharacterTextSplitter...")
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
)2 – Create Vector Embeddings: Each of these text chunks is then passed to an embedding model, which converts the text into a vector embedding—a unique array of numbers that represents its meaning.
local_embeddings = HuggingFaceEmbeddings(model_name=MODEL_NAME)3 – Elasticsearch (Vector Index): These vector embeddings, along with the original text chunks, are stored in Elasticsearch. This database functions as a specialised knowledge base that is optimised for searching these vectors.
response = es_client.index(
index=INDEX_NAME,
document={
"description": chunk.page_content,
"page_number": chunk.metadata.get('page', -1) + 1,
"description_embedding": vector
}
)Step 2: Main Application / Query Flow – Getting an Answer
The second half of the diagram shows what happens when a user interacts with the system.
1 – User Question: A user types in a natural language question.
Example - st.chat_input("How to setup Kibana SSO ? ")2 – Create Query Embedding: The system uses the same embedding model to convert the user’s question into a vector, just as it did with the document chunks.
question_embedding = embeddings.embed_query(question)3 – Elasticsearch (Vector Index): The query’s vector is sent to Elasticsearch, which then performs a vector search. It finds the stored document chunks whose vectors are the most similar to the query’s vector.
knn_query = {"knn": {"field": "text_embedding.predicted_value", "query_vector": question_embedding, "k": 5, "num_candidates": 20}, "_source": ["text"]}4 – Retrieve Relevant Chunks: The most similar document chunks are retrieved from Elasticsearch.
response = es_client.search(index=index_name, body={"knn": knn_query['knn'], "
_source": knn_query['_source']})
context = "\n\n---\n\n".join([hit['_source']['text'] for hit in response['hits']['hits']])5 – Local LLM: The original user question and the retrieved relevant chunks are combined and sent to a local LLM. The LLM is instructed to use these chunks as context to answer the question.
##### Creating Prompt #####
prompt = f"Answer based only on this context:\nCONTEXT:{context}\nQUESTION:{question}\nANSWER:”
##### Invoking LLM #####
answer = local_llm.invoke(prompt).content
return {"answer": answer, "context": context}
6 – Generated Answer: The LLM uses the provided context to formulate a coherent and accurate answer, which is then sent back to the user. Note: Check screenshots below to see some example responses.
Business and real-life use cases
This RAG architecture is a versatile framework for building intelligent applications across various industries.
1. Enterprise Knowledge Management and Internal Support
This RAG architecture is a versatile framework for building intelligent applications across various industries.
- Employee Assistants: A company can train an internal RAG system on its HR policies, IT documentation, and internal wikis. When an employee asks, “What’s the policy for sick leave?” or “How do I connect to the VPN?”, the RAG system retrieves the exact policy from the company’s documents and provides a concise, accurate answer, rather than a generic one.
- Customer Support Agents: RAG can create a ‘copilot’ for call center and support agents. The system can instantly pull up relevant information from product manuals, customer records, and internal knowledge bases to help the agent provide fast and accurate solutions to customer problems. This reduces the time an agent spends searching for information and improves the quality of service.
2. Legal Research and Document Analysis
The legal field is heavily reliant on a vast and complex body of documents. RAG is a perfect fit for this environment.
- Legal Research Assistant: A RAG system can be trained on a massive database of case law, statutes, and legal precedents. A lawyer can ask a natural language question like, “What are the key precedents regarding trademark infringement for a company in the technology sector?” The system will retrieve relevant cases and legal texts and provide a summarised, coherent response, complete with citations to the source documents. This drastically reduces the time and effort spent on legal research.
- Contract Analysis: RAG can quickly analyse and summarise complex legal documents like contracts. It helps legal professionals with due diligence and review by identifying key clauses, highlighting risks, and comparing terms against a standard template. It can identify key clauses, highlight potential risks, and compare terms against a standard template to help with due diligence and contract review processes.
3. Personalised Education and E-Learning
RAG can personalise the learning experience and provide students with more dynamic and interactive educational tools.
- Intelligent Tutoring Systems: RAG-powered tutors can be integrated with a student’s course materials, textbooks, and notes. When a student asks a question about a specific topic, the system can retrieve information from those materials and generate a personalised explanation or an example tailored to the student’s learning context.
- Research Assistants: For university students and researchers, RAG can act as a powerful tool to synthesise information from academic papers, journals, and a university’s internal research database. This helps them quickly get up to speed on a topic, identify key findings, and find relevant citations.
4. Healthcare and Medical Diagnostics
In healthcare, where information accuracy is critical, RAG can provide support for professionals.
- Clinical Decision Support: Medical literature, electronic health records, and clinical guidelines can be used to train a RAG system. Doctors can use a RAG system to get synthesised responses to questions about patient symptoms, including potential diagnoses and treatment protocols, based on up-to-date medical data.
- Patient Education: RAG can create more effective patient-facing tools. A system can pull information from a hospital’s resources to give patients clear answers about their condition, medication, and instructions.
5. Financial Services and Market Intelligence
RAG can help analysts and traders navigate the deluge of financial data.
- Financial Document Summarisation: A RAG system can ingest and summarise extensive financial documents like earnings reports and analyst briefings. An analyst can ask, “What were the main drivers of revenue growth for Company X in the last quarter?” and the system will provide a real-time, data-driven summary based on the latest filings.
- Market Intelligence: RAG can pull information from news feeds, market data, and proprietary research to provide up-to-the-minute insights. A trader could ask, “What’s the public sentiment around a new product launch from Company Y?”, and the system would synthesise a response based on recent articles and social media trends.
Screenshots of Demo application
Screenshot 1: Query Interface
This image displays the user interface where a question is entered. The system processes the query to retrieve the most relevant information.
Screenshot 2: Generated Answer with Sources
The system shows the generated answer and the specific document chunks used to formulate the response, which highlights the transparency and accuracy of the RAG system.
You can always see the source of truth, this way users can see exactly where the information came from, increasing trust in the generated answer.
When properly implemented and grounded in your specific data, one can avoid hallucinations and provide accurate, contextually relevant information.
Here you can find step by step instructions only based on information it fetched from the knowledge base.
Final conclusion
The RAG architecture leverages Elasticsearch for vector search and local LLMs for generation, offering a robust and adaptable solution for overcoming the limitations of traditional chatbots and assistants. This system grounds responses in private, domain-specific data, ensuring accuracy, reducing hallucinations, and providing unparalleled control over information.
The diverse range of real-life applications, from enterprise knowledge management to healthcare diagnostics and financial intelligence, demonstrates the immense potential of RAG. This technology can revolutionise how organisations interact with their data and deliver precise, contextually rich answers. Embracing RAG empowers businesses to unlock new levels of efficiency, trust, and intelligence in an increasingly data-centric world.
Please let us know what your take is on RAG Systems and what are you thinking to solve / build / enhance using this framework.