I Built a Document Q&A System Without a Single Vector - Here's How

 What if you could ask questions across all your documents and get cited, trustworthy answers — without training embeddings, running a vector database, or managing any ML infrastructure? 

Vectorless RAG

That was the question behind Vectorless RAG, a project I built to explore a simpler, more transparent path to document intelligence.

In this post I'll walk through:

• The core idea behind Vectorless RAG
• How the system works end to end
• The practical trade-offs of going vectorless

The Problem With Most RAG Systems

Retrieval-Augmented Generation (RAG) has become the standard approach for making large language models answer questions about your own documents. The typical recipe is:

1. Break documents into chunks.
2. Convert every chunk into a vector embedding.
3. Store those vectors in a specialized vector database.
4. At question time, embed the question and find the closest chunks.
5. Feed those chunks to an LLM to write an answer.

This works well, but it comes with real costs. You need an embedding model running at all times. You need a vector database (Pinecone, Weaviate, Chroma, etc.) as a separate service to operate and maintain. The vectors themselves are opaque—you cannot look at a list of numbers and understand why a particular passage was retrieved. And if something goes wrong in retrieval, it is very hard to debug.

I wanted to see how far I could get without any of this. No embeddings. No vector store. No specialized numeric pipeline.

The Core Idea: Keyword Search + a Knowledge Graph

Vectorless RAG reaches the same goal through two ideas that have been around for decades — used together in a way that covers each other's weaknesses.

1. Keyword Search (BM25)

When a document is uploaded, its text is broken into passages and indexed in a full-text search index powered by SQLite FTS5 with BM25 ranking.

BM25 is the battle-tested relevance algorithm behind search engines for the past thirty years. It is fast, precise, and completely interpretable — you can inspect exactly which words matched and how strongly.

This handles questions that use the same language as the document perfectly. But it has a known blind spot: ask about a "vehicle" and you may miss passages that only mention a "car."

2. A Knowledge Graph (built by an LLM)

To bridge that gap, the system builds a knowledge graph from every uploaded document.

As each document is ingested, an LLM reads through it and extracts important entities such as:

• People
• Organizations
• Concepts
• Dates
• Figures

It also captures the relationships between those entities. This graph is stored in the same SQLite database alongside the text.

At question time, the system looks up entities mentioned in the question, then follows the graph's connections one step outward to gather related ideas.

These related terms are folded back into the keyword search. A question about one company can surface passages about its subsidiaries, time periods, or associated financial figures — even when none of those words appear in the original question.

The knowledge graph gives the system the "meaning-aware" quality that vector search would normally provide, but in a form a human can actually read, inspect, and trust.

3. A Cross-Encoder Reranker (the optional finishing touch)

After keyword search and graph expansion produce a shortlist of candidate passages, a small cross-encoder model (ms-marco-MiniLM-L-6-v2 via sentence-transformers) re-reads each candidate against the question and reorders them for precision.

This step is entirely optional — if the model cannot be loaded, the system falls back to keyword ranking and continues working normally.

The Ingestion Journey

Here is what happens the moment you upload a file.

Step 1 — Identify the File Type

Apache Tika inspects the file's actual bytes (its "magic bytes") to determine the true MIME type, ignoring whatever file extension the file may have.

It also pulls any metadata the file carries — such as:

• Author
• Title
• Language
• Page Count

This lightweight metadata-only pass avoids re-reading the entire document. If Java or Tika is unavailable, the system falls back to extension-based detection and continues without the extra metadata.

Step 2 — Extract Real Structure

IBM Docling parses the document into its genuine structure, including:

• Headings
• Paragraphs
• Tables
• Figures

Tables are preserved as readable units rather than flattened into surrounding text. Each passage is tagged with its section title and page number for precise citation later.

For plain PDFs where Docling is unavailable, PyMuPDF provides a lightweight text fallback.

Step 3 — Index It (The Document Is Now Ready)

The structured passages are added to the FTS5 index. At this point the document is marked as ready, and the user can immediately begin asking questions about it.

This is a deliberate design choice — users never have to wait for slower background processing before interacting with their documents.

Step 4 — Build the Knowledge Graph (Background Process)

An LLM reads through the passages in batches and extracts entities and relationships. This process runs quietly in the background after the document is already searchable.

If graph extraction fails for any passage, it is skipped gracefully, while keyword search remains fully functional.

Implementation Detail: Model Loading Strategy

IBM Docling's ML models are relatively heavy. Loading them for every upload would significantly slow down processing.

To avoid this, the application uses a process-wide singleton. The models are loaded once when the first file arrives and are then reused for every subsequent upload, with access serialized through a threading lock.

The Query Journey

Here is what happens when a user types a question.

Step 1 — Understand the Question

The system extracts the meaningful keywords and identifies the user's intent — whether the question is asking for a fact, definition, comparison, or summary.

Intent detection influences the rest of the retrieval process. For example, a comparison query may deliberately pull information from multiple documents.

Step 2 — Expand Using the Graph

Entities mentioned in the question are matched against the knowledge graph.

The system follows their connections one hop outward, gathering related entities and adding them to the search terms.

Step 3 — Search All Documents

A single BM25 query runs across all selected documents, returning the strongest candidate passages.

Step 4 — Score and Filter Documents

Each document is evaluated using multiple signals, including:

• Keyword match strength
• Whether matches appear in headings
• Density of relevant entities within the passage
• Document metadata

Only documents that clearly matter are retained, while the rest are discarded.

A simple question may require information from only one document, whereas a broader comparison could draw from several.

Step 5 — Rerank and Add Context

The cross-encoder reranks the top passages for improved precision.

Immediately surrounding passages are also included to preserve context, since an isolated sentence often depends on neighboring content to make complete sense.

Step 6 — Write a Cited Answer

The LLM generates the final answer using only the retrieved passages.

It is explicitly instructed to state when the documents do not contain sufficient information rather than inventing facts.

The answer is returned with citations pointing to the exact document, page, and section from which each fact was sourced.

The final result is cached for faster retrieval when the same or similar question is asked again.

The Technology Stack

Here is every piece of technology in the system, honestly stated.

What I Got Right (and the Honest Trade-offs)

What Works Well

Simplicity of Operation. There is no vector database to run, no embedding service to maintain, and no specialized infrastructure of any kind. The whole system is a Python process and a SQLite file. You can inspect the knowledge graph directly with any SQLite browser.

Transparency. Every retrieval decision can be explained. The BM25 score is a number with a clear meaning. The graph expansion path—which entities were found, which connections were followed, and how retrieval decisions were made—can be traced exactly. There are no opaque 1536-dimensional vectors involved.

Immediate Availability. Documents become answerable within seconds of upload. The knowledge-graph building process runs in the background. Users never have to wait for the slower step before they can begin asking questions.

Graceful Degradation. Every optional component—the reranker, the cache, Apache Tika, and the graph extraction pipeline—can be unavailable without breaking the core experience. The system includes clear fallback paths at every level.

Honest Citations. The system credits only the passages that genuinely contributed to an answer. Supporting context that was included for readability is used during answer generation but is not cited. This distinction improves trustworthiness and source transparency.

The Honest Trade-Offs

Vocabulary Gap. Pure keyword matching can miss passages that are phrased very differently from the user's question. Knowledge graph expansion and reranking narrow this gap significantly, but they do not eliminate it entirely. Systems that use dense vector embeddings still have an advantage when handling heavily paraphrased queries.

Graph Dependency. Retrieval quality improves with a strong knowledge graph. Documents whose graph extraction failed—or was never performed—fall back to keyword search alone, which provides less context-rich retrieval.

Scale Ceiling. SQLite is excellent for single-server, moderate-concurrency workloads. However, it reaches a natural limit when handling extremely high volumes of simultaneous writes. For teams and internal organizational deployments this is usually acceptable, but a public service operating at massive scale would eventually require distributed storage.

Surprising Decisions Worth Noting

FTS5 Scores Are Negative. In SQLite FTS5, bm25() returns negative numbers—where a more negative value indicates higher relevance.

I spent an embarrassing amount of time wondering why the "worst" results appeared first before realizing that results should be ordered by score in ascending order, not descending.

Making Async Endpoints Sync. FastAPI's event loop runs on a single thread, while SQLite operations and cross-encoder inference are blocking tasks.

Making the chat endpoint a regular def instead of an async def allows FastAPI to automatically execute it within its thread pool, keeping the event loop free without requiring any manual executor management.

The One-Time FTS Rebuild Guard. Every time a Database() object was constructed, the original implementation rebuilt the FTS index.

The fix was a single module-level Boolean flag—set it to True after the first rebuild and check it before every subsequent rebuild. Zero cost, complete fix.

The End Result

The finished system is a self-contained web application. Users can drag and drop documents onto the page, watch them move through queued → processing → ready states in real time, and then ask questions through a centered chat interface that returns cited answers with confidence scores.

The application also includes a slide-in documents drawer, a dark mode, suggested prompts for the empty state, and a built-in evaluation dashboard that displays answer quality scores. Everything runs from a single uvicorn command.

The core retrieval pipeline—keyword search, knowledge graph expansion, and optional reranking—proved to be genuinely competitive with embedding-based approaches for the workloads it was tested against, while remaining significantly easier to understand, operate, and explain.

If you are building a document question-and-answer system and want something you can fully reason about, deploy with minimal infrastructure overhead, and actually debug when problems occur, a vectorless approach is a serious option worth considering.