Knowledge Graphs + LLMs: Beyond Vector Search

Vector search is the workhorse of RAG, but it has a structural weakness: it understands "semantic similarity," not "relationships."

Ask, "Who reports directly to Zhang San?" A vector database can only find document fragments that mention Zhang San; it struggles to return the exact relationship. Ask, "Which modules does this feature depend on?" Vector search retrieves text that mentions dependencies, not the actual dependency graph.

Knowledge Graphs fill this gap. This article explains how they work with LLMs and when they are worth adopting.

1. The Limits of Vector Search

At its core, vector search performs semantic-similarity matching. It excels at:

• Fuzzy semantics ("deployment" ≈ "going live").
• Working across languages and forms of expression.
• Finding "related" content.

But it struggles with:

• Multi-hop relationships. "Who is the manager of A's manager?" This requires traversing two hops along a relationship chain, which vectors cannot do.
• Structured constraints. "Customers in East China during Q3 2023 with sales above one million" combines exact conditions that vector retrieval handles poorly.
• Global views. "The structure of the entire product line" requires the graph as a whole, not isolated fragments.

These are precisely the strengths of knowledge graphs.

2. What Knowledge Graphs Add

Knowledge graphs structure knowledge as "entities + relationships":

(Zhang San)-[manages]->(Li Si)
(Li Si)-[belongs to]->(Sales Department)
(Sales Department)-[responsible for]->(East China)

They support:

• Multi-hop reasoning: Traverse N hops along a relationship chain (Zhang San → Li Si → Sales Department → East China).
• Exact queries: Apply structured conditions (WHERE region='East China' AND role='sales').
• Global structure: Examine the topology of the entire graph, such as product dependency trees and organizational structures.

Vectors handle "fuzzy relevance"; graphs handle "exact relationships." For complex knowledge-based question answering, combining them is more powerful than using either alone.

3. GraphRAG: Combining Graphs with Generation

The idea behind GraphRAG is to first build a knowledge graph from documents, combine graph queries with vector retrieval at query time, and feed both sets of results to the model during generation.

The process:

1. Build the graph: Use an LLM to extract entities + relationships from documents and store them in a graph database such as Neo4j, memgraph, or PG + Apache AGE.
2. Query: Decompose the user's question. Send structured parts to a graph query in Cypher and semantic parts to vector retrieval.
3. Fuse: Combine the exact results from the graph query with the relevant fragments returned by vector search as context.
4. Generate: Have the model answer using the fused context.

For example: "Which open-source components does our Product X use, and what is each component's license?"

• Graph query: (Product X)-[depends on]->(?component)-[license]->(?license) precisely retrieves the dependency chain and license.
• Vectors: Add semantic text about "considerations for these licenses."
• Fused answer: Precise and complete.

With vector search alone, retrieval is either incomplete because dependencies are missed or inaccurate because text that merely mentions licenses is treated as the answer.

4. When a Knowledge Graph Is Worthwhile

Knowledge graphs are not a free lunch: building, maintaining, and querying them all carry costs. Do not adopt one merely because it seems "advanced." They are worthwhile when:

1. Relationships are dense. Organizational structures, supply chains, product dependencies, and knowledge systems all make relationships between entities central.
2. Multi-hop reasoning is common. Users frequently ask chained questions in the form "A's B's C."
3. There are many structured constraints. Queries combine exact conditions such as time, region, and attribute filters.
4. A global view matters. You need to see the overall structure, not an isolated point.

They are not worthwhile for:

• Document-based question answering such as "What does this manual say?" Vector search is enough.
• Single-hop factual queries. Vectors + reranking are enough.
• Small datasets with simple relationships. That would be overkill.

In practice, vectors + reranking are sufficient for 90% of RAG scenarios. Graphs become worthwhile only for the 10% with dense relationships and multi-hop reasoning. Do not be swayed by claims that "GraphRAG is more advanced."

5. Practical Barriers to Adoption

The real barriers to adopting a graph are:

1. Graph construction costs. Extracting entities and relationships from unstructured documents requires an LLM plus human validation. Quality determines whether the graph is usable; incorrect extraction produces an incorrect graph.
2. Maintenance costs. When data changes, the graph must be updated. Incremental graph construction is more complicated than updating a vector index.
3. Query complexity. Cypher and graph querying require specialized skills and have a higher learning curve than SQL or vector retrieval.
4. Integration with the existing stack. Most teams already use PG and a vector database; adding a graph database creates additional operational work.

A pragmatic path: Start with PG + pgvector + Apache AGE, the graph extension for PG, on a small scale. Move to a dedicated graph database such as Neo4j only when the need is proven. Do not begin with a heavyweight Neo4j deployment.

6. A Lightweight Alternative: Use Structured Tables Instead of a Graph

Often, what appears to require a graph can be handled with structured relationship tables:

• A hierarchy such as "user → department → region" can be queried in PG using a few related tables + JOINs; no graph is required.
• Dependencies such as "product → component" can support multiple hops using relationship tables + recursive queries with PG's WITH RECURSIVE.

The value of a graph database lies in arbitrarily complex, network-shaped relationships with frequent multi-hop traversal. A relational database is sufficient for simple hierarchies, trees, and one-to-many relationships. Do not equate "relational data" with "requiring a graph database."

7. The Right Way to Implement Hybrid Retrieval

For production-grade complex knowledge-based question answering, the right solution is often a three-layer hybrid:

User question
  → Intent detection: structured query / semantic retrieval / both?
  → In parallel: graph query (exact relationships) + vectors (semantics) + keywords (exact terms)
  → Fusion (RRF / weighted)
  → Generation (with citations)

This is not a binary choice between vectors and graphs. Route each question type to the most suitable retrieval method, then fuse the results. That is the current leading approach to knowledge-based question answering.

Summary

Knowledge Graphs + LLMs are a powerful complement to vector-based RAG, not a replacement:

• Vectors: Fuzzy semantics and relevant retrieval, serving as the primary method for 90% of scenarios.
• Graphs: Exact relationships, multi-hop reasoning, and global structure for the 10% of scenarios with dense relationships.
• Hybrid retrieval: Routing by question type + fusion is the right solution for complex question answering.

Remember: Do not adopt a graph merely because it seems "advanced." First build a solid vector RAG system and confirm that relationship reasoning is genuinely the bottleneck, meaning users really are asking multi-hop questions. Only then should you add a graph, preferably starting with a lightweight PG-based approach. A graph is a powerful tool for cases where vectors are insufficient, not the default configuration for RAG. Its value comes from using it where it is genuinely needed.