Building and Evaluating a Local-First Research Assistant with GraphRAG and vero-eval
Author: Daniel Kliewer
Date: 2025-11-15
Tags: AI, GraphRAG, Local LLM, Neo4j, Ollama, vero-eval, Research Assistant, Knowledge Graph, RAG, AI Evaluation
Description: Complete technical guide to building a production-ready research assistant using GraphRAG, Neo4j knowledge graphs, Ollama local LLMs, and vero-eval evaluation framework for rigorous AI system testing.
---
# Building and Evaluating a Local-First Research Assistant with GraphRAG and vero-eval
*A comprehensive guide to creating a persona-driven AI assistant with rigorous evaluation using Neo4j, Ollama, and the vero-eval framework*
## Introduction: Why Local GraphRAG Matters for Research Workflows
If you're building AI-powered applications in 2025, you've likely hit two major pain points: **context limitations** and **lack of systematic evaluation**. Large Language Models are powerful, but they struggle with long-term memory and consistent performance across edge cases. Enter GraphRAG—a methodology that combines knowledge graphs with retrieval-augmented generation to give your AI genuine memory and contextual awareness.
In this guide, we'll build a **Local Research Assistant** that:
- Stores and retrieves research papers, notes, and conversations in a Neo4j knowledge graph
- Uses Ollama for completely local inference (no API costs, full privacy)
- Implements persona-driven responses that adapt based on RLHF feedback
- **Most importantly**: Measures performance rigorously using the [vero-eval framework](https://github.com/vero-labs-ai/vero-eval)
This isn't another "hello world" tutorial. We're building production-ready infrastructure that you can deploy for real research workflows, with proper testing and evaluation baked in from day one.
## Prerequisites and Starting Point
Before we dive in, you'll need:
**System Requirements:**
- Python 3.9+
- Node.js 18+
- Docker (for Neo4j)
- 16GB+ RAM recommended
**Core Technologies:**
- [Ollama](https://ollama.ai) for local LLM inference
- [Neo4j](https://neo4j.com) for graph database
- [vero-eval](https://github.com/vero-labs-ai/vero-eval) for evaluation
- Next.js + FastAPI (from the starter template)
**Clone the Starter Repository:**
```bash
git clone https://github.com/kliewerdaniel/chrisbot.git research-assistant
cd research-assistant
```
This gives us a solid foundation with the frontend, basic chat interface, and project structure already in place. We'll extend it to build our research-focused GraphRAG system.
## Part 1: Understanding the Architecture
Our Research Assistant follows the **PersonaGen architecture** pattern outlined by Daniel Kliewer, but applied to academic research workflows:
```
┌─────────────────────────────────────────────────────────┐
│ User Interface │
│ (Next.js Chat Interface) │
└────────────────────┬────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Reasoning Agent │
│ (Tool Calling + RLHF Threshold Logic) │
└────────────────────┬────────────────────────────────────┘
│
┌──────────┴──────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Neo4j Graph │ │ Ollama LLM │
│ RAG System │ │ (Mistral/Llama) │
│ │ │ │
│ • Papers │ │ • Generation │
│ • Authors │ │ • Embeddings │
│ • Concepts │ │ • Extraction │
│ • Citations │ │ │
└──────────────────┘ └──────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ vero-eval Framework │
│ • Test Dataset Generation │
│ • Retrieval Metrics (Precision, Recall, MRR) │
│ • Generation Metrics (Faithfulness, BERTScore) │
│ • Persona Stress Testing │
└─────────────────────────────────────────────────────────┘
```
**Key Insight**: The persona system adapts its behavior based on evaluation feedback. If vero-eval shows poor retrieval for technical queries, the RLHF thresholds adjust to require more context before responding.
## Part 2: Setting Up Neo4j GraphRAG
Neo4j is our memory layer. Following the [official Neo4j GenAI integration patterns](https://neo4j.com/docs/cypher-manual/current/genai-integrations/), we'll create a graph schema optimized for research.
### Installing Neo4j GraphRAG for Python
```bash
# Install the official Neo4j GraphRAG package
pip install neo4j-graphrag
# Install Ollama integration
pip install "neo4j-graphrag[ollama]"
# Start Neo4j (using Docker)
docker run \
--name research-neo4j \
-p 7474:7474 -p 7687:7687 \
-e NEO4J_AUTH=neo4j/research2025 \
-v $PWD/neo4j-data:/data \
neo4j:latest
```
### Defining the Research Knowledge Schema
Create `scripts/graph_schema.py`:
```python
from neo4j_graphrag import GraphSchema
from dataclasses import dataclass
@dataclass
class ResearchSchema(GraphSchema):
"""
Knowledge graph schema for research assistant.
Nodes:
- Paper: Research papers with metadata
- Author: Paper authors with affiliation
- Concept: Extracted key concepts/topics
- Note: User's research notes
- Question: User queries with context
Relationships:
- AUTHORED: Author -> Paper
- CITES: Paper -> Paper
- DISCUSSES: Paper -> Concept
- RELATES_TO: Concept -> Concept
- ANSWERS: Paper -> Question
"""
node_types = {
'Paper': {
'properties': ['title', 'abstract', 'year', 'doi', 'pdf_path'],
'embedding_property': 'abstract_embedding'
},
'Author': {
'properties': ['name', 'affiliation', 'h_index'],
'embedding_property': None
},
'Concept': {
'properties': ['name', 'definition', 'domain'],
'embedding_property': 'definition_embedding'
},
'Note': {
'properties': ['content', 'timestamp', 'tags'],
'embedding_property': 'content_embedding'
},
'Question': {
'properties': ['query', 'timestamp', 'answered'],
'embedding_property': 'query_embedding'
}
}
relationship_types = {
'AUTHORED': ('Author', 'Paper'),
'CITES': ('Paper', 'Paper'),
'DISCUSSES': ('Paper', 'Concept'),
'RELATES_TO': ('Concept', 'Concept'),
'ANSWERS': ('Paper', 'Question'),
'ANNOTATES': ('Note', 'Paper')
}
```
**Why this schema?** Research workflows have natural graph structures:
- Papers cite each other (transitive relationships)
- Concepts relate to multiple papers
- Authors collaborate across papers
- User notes connect to specific papers
This lets us traverse the graph to find: "What papers discussing transformer architectures were cited by papers on RAG systems after 2023?"
### Building the Graph Ingestion Pipeline
Create `scripts/ingest_research_data.py`:
```python
import ollama
from neo4j import GraphDatabase
from neo4j_graphrag import GraphRAG
from pathlib import Path
import PyPDF2
class ResearchGraphBuilder:
def __init__(self, neo4j_uri="bolt://localhost:7687",
neo4j_user="neo4j",
neo4j_password="research2025",
ollama_model="mistral"):
self.driver = GraphDatabase.driver(neo4j_uri,
auth=(neo4j_user, neo4j_password))
self.ollama_model = ollama_model
self.graph_rag = GraphRAG(self.driver)
def extract_paper_metadata(self, pdf_path: Path) -> dict:
"""Extract title, abstract, and key sections from PDF"""
with open(pdf_path, 'rb') as file:
reader = PyPDF2.PdfReader(file)
# Extract first 3 pages (usually contains abstract)
text = ""
for page in reader.pages[:3]:
text += page.extract_text()
# Use Ollama to extract structured metadata
prompt = f"""Extract from this research paper excerpt:
1. Title
2. Authors (list)
3. Abstract
4. Key concepts (5-7 main topics)
Text: {text[:4000]}
Return as JSON."""
response = ollama.generate(
model=self.ollama_model,
prompt=prompt,
format='json'
)
return json.loads(response['response'])
def create_paper_node(self, metadata: dict, pdf_path: Path):
"""Create Paper node with embeddings"""
# Generate embedding for abstract
abstract_embedding = ollama.embeddings(
model='nomic-embed-text',
prompt=metadata['abstract']
)['embedding']
with self.driver.session() as session:
session.run("""
CREATE (p:Paper {
title: $title,
abstract: $abstract,
year: $year,
pdf_path: $pdf_path,
abstract_embedding: $embedding
})
WITH p
UNWIND $authors AS author_name
MERGE (a:Author {name: author_name})
CREATE (a)-[:AUTHORED]->(p)
WITH p
UNWIND $concepts AS concept_name
MERGE (c:Concept {name: concept_name})
CREATE (p)-[:DISCUSSES]->(c)
""",
title=metadata['title'],
abstract=metadata['abstract'],
year=metadata.get('year', 2024),
pdf_path=str(pdf_path),
embedding=abstract_embedding,
authors=metadata['authors'],
concepts=metadata['concepts']
)
def ingest_directory(self, papers_dir: Path):
"""Ingest all PDFs in a directory"""
pdf_files = list(papers_dir.glob("*.pdf"))
print(f"Found {len(pdf_files)} papers to ingest...")
for pdf_path in pdf_files:
print(f"Processing: {pdf_path.name}")
try:
metadata = self.extract_paper_metadata(pdf_path)
self.create_paper_node(metadata, pdf_path)
print(f"✓ Ingested: {metadata['title']}")
except Exception as e:
print(f"✗ Failed {pdf_path.name}: {e}")
```
**Key Pattern**: We're using Ollama for both extraction (via `generate`) and embeddings (via `embeddings`). This keeps everything local. For production, you might cache embeddings in a vector index.
### Creating Vector Indexes for Hybrid Search
Following [Neo4j's GenAI integration guide](https://neo4j.com/docs/cypher-manual/current/genai-integrations/), we create vector indexes:
```python
def create_vector_indexes(self):
"""Create vector indexes for similarity search"""
with self.driver.session() as session:
# Abstract embeddings (4096 dimensions for nomic-embed-text)
session.run("""
CREATE VECTOR INDEX paper_abstracts IF NOT EXISTS
FOR (p:Paper)
ON p.abstract_embedding
OPTIONS {
indexConfig: {
`vector.dimensions`: 4096,
`vector.similarity_function`: 'cosine'
}
}
""")
# Concept embeddings
session.run("""
CREATE VECTOR INDEX concept_definitions IF NOT EXISTS
FOR (c:Concept)
ON c.definition_embedding
OPTIONS {
indexConfig: {
`vector.dimensions`: 4096,
`vector.similarity_function`: 'cosine'
}
}
""")
# Note embeddings
session.run("""
CREATE VECTOR INDEX note_contents IF NOT EXISTS
FOR (n:Note)
ON n.content_embedding
OPTIONS {
indexConfig: {
`vector.dimensions`: 4096,
`vector.similarity_function`: 'cosine'
}
}
""")
```
**Critical**: The dimension count (4096) must match your embedding model. `nomic-embed-text` uses 4096, but if you switch to `all-MiniLM-L6-v2`, you'd need 384.
## Part 3: Implementing Hybrid Retrieval
Now we implement the retrieval layer that combines vector similarity with graph traversal:
```python
class HybridRetriever:
def __init__(self, driver, ollama_model="mistral"):
self.driver = driver
self.ollama_model = ollama_model
def retrieve_context(self, query: str, limit: int = 5) -> list[dict]:
"""
Hybrid retrieval combining:
1. Vector similarity search
2. Graph traversal for related concepts
3. Citation network expansion
"""
# Generate query embedding
query_embedding = ollama.embeddings(
model='nomic-embed-text',
prompt=query
)['embedding']
with self.driver.session() as session:
# Vector similarity search
vector_results = session.run("""
CALL db.index.vector.queryNodes(
'paper_abstracts',
$limit,
$query_embedding
)
YIELD node, score
MATCH (node)<-[:AUTHORED]-(author:Author)
MATCH (node)-[:DISCUSSES]->(concept:Concept)
RETURN
node.title AS title,
node.abstract AS abstract,
node.year AS year,
score AS relevance_score,
collect(DISTINCT author.name) AS authors,
collect(DISTINCT concept.name) AS concepts,
'vector_search' AS retrieval_method
ORDER BY score DESC
""",
query_embedding=query_embedding,
limit=limit
).data()
# Graph traversal for cited papers
graph_results = []
if vector_results:
top_paper_title = vector_results[0]['title']
graph_results = session.run("""
MATCH (seed:Paper {title: $seed_title})
MATCH (seed)-[:CITES]->(cited:Paper)
MATCH (cited)<-[:AUTHORED]-(author:Author)
MATCH (cited)-[:DISCUSSES]->(concept:Concept)
WHERE any(c IN $query_concepts WHERE c IN collect(concept.name))
RETURN
cited.title AS title,
cited.abstract AS abstract,
cited.year AS year,
0.7 AS relevance_score,
collect(DISTINCT author.name) AS authors,
collect(DISTINCT concept.name) AS concepts,
'citation_traversal' AS retrieval_method
LIMIT $limit
""",
seed_title=top_paper_title,
query_concepts=self._extract_query_concepts(query),
limit=limit // 2
).data()
# Combine and deduplicate
all_results = vector_results + graph_results
seen_titles = set()
unique_results = []
for result in all_results:
if result['title'] not in seen_titles:
seen_titles.add(result['title'])
unique_results.append(result)
return sorted(unique_results,
key=lambda x: x['relevance_score'],
reverse=True)[:limit]
def _extract_query_concepts(self, query: str) -> list[str]:
"""Extract key concepts from query using LLM"""
response = ollama.generate(
model=self.ollama_model,
prompt=f"Extract 3-5 key technical concepts from this query: {query}. Return as comma-separated list.",
options={'temperature': 0.1}
)
return [c.strip() for c in response['response'].split(',')]
```
**Why hybrid?** Pure vector search might miss important papers that don't match semantically but are cited by relevant papers. Graph traversal captures these relationships.
## Part 4: The Reasoning Agent and Persona Layer
The reasoning agent decides when to query the graph and how to format responses based on RLHF-adjusted thresholds:
```python
# In scripts/reasoning_agent.py
import json
from pathlib import Path
class PersonaReasoningAgent:
def __init__(self, persona_config_path: Path = Path("data/persona.json")):
self.persona_config = self._load_persona(persona_config_path)
self.retriever = HybridRetriever(driver, ollama_model)
def _load_persona(self, config_path: Path) -> dict:
"""Load persona configuration with RLHF thresholds"""
with open(config_path) as f:
return json.load(f)
def should_retrieve_context(self, query: str) -> bool:
"""
Decide if we need to retrieve context based on:
1. Query complexity
2. RLHF confidence threshold
3. Recent retrieval success rate
"""
# Simple heuristic: technical terms or specific paper requests
technical_indicators = [
'paper', 'research', 'study', 'findings',
'method', 'algorithm', 'experiment', 'results'
]
needs_retrieval = any(term in query.lower()
for term in technical_indicators)
# Check RLHF threshold
confidence_threshold = self.persona_config['rlhf_thresholds']['retrieval_required']
# If recent queries had low-quality responses, lower threshold
if self.persona_config['recent_success_rate'] < 0.7:
confidence_threshold *= 0.8
return needs_retrieval or confidence_threshold > 0.5
def generate_response(self, query: str, chat_history: list = None) -> dict:
"""
Main orchestration logic:
1. Decide if retrieval needed
2. Retrieve context if necessary
3. Generate response with persona coloring
4. Grade output (RLHF scoring)
"""
# Step 1: Retrieval decision
needs_context = self.should_retrieve_context(query)
context_docs = []
if needs_context:
context_docs = self.retriever.retrieve_context(query, limit=5)
# Step 2: Format context for LLM
context_str = self._format_context(context_docs)
# Step 3: Generate with persona
system_prompt = self._build_persona_prompt(context_str)
response = ollama.generate(
model='mistral',
prompt=query,
system=system_prompt,
context=chat_history
)
# Step 4: RLHF grading
quality_grade = self._grade_response(query, response['response'], context_docs)
# Update RLHF thresholds based on grade
self._update_persona_thresholds(quality_grade)
return {
'response': response['response'],
'context_used': context_docs,
'quality_grade': quality_grade,
'retrieval_method': context_docs[0]['retrieval_method'] if context_docs else None
}
def _build_persona_prompt(self, context: str) -> str:
"""
Build system prompt from persona configuration.
This is the 'coloring' step mentioned in the architecture.
"""
base_template = self.persona_config['system_prompt_template']
# Insert context if available
if context:
base_template += f"\n\nRelevant Research Context:\n{context}"
# Add persona modifiers based on RLHF values
formality = self.persona_config['rlhf_thresholds']['formality_level']
if formality > 0.7:
base_template += "\n\nUse academic, formal language with proper citations."
else:
base_template += "\n\nExplain concepts clearly and conversationally."
return base_template
def _grade_response(self, query: str, response: str, context: list) -> float:
"""
RLHF grading: 0 (needs improvement) to 1 (excellent).
In production, this would be human feedback, but we start with heuristics.
"""
# Heuristic checks:
# 1. Did we use retrieved context?
used_context = any(
doc['title'].lower() in response.lower()
for doc in context
) if context else True
# 2. Is response substantive (not too short)?
is_substantive = len(response.split()) > 50
# 3. Does response directly address query?
query_terms = set(query.lower().split())
response_terms = set(response.lower().split())
overlap = len(query_terms & response_terms) / len(query_terms)
# Weighted score
score = (
0.4 * float(used_context) +
0.3 * float(is_substantive) +
0.3 * overlap
)
return min(1.0, score)
def _update_persona_thresholds(self, quality_grade: float):
"""
Update RLHF thresholds based on response quality.
This is the adaptive learning mechanism.
"""
# If grade < 0.5, we need more context
if quality_grade < 0.5:
self.persona_config['rlhf_thresholds']['retrieval_required'] += 0.05
else:
# Successful response, can relax threshold slightly
self.persona_config['rlhf_thresholds']['retrieval_required'] -= 0.02
# Clamp values
self.persona_config['rlhf_thresholds']['retrieval_required'] = max(
0.0,
min(1.0, self.persona_config['rlhf_thresholds']['retrieval_required'])
)
# Save updated config
with open("data/persona.json", 'w') as f:
json.dump(self.persona_config, f, indent=2)
```
**Key Insight**: The persona adapts over time. If vero-eval (which we'll integrate next) shows poor performance, these thresholds shift to require more evidence before responding.
## Part 5: Integrating vero-eval for Rigorous Testing
This is where the magic happens. **vero-eval** provides production-grade evaluation that goes far beyond simple accuracy metrics. It tests edge cases, persona stress scenarios, and real-world failure modes.
### Installing and Configuring vero-eval
```bash
# Install vero-eval
pip install vero-eval
# Initialize evaluation directory
mkdir -p evaluation/datasets evaluation/results
```
### Generating a Research-Specific Test Dataset
vero-eval can generate test datasets tailored to your domain:
```python
# evaluation/generate_test_dataset.py
from vero.test_dataset_generator import generate_and_save
from pathlib import Path
def generate_research_test_dataset():
"""
Generate challenging test queries for research assistant.
vero-eval creates persona-based edge cases automatically.
"""
# Point to your research papers directory
data_path = Path('data/research_papers')
# Define the use case
use_case = """
This is a research assistant that helps academics:
- Find relevant papers on specific topics
- Understand connections between research areas
- Get summaries of complex papers
- Discover citation networks
- Answer technical questions about methodologies
Edge cases to test:
- Queries about very recent papers (after knowledge cutoff)
- Multi-hop reasoning (papers that cite papers that discuss X)
- Ambiguous author names
- Requests for specific experimental results
- Cross-domain queries (e.g., physics papers relevant to biology)
"""
# Generate dataset with persona variations
generate_and_save(
data_path=str(data_path),
usecase=use_case,
save_path_dir='evaluation/datasets/research_assistant_v1',
n_queries=150, # Generate 150 test queries
# Persona variations
personas=[
{
'name': 'PhD Student',
'characteristics': 'Detail-oriented, asks follow-up questions, wants methodology details'
},
{
'name': 'Senior Researcher',
'characteristics': 'Broad queries, interested in connections, asks about citations'
},
{
'name': 'Industry Practitioner',
'characteristics': 'Practical focus, wants applicable results, less theory'
}
],
# vero-eval will use Ollama for generation
llm_provider='ollama',
model_name='mistral'
)
print("✓ Generated test dataset with persona variations")
print(" Check: evaluation/datasets/research_assistant_v1/")
if __name__ == "__main__":
generate_research_test_dataset()
```
**Run this:**
```bash
python evaluation/generate_test_dataset.py
```
This creates a JSON file with queries like:
```json
{
"query": "What papers discuss attention mechanisms in the context of graph neural networks published after 2022?",
"persona": "Senior Researcher",
"expected_characteristics": ["multi-hop", "temporal_constraint", "domain_crossing"],
"ground_truth_chunk_ids": ["paper_47", "paper_89", "paper_102"],
"complexity_score": 0.85
}
```
### Running the Evaluation Suite
Now we test our system against this dataset:
```python
# evaluation/run_evaluation.py
from vero.evaluator import Evaluator
from vero.metrics import (
PrecisionMetric, RecallMetric, SufficiencyMetric,
FaithfulnessMetric, BERTScoreMetric, RougeMetric,
MRRMetric, MAPMetric, NDCGMetric
)
from reasoning_agent import PersonaReasoningAgent
import json
def run_full_evaluation():
"""
Run comprehensive evaluation using vero-eval framework.
Tests both retrieval and generation quality.
"""
# Initialize our system
agent = PersonaReasoningAgent()
# Load test dataset
with open('evaluation/datasets/research_assistant_v1/queries.json') as f:
test_queries = json.load(f)
# Initialize vero-eval
evaluator = Evaluator(
test_dataset=test_queries,
trace_db_path='evaluation/trace.db' # Logs all queries
)
# Define evaluation metrics
retrieval_metrics = [
PrecisionMetric(k=5),
RecallMetric(k=5),
SufficiencyMetric(), # Are retrieved docs sufficient to answer?
]
generation_metrics = [
FaithfulnessMetric(), # Is response faithful to retrieved docs?
BERTScoreMetric(), # Semantic similarity to reference answers
RougeMetric() # Token overlap with references
]
ranking_metrics = [
MRRMetric(), # Mean Reciprocal Rank
MAPMetric(), # Mean Average Precision
NDCGMetric() # Normalized Discounted Cumulative Gain
]
results = {
'retrieval': {},
'generation': {},
'ranking': {},
'per_persona': {}
}
# Run evaluation for each query
for query_data in test_queries:
query = query_data['query']
persona = query_data['persona']
ground_truth = query_data['ground_truth_chunk_ids']
# Generate response using our system
response_data = agent.generate_response(query)
# Extract retrieved document IDs
retrieved_ids = [
doc.get('paper_id', doc['title'])
for doc in response_data['context_used']
]
# Log to vero-eval's trace database
evaluator.log_query(
query=query,
retrieved_docs=retrieved_ids,
generated_response=response_data['response'],
metadata={'persona': persona}
)
# Evaluate retrieval
for metric in retrieval_metrics:
score = metric.compute(
retrieved=retrieved_ids,
relevant=ground_truth
)
metric_name = metric.__class__.__name__
if metric_name not in results['retrieval']:
results['retrieval'][metric_name] = []
results['retrieval'][metric_name].append(score)
# Evaluate generation
for metric in generation_metrics:
score = metric.compute(
generated=response_data['response'],
reference=query_data.get('reference_answer', ''),
context=response_data['context_used']
)
metric_name = metric.__class__.__name__
if metric_name not in results['generation']:
results['generation'][metric_name] = []
results['generation'][metric_name].append(score)
# Track per-persona performance
if persona not in results['per_persona']:
results['per_persona'][persona] = {
'precision': [],
'faithfulness': []
}
results['per_persona'][persona]['precision'].append(
results['retrieval']['PrecisionMetric'][-1]
)
results['per_persona'][persona]['faithfulness'].append(
results['generation']['FaithfulnessMetric'][-1]
)
# Aggregate results
for category in ['retrieval', 'generation']:
for metric_name, scores in results[category].items():
results[category][metric_name] = {
'mean': sum(scores) / len(scores),
'min': min(scores),
'max': max(scores),
'std': np.std(scores)
}
# Save results
with open('evaluation/results/full_evaluation.json', 'w') as f:
json.dump(results, f, indent=2)
print("✓ Evaluation complete!")
print(f" Retrieval Precision@5: {results['retrieval']['PrecisionMetric']['mean']:.3f}")
print(f" Retrieval Recall@5: {results['retrieval']['RecallMetric']['mean']:.3f}")
print(f" Generation Faithfulness: {results['generation']['FaithfulnessMetric']['mean']:.3f}")
return results
if __name__ == "__main__":
results = run_full_evaluation()
```
**Run the evaluation:**
```bash
python evaluation/run_evaluation.py
```
### Generating Performance Reports
vero-eval includes a report generator:
```python
from vero.report import ReportGenerator
# Generate comprehensive HTML report
generator = ReportGenerator(
trace_db_path='evaluation/trace.db',
results_path='evaluation/results/full_evaluation.json'
)
generator.generate_report(
output_path='evaluation/results/performance_report.html',
include_sections=[
'executive_summary',
'retrieval_analysis',
'generation_analysis',
'persona_breakdown',
'failure_cases',
'recommendations'
]
)
print("✓ Report generated: evaluation/results/performance_report.html")
```
This creates an interactive HTML report showing:
- Overall metrics with confidence intervals
- Per-persona performance breakdown
- Failure case analysis (queries where system performed poorly)
- Recommendations for improvement
## Part 6: The RLHF Feedback Loop
Now we close the loop: use vero-eval results to update the persona's RLHF thresholds:
```python
# evaluation/update_persona_from_results.py
import json
def update_persona_thresholds(evaluation_results: dict):
"""
Analyze vero-eval results and adjust persona thresholds.
This is the core RLHF mechanism.
"""
# Load current persona config
with open('data/persona.json') as f:
persona_config = json.load(f)
# Analyze retrieval performance
retrieval_recall = evaluation_results['retrieval']['RecallMetric']['mean']
if retrieval_recall < 0.6:
# Low recall → need to retrieve more documents
persona_config['rlhf_thresholds']['retrieval_limit'] += 2
persona_config['rlhf_thresholds']['retrieval_required'] += 0.1
print("⚠️ Low recall detected. Increasing retrieval aggressiveness.")
# Analyze generation faithfulness
faithfulness = evaluation_results['generation']['FaithfulnessMetric']['mean']
if faithfulness < 0.7:
# Responses not faithful to sources → need stronger grounding
persona_config['rlhf_thresholds']['minimum_context_overlap'] = 0.4
persona_config['system_prompt_template'] += (
"\n\nIMPORTANT: Always cite specific papers when making claims. "
"Do not speculate beyond what the retrieved papers state."
)
print("⚠️ Low faithfulness detected. Strengthening citation requirements.")
# Per-persona adjustments
for persona_name, metrics in evaluation_results['per_persona'].items():
avg_precision = sum(metrics['precision']) / len(metrics['precision'])
if avg_precision < 0.5:
print(f"⚠️ {persona_name} persona underperforming (Precision: {avg_precision:.2f})")
# Could adjust persona-specific prompts here
# For now, log for manual review
# Save updated config
with open('data/persona.json', 'w') as f:
json.dump(persona_config, f, indent=2)
print("✓ Persona thresholds updated based on evaluation results")
# Usage after evaluation
with open('evaluation/results/full_evaluation.json') as f:
results = json.load(f)
update_persona_thresholds(results)
```
**The workflow becomes:**
1. Run system on test queries
2. vero-eval measures performance
3. Script analyzes metrics
4. Persona thresholds adjust automatically
5. Re-evaluate to confirm improvement
This is **reinforcement learning through human feedback** (RLHF) in action, but guided by rigorous automated evaluation rather than ad-hoc human ratings.
## Part 7: Integrating with the Frontend
Now we wire this into the Next.js chat interface. Update `src/app/api/chat/route.ts`:
```typescript
import { NextRequest } from 'next/server'
import { spawn } from 'child_process'
import path from 'path'
export async function POST(request: NextRequest) {
const { message, messages, graphRAG = true } = await request.json()
if (!graphRAG) {
// Regular chat without RAG
return handleRegularChat(message, messages)
}
// Call our Python reasoning agent
const agentPath = path.join(process.cwd(), 'scripts', 'reasoning_agent.py')
const result = await new Promise<{response: string, context: any[]}>((resolve, reject) => {
const pythonProcess = spawn('python3', [
agentPath,
'generate',
JSON.stringify({ query: message, chat_history: messages })
])
let stdout = ''
let stderr = ''
pythonProcess.stdout.on('data', (data) => {
stdout += data.toString()
})
pythonProcess.stderr.on('data', (data) => {
stderr += data.toString()
})
pythonProcess.on('close', (code) => {
if (code === 0) {
try {
const result = JSON.parse(stdout)
resolve(result)
} catch (e) {
reject(new Error(`Failed to parse response: ${e}`))
}
} else {
reject(new Error(`Agent failed: ${stderr}`))
}
})
})
// Stream response back to client
const stream = new ReadableStream({
start(controller) {
// Send response with context metadata
const formatted = `${result.response}\n\n---\n**Sources:**\n${
result.context.map((doc, i) =>
`[${i+1}] ${doc.title} (${doc.year})`
).join('\n')
}`
controller.enqueue(new TextEncoder().encode(formatted))
controller.close()
}
})
return new Response(stream, {
headers: {
'Content-Type': 'text/plain; charset=utf-8',
},
})
}
```
Update the chat UI to show retrieval metadata:
```typescript
// In src/components/Chat.tsx
{message.role === 'assistant' && message.context && (
"""
msg.attach(MIMEText(html_body, 'html'))
# Attach visualization
with open('evaluation/results/weekly_report.png', 'rb') as f:
img = MIMEImage(f.read())
img.add_header('Content-Disposition', 'attachment',
filename='weekly_trends.png')
msg.attach(img)
# Send email
with smtplib.SMTP('smtp.lab.edu', 587) as smtp:
smtp.starttls()
smtp.login('research-assistant@lab.edu', os.getenv('EMAIL_PASSWORD'))
smtp.send_message(msg)
print("✓ Weekly report sent to lab members")
if __name__ == "__main__":
generate_weekly_report()
```
## Conclusion: The Complete Picture
You now have everything needed to build, evaluate, and deploy a production-ready Research Assistant:
**Core Architecture:**
✅ Neo4j knowledge graph for research papers
✅ Ollama for local LLM inference
✅ Hybrid retrieval (vector + graph)
✅ Persona-driven responses with RLHF
**Evaluation & Quality:**
✅ vero-eval for rigorous testing
✅ Automated adversarial testing
✅ Continuous monitoring with alerts
✅ Weekly performance reports
**Production Features:**
✅ Caching for performance
✅ Batch processing for scale
✅ Automated paper updates
✅ Multi-persona support
**The vero-eval Advantage:**
What makes this system production-ready is the evaluation framework. Unlike traditional RAG systems that rely on gut feeling and spot-checking, we have:
1. **Systematic edge case testing** - adversarial queries expose weaknesses
2. **Persona stress testing** - ensures all user types are served well
3. **Automated regression detection** - alerts when quality degrades
4. **Actionable metrics** - precision/recall/faithfulness directly inform improvements
5. **Continuous learning** - RLHF loop closes based on real performance data
This is the difference between a demo and a system you'd trust with real research workflows.
**Next Steps:**
1. Clone the starter repo and follow the setup script
2. Ingest your first 100 papers to test the pipeline
3. Run vero-eval to establish your baseline
4. Iterate on retrieval and persona prompts
5. Deploy to staging and gather feedback
6. Use weekly reports to drive improvements
**Remember:** The goal isn't perfect accuracy on day one. It's building a system that measurably improves over time through evaluation-driven iteration.
Now go build something that makes research more efficient! 🚀
---
**Resources:**
- [Complete Code Repository](https://github.com/kliewerdaniel/chrisbot)
- [vero-eval Documentation](https://github.com/vero-labs-ai/vero-eval)
- [Neo4j GenAI Integration Guide](https://neo4j.com/docs/cypher-manual/current/genai-integrations/)
- [llama.cpp Guide](https://danielkliewer.com/blog/2025-11-12-mastering-llama-cpp-local-llm-integration-guide)
Questions? Open an issue in the repo or reach out to the community.