Verbatim
Grounded answers with traceable sources
A documentation Q&A and support-assistance system that retrieves relevant internal knowledge, generates concise answers over that context, and structurally maps every citation back to the exact source section it came from.
- Role
- Developer Support Engineer
- Stack
- Next.js • React • TypeScript • PostgreSQL • Prisma • Auth.js • Tailwind
- AI & Retrieval
- OpenAI • Anthropic • Gemini • Deterministic keyword retrieval
The Problem
Generic documentation assistants have two related failure modes: an answer can sound plausible even when the underlying documentation only weakly supports it, and a citation shown alongside that answer can still be something the model loosely associated with the response rather than the actual source it was grounded in.
Verbatim was designed around a stronger constraint: an answer should be traceable to the exact source content used to produce it, and uncertainty should change the workflow rather than encourage the model to guess.
High enough confidence
Answer with traceable citations
Low confidence
Structured support-ticket draft
From Documents to Addressable Knowledge
Verbatim ingests two kinds of source material: Next.js MDX documentation pages and Markdown knowledge-base articles. The interesting problem isn't file upload — it's turning heterogeneous source files into stable, addressable chunks that retrieval can point back to precisely.
Route-First Identity for Docs
Documentation files often share generic filenames like page.mdx. Document identity is derived from the actual documentation route rather than the filename, so many identically named files don't collide.
Path-First Identity for KB Content
Knowledge-base articles don't have the same navigable-route model, so they use path-based identity instead.
Stable Section Anchors
Documentation chunks are split at H2 boundaries and assigned GitHub-style heading anchors, so retrieved knowledge resolves to an exact navigable section, not just a document.
Incremental Re-Ingestion
SHA-256 content hashes let unchanged documents skip chunk regeneration while changed content replaces only the relevant chunks. Ingestion runs synchronously — there's no background sync job.
Verbatim deliberately consumes portable document formats rather than coupling retrieval directly to a specific SaaS knowledge platform. A separate tool, Knowledge Exporter, can prepare Freshdesk and Confluence content as deterministic Markdown for downstream use — the two are complementary, not integrated: there's no shared database, live sync, or direct handoff between them today.
Retrieval Before Generation
Retrieval is completely deterministic — and it's keyword-based, on purpose. Query terms are tokenized, normalized, and expanded (plurals, synonyms), then scored against candidate chunks using term-frequency weighting, quoted-phrase matching, a proximity bonus, heading weighting, and square-root length normalization, scoped to the requesting workspace. The result is a ranked, top-K set of chunks — not a formal BM25 implementation, but a deliberate, inspectable weighted lexical scoring engine.
Retrieval stays inspectable: the system can show exactly which chunks ranked highest and why — not opaque model reasoning — became the context window.
Trust Lives Outside the Model
Three guarantees hold regardless of which model is answering:
Retrieval is deterministic
The model does not retrieve content.
Confidence is deterministic
The model is never asked whether it's confident. Confidence is computed from retrieval signals — score strength and distribution — not model self-assessment.
Citation resolution is deterministic
The model may emit markers like [1] and [2], but Verbatim doesn't trust the model to identify the source. The application maps each marker back to the retrieved chunk at that index and builds the citation from that chunk's stored route, anchor, and source path. If the model produces no valid markers, the system still returns citations derived from the top retrieved chunks — the response is never citation-less.
The model writes the prose. The application decides what counts as evidence.
When Confidence Is Low
Low confidence isn't an invitation for the model to improvise. Instead, it can route the outcome toward a structured support-ticket draft — title, summary, suggested next information, and citations — instead of an answer. Freshdesk submission is feature-flagged off, so the system stops at a draft rather than filing a ticket automatically.
When the system doesn't have enough evidence to answer confidently, it changes the task from “answer” to “help a human escalate.”
One Retrieval Core, Multiple Products
The retrieval and grounding layer isn't coupled to one chatbot UI. The same corpus and ranking logic support three different surfaces:
Documentation Q&A
Question in, grounded answer with citations, confidence, and suggested navigation out.
Support Reply Composer
Uses the same document corpus to generate grounded customer-support reply drafts.
Internal Pilot & Debugging Tools
Ingest content, inspect documents and chunks, test retrieval, inspect answers, and monitor usage and cost.
The embedded documentation widget is currently a UI mock, not a completed integration.
Multi-Provider AI Without Moving the Trust Boundary
Answer synthesis runs through OpenAI, Anthropic, or Google Gemini behind a shared provider interface, selectable by request parameter or environment configuration. Changing the model doesn't change retrieval, ranking, confidence, or citation mapping — those guarantees live outside the provider implementation entirely.
Model providers are interchangeable; retrieval and evidence rules are not.
LLM calls run under a timeout. If a call times out or fails, Verbatim can still return a deterministic response built directly from the highest-ranked source chunk, rather than failing the request outright.
The support reply composer currently calls OpenAI directly rather than through the shared provider abstraction.
Operating Verbatim: Observability & Access
Observability & Cost
Every query can record provider, model, token usage, retrieval latency, LLM latency, estimated cost, citations, and a question hash or preview — full question logging is configurable rather than unconditional.
Model quality is only one operational concern; latency, cost, retrieval behavior, and what gets logged all need to stay inspectable.
Access
Internal pilot tooling is protected through Google OAuth (via Auth.js) with JWT sessions, gated by middleware behind an email/domain allowlist. In-memory rate limiting applies per IP and workspace.
Results
- Built ingestion for MDX documentation and Markdown knowledge-base content.
- Created stable route/path-based document identities and section-addressable chunking.
- Implemented deterministic lexical retrieval and confidence scoring.
- Built structurally verified citation mapping back to retrieved source chunks.
- Implemented graceful non-LLM answer fallback.
- Added interchangeable OpenAI, Anthropic, and Gemini providers for answer synthesis.
- Built internal ingestion/retrieval/debugging interfaces.
- Built a grounded support-reply composer over the same knowledge corpus.
- Added per-query latency, token, provider, and cost observability.
- Added unit/integration coverage around chunking, scoring, confidence, and rate limiting.
Technical Summary
- Application
- Next.js • React • TypeScript • Tailwind
- Data
- PostgreSQL • Prisma
- Retrieval
- Deterministic lexical scoring • H2-aware chunking • Stable anchors • Content hashing
- AI
- OpenAI • Anthropic • Gemini
- Trust & Operations
- Deterministic confidence • Structural citations • Auth.js • Usage/cost observability
What I'd Build Next
Let’s Work Together
Interested in working together or have a question? Reach out and let’s talk!