Lab build, 2026
Runbooks that answer back, with a scorecard
Retrieval over runbooks and architecture decision records, graded by golden questions and an LLM judge on every pull request.
The idea
Teams write runbooks and architecture decision records and then nobody can find them at the moment they matter. Retrieval-augmented generation is the obvious answer, and the obvious failure is shipping it without knowing whether it retrieves the right thing.
The design
Documents are chunked by heading so each chunk keeps its context, embedded, and indexed in Azure AI Search with hybrid keyword and vector retrieval plus reranking. A Semantic Kernel service answers questions with citations to the exact section it used, and says so when the documents do not contain an answer.
The scorecard
A golden set of real questions with known source sections runs in GitHub Actions on every change to prompts, chunking or retrieval settings. Retrieval is scored on whether the right section was found; answers are scored by an LLM judge for faithfulness to the cited text. Traces go to Langfuse, so a regression in the scorecard points straight to the retrieval call that caused it.
This is a lab build: a working design I am developing in the open, not a client system.