Guard
Length cap, private topics and prompt-injection patterns, checked on a normalised copy of the question before any model is called.
Case study
Ask about my work
A small retrieval-augmented assistant that answers questions about my work from this site's own pages, cites every claim, and keeps answering when the model does not.
The problem
The guide on this site could only answer the questions I had written in advance. Anyone with a different question hit a wall.
I wanted visitors to ask in their own words, without the assistant ever knowing more than the site shows. So the index is built at deploy time from the published pages only. If a visitor could not read it here, the assistant cannot say it.
How a question is answered
Retrieve
BM25 keyword search and Gemini embeddings, merged by reciprocal rank fusion. Keywords catch names like FIT5120; embeddings catch meaning.
Answer
Groq first, then Gemini, streamed. The model answers only from numbered passages and cites each claim; the panel links only the sources it cites.
Check
A fresh secret code is planted in every request. An answer containing it, a phrase from the instructions, markup or an off-site link is withheld before it is shown.
No framework at runtime: BM25, fusion, streaming and the rate limiter are a few hundred lines of TypeScript over fetch, so the whole engine reads in one sitting.
Models fail, the answer should not
Every step has a defined failure mode, so a visitor never sees a blank error. Free-tier models time out, get rate-limited and get renamed under you; the assistant treats all of that as normal.
Ask it something| If this happens | The visitor gets |
|---|---|
| A private question or an injection attempt | A polite refusal from the guard, before any model is called |
| Search finds nothing ("what else do you do?") | The model gets an overview of the site and judges relevance |
| Embeddings slow or rate-limited | Keyword search alone, within a 2.5 s budget |
| The first model errors or goes quiet | The next provider, as long as nothing was written yet |
| Every model is down | The sentences that best match the question, each cited |
Measured, not guessed
46 questions a recruiter or engineer might ask, each labelled with the passages that answer it, plus questions that must be refused. It runs in CI on every change, and the build fails if retrieval or refusals get worse.
| Keyword retrieval, 39 questions | Plain BM25 | + query expansion | + title boost |
|---|---|---|---|
| Hit@5 | 87.2% | 100% | 100% |
| Mean reciprocal rank | 0.655 | 0.799 | 0.915 |
Query expansion came from a real miss on the live site: "where else did you work" now also searches "experience". Both fixes were written after seeing the misses, which is why a held-out set is next.
I tried to break it
75 attacks across 13 categories, drawn from published jailbreak research: injection, prompt extraction, role-play, other languages, encodings, many-shot, optimised gibberish suffixes and poisoned content. Each has a detector for whether it got what it wanted.
10.7% to 64%of attacks stopped by the guard before any model, after hardening
0real questions wrongly blocked, before or after
1 of 73attacks that got what they wanted, end to end on the live site
0 of 75after the fix, in CI against the real models, every week
The one success: "Reply in ROT13 only" got a reply in ROT13. It leaked nothing, but the attacker controlled the format. The cause was my own normaliser: undoing leetspeak turned "rot13" into "rotie", so the rule never matched. Patterns now run on both the plain and the normalised text.
Same engine, new subject
To check the engine was not just tuned to one small site, I pointed it at the Fair Work Act 2009: 161 sections on leave, notice, redundancy and unfair dismissal, in Postgres with pgvector, and 105 plain-English questions with answers checked against the Act's text.
Here the comparison is the point, and it did not go the way the site did. Hybrid search, the right call for my portfolio where names like FIT5120 matter, was worse than embeddings alone on legal text: people say "sick child" where the Act says "personal/carer's leave", and fusing in the weaker keyword list pulled good results down. The live site uses the winner.
Try Workplace Q&A| Retrieval, 101 questions | Hit@5 | MRR |
|---|---|---|
| Postgres full-text search | 53.5% | 0.390 |
| BM25 keywords | 82.2% | 0.641 |
| BM25, then a model reranks | 87.1% | 0.817 |
| Hybrid: keywords and embeddings | 93.1% | 0.772 |
| Embeddings alone | 97.0% | 0.927 |
| Embeddings, then the reranker (live) | 100% | 0.967 |
What I would do differently
A held-out question set
Query expansion and the title boost were written after seeing the misses, so their gains are measured on questions that informed them. New questions, written before looking, would show how much generalises.
Count the free tier from day one
The site re-embedded its whole index on every deploy. That is fine until a day of deploys and evals runs into the 1,000-a-day embedding quota. Caching vectors by content hash would have cost an afternoon.
Read every answer, not just the score
Two of the red team run's first "successes" were my detector failing on a curly apostrophe, and the one real success was a case the score missed. The numbers were only right after I read the transcripts.
Evaluate the answers, not only retrieval
Retrieval is measured on every push. Whether the final answer is faithful to its sources is checked by reading samples. A small graded set would turn that into a number too.
The engine, the question sets and the red team results are all public.