Building a Semantic Search Engine From Scratch (and What Broke Along the Way)

I wanted to actually understand retrieval before I let myself touch RAG, so I built a small semantic search tool from the ground up. It runs entirely on my laptop, indexes a corpus of Wikipedia articles on AI/ML topics, and lets me search them in plain English instead of exact keywords. Here's what it does, how it works, and the handful of bugs that taught me more than the parts that worked on the first try.

The Problem Keyword Search Can't Solve

Say I want to know how transformers keep track of word order in a sentence. If I grep or Ctrl+F my way through a page on attention mechanisms looking for the word "order," I'll probably come up empty, because the actual answer lives under a completely different name: positional encodings. The query and the answer don't share vocabulary, so exact-match search fails even though the information is right there.

That's the entire reason semantic search exists. Instead of matching words, it matches meaning. Both the query and the documents get converted into vectors (long lists of numbers) that capture what the text is about, then the search finds which document vectors sit closest to the query vector in that space.

Why I Built This Before Touching RAG

The tempting thing would've been to jump straight into a full RAG pipeline: retrieval plus an LLM generating an answer on top. I deliberately didn't. If you build both halves at once and the final answer comes out wrong, you can't tell whether retrieval fetched the wrong chunk of text or the LLM misread a perfectly good chunk. Those are two completely different bugs with two completely different fixes.

So I scoped this down to retrieval. Given a query, return the most relevant passages with their sources and a similarity score, and nothing else. No generation involved. Once that layer is trustworthy on its own, a RAG system built on top of it inherits that trust instead of hiding new problems behind a fluent-sounding LLM answer. What this project taught me is that a lot of what people call "RAG quality" is retrieval quality wearing a disguise.

The Stack, and Why Each Piece

Everything runs locally, which I chose deliberately. For embeddings, I used nomic-embed-text through Ollama, a small (137M parameter) model that turns text into 768-dimensional vectors. It runs on CPU, needs no API key, costs nothing, and never sends data off the machine. The alternative, something like OpenAI's embedding API, is likely higher quality, but it's paid, rate-limited, and every chunk of text leaves your computer. For a learning project on public Wikipedia text, none of that tradeoff was worth it.

For storing and searching the vectors, I used Chroma, an embedded vector database that lives as files on disk, with no server process to manage. At 818 vectors total, this was comfortably the right tool. Heavier options like Qdrant or Pinecone assume you either want a managed cloud service or are running at a scale where the operational overhead pays for itself. Neither applied here.

For similarity, I used cosine similarity instead of raw Euclidean distance. Two embeddings on the same topic can end up with different magnitudes from length or phrasing differences alone, even when they mean almost the same thing. Cosine similarity looks at the angle between vectors rather than the distance, which lines up better with "do these mean the same thing" than raw distance does.

The corpus itself is 18 Wikipedia articles (transformers, backpropagation, attention, RNNs, and similar AI/ML topics), pulled through Wikipedia's own API since it's cleanly licensed and needs no API key. The whole CLI is built on Python's standard argparse. Nothing fancier needed for two subcommands, index and search.

How It Works, Step by Step

The indexing pipeline runs in four stages:

Load. Every article file gets read in, and the first couple of lines (title and source URL) get peeled off into metadata rather than embedded as text. Otherwise every chunk would start with nearly identical boilerplate, which would make unrelated articles look artificially similar.

Chunk. Long articles get split into overlapping windows of about 800 characters each, snapped to the nearest whitespace so no chunk cuts a word in half. Consecutive chunks overlap by about 150 characters, so a fact sitting right on a chunk boundary still survives intact in at least one neighboring chunk.

Embed. Each chunk gets converted into a 768-dimensional vector via the local Ollama model, sent in batches of 32 to cut down on HTTP overhead. Under the hood this is the same idea covered in earlier posts on how transformers work: tokenize, run through attention layers, then compress the result into a single fixed-length vector.

Store. Every vector gets written into Chroma along with its original text and metadata (source article, title, URL, chunk position), using a deterministic ID built from the filename and chunk index. That naming matters because re-running the indexer overwrites existing entries instead of creating duplicates and we don't want duplicates else we will get multiple same results when quering.

Searching is simpler. The query gets embedded (about 200ms, almost entirely spent on that one embedding call), Chroma returns its nearest neighbors in milliseconds, and the CLI prints the ranked results with similarity scores and sources attached.

The Bugs That Taught Me Something

Silent truncation past the token limit. This one fails silently, which is what makes it scary. nomic-embed-text has an 8,192 token limit, and feeding it something longer doesn't throw an error. It quietly ignores everything past the cutoff. I proved this by embedding a huge document and comparing the vector for the full text against the vector for its first half. They came back mathematically identical. Everything after the token limit had been thrown away without a single warning. The fix was structural. Every chunk stays comfortably under the limit in the first place.

Meaning dilution in whole-document embeddings. I tested the query "how do transformers encode word position" against a properly chunked passage versus the embedding of an entire article page. The chunk scored 0.78. The whole page scored 0.57. Averaging an entire article into one vector leaves it weakly similar to everything the article touches on and strongly similar to nothing specific. That's the empirical case for chunking.

Wikipedia's math notation poisoning the embeddings. Raw Wikipedia extracts are full of LaTeX math markup, and some of it renders as genuinely bizarre artifacts, entire equations exploded into one character per line. Some chunks ended up almost entirely fragments like stray partial derivative symbols with no surrounding context, wasting a whole chunk on math notation. A naive regex to strip this out doesn't work because the braces nest (something like \mathrm{Attention} sitting inside a larger wrapped expression), so a lazy match stops at the first closing brace and leaves the rest mangled. The fix was a balanced-brace scanner that tracks nesting depth character by character, plus a second pass merging fragments broken across blank lines. Cleaning the corpus this way shrank it by about 28 percent and made the resulting chunks far more readable.

Silent garbage from a model mismatch. If you index with one embedding model and search with a different one, nothing crashes. You get nonsense results, because the two models don't share a coordinate system. Vectors from one model are meaningless next to vectors from another. So I exposed the model choice explicitly on both the index and search commands, turning a mismatch into a deliberate flag rather than a hidden default.

Wikipedia's rate limiting during the initial fetch. Pulling 18 articles back to back occasionally got rate limited, so the fetch script uses exponential backoff on failed requests, a short pause between calls, and per-article error handling so one bad title doesn't take down the whole run.

What the Chunk Size Experiment Showed

The biggest open design question was chunk size, so instead of guessing I measured it directly. I built three separate indexes of the same corpus at 300, 800, and 1200 character chunks and ran the same five test queries against all three.

The pattern that came back was useful beyond the academic. Smaller chunks (300 chars) consistently scored higher on precise factual queries like "who invented backpropagation," because the answer sits tightly packed with little surrounding dilution. But those same small chunks sometimes returned a fact stripped of the context needed to understand it. Larger chunks (1200 chars) did better on broader conceptual queries like explaining why RNNs suffer from vanishing gradients, because the reasoning behind an answer often needs more surrounding text than a single fact allows. There wasn't one "correct" chunk size. The right choice depends on whether your queries lean toward pinpoint facts or conceptual explanations.

Scores fell into a useful pattern of their own. Relevance above roughly 0.75 reliably meant a genuinely good match, 0.55 to 0.70 meant "topically related, worth a second look," and when every result for a query capped out around 0.50 or below, the corpus simply didn't contain a good answer. That ceiling is the signal a future RAG layer needs to say "I don't have a good source for this" instead of answering confidently from a weak match.

A Couple of Code Bits Worth Sharing

The chunker's core logic, snapping window boundaries to whitespace and computing overlap from the actual snapped position rather than the raw target position:

snap_end = normalized_text.rfind(" ", window_start, tentative_end)
if snap_end == -1:
    snap_end = tentative_end
next_window_start = max(snap_end + 1 - overlap, window_start + 1)
window_start = next_window_start

Three small decisions packed into four lines: never cut mid-word, calculate the overlap from where the chunk ended rather than where it was aimed to end, and always guarantee forward progress so the loop can't get stuck.

And one sanity check that demystified what a vector database does under the hood. It's brute-force nearest neighbor search in three lines of plain Python:

scored = [(cosine(query_vector, v), s) for v, s in zip(stored_vectors, stored_sources)]
scored.sort(reverse=True)

Run against all 818 stored vectors directly, this produced the same top results, in the same order, as Chroma's index. The fancy indexing structures underneath a vector database aren't doing anything conceptually different. They're an optimization for when brute-force comparison gets too slow at scale. At under a thousand vectors, brute force and the "real" index are indistinguishable.

Where I'd Take This Next

A few concrete next steps if I keep building on this: adding an explicit minimum-score cutoff so "no good match in the corpus" becomes a real signal instead of a weak result returned quietly, grouping results by source article instead of flat chunk ranking so one article can't crowd out others, running the same chunk-size experiment against a different embedding model to see if the pattern holds, and replacing character-based chunking with something sentence-aware for cleaner boundaries. The other honest gap is evaluation. Right now I'm eyeballing scores rather than tracking retrieval accuracy against a labeled set of test queries, and that's what would make future changes to chunk size or model choice measurable instead of a vibe check.

Full code is up on GitHub: github.com/codeSpicer/semantic-search