Say "chatbot on our own data" in 2026 and a stack materialises by reflex: chunk the documents, embed the chunks, stand up a vector database, search by similarity, feed the model. That stack is real and sometimes right. But for the knowledge bases most businesses actually have (hundreds of documents, not millions) it is often the expensive way to get a worse answer.
What the vector stack really costs
The embedding pipeline is never just the database. It is a sync job that re-embeds documents when they change, a chunking strategy that silently decides what can ever be found together, an index to host and version, and a bill that starts before the first question is asked. And when retrieval goes wrong, it goes wrong opaquely: the wrong chunk came back because two vectors were close, and nobody can read a vector. Debugging similarity is squinting at cosine scores.
There is also a quieter trap: embeddings are married to the model that made them. Swap embedding models, because of price, deprecation, or quality, and the honest move is re-embedding the entire corpus.
The unfashionable alternative
Keyword search (the BM25 family, the algorithm behind decades of search engines) over documents that are simply kept well: one topic per file, clear headings, real titles. On a corpus of hundreds of documents it is
- free, in both infrastructure and API calls: there is nothing to host and nothing to embed;
- instant to update: save the file and the next search sees it;
- debuggable: when a document matches, the matching words are right there; when it doesn't, you can see why and fix the document, which is usually the actual problem;
- traceable: the unit of retrieval is a file with a name and a history, so "every answer comes with its source" stops being a feature and starts being a property.
The assistant on this site works exactly this way. Its knowledge, the services, the boundaries, the blog post you are reading now, is plain markdown files in the same repository as the code, found by keyword search and loaded on demand. There is no vector database anywhere in the building. When it quotes a post, the source is a file I can open in git.
When vectors do earn their keep
Honesty cuts the other way too. Embeddings win when the corpus is large enough that keywords collide, when questions are phrased nothing like the documents answer them, when you search across languages, or when the thing being searched isn't prose at all. At tens of thousands of documents the balance genuinely tips, and hybrid setups, keywords plus vectors, are a respectable middle.
The sizing question to ask before any architecture is picked: how many documents, how phrased are the questions, who maintains this in a year? If the answers are "hundreds", "like the documents", and "nobody full-time", start with the card catalog, not the math.
Unsure which side of that line your data falls on? Describe it to the assistant here: it will reason about it in public, sources attached.