While building a RAG chatbot from scratch and orchestrating the entire pipeline, I stumbled into a problem I had not really thought about before. We are taking a transformer-based model that has been trained on an absurdly diverse corpus of information and then deliberately restricting what it is allowed to know. It can suddenly only answer questions from a particular document, a particular knowledge base, or a particular domain. In a strange way, limiting the knowledge of an LLM turned out to be more interesting to me than giving it access to more knowledge.
Most of the time, prompt engineering and parameter tuning get you quite far. The interesting part appeared elsewhere. Since everything I was building was running CPU-only, every operation mattered. LLM calls are expensive. Embedding generation is expensive. Retrieval is expensive. If a user opens the chatbot and simply says, "Hey, how's it going?", it feels somewhat ridiculous to generate embeddings for that query, perform semantic search against a vector database, retrieve chunks from a knowledge base, construct a prompt, invoke an LLM, and finally return a response that effectively says, "I'm good, what's up?"
The regex trap
There are multiple ways of dealing with this. One option is introducing a lightweight model whose only job is intent classification. Another option is to ask the same LLM whether the incoming message is a greeting or an actual task. The problem with the second approach is that you have already paid for a model call before deciding whether you need a model call. If it turns out to be a task, you then go ahead and perform retrieval, fetch history, build the full prompt, and invoke the model again. It works, but it feels wasteful.
The cleanest solution ended up being the least sophisticated one. A regex layer sitting at the first point of contact with a dictionary of predefined responses. If a message matches one of the many ways people tend to greet each other, the request never enters the retrieval pipeline. No embeddings are generated. No vector search is performed. No LLM is called. The response is simply picked from a predefined set and returned immediately.
The regex solution worked surprisingly well, but after a point I realized I was solving a slightly different problem. A regex or dictionary lookup can tell me whether a greeting exists in the message. What I actually cared about was whether the query required retrieval. Those are not necessarily the same thing. A message like "hey" clearly does not need retrieval, but "hey, can you explain something?" absolutely does. The more edge cases I encountered, the more rules I found myself adding. First it was greetings. Then greetings followed by tasks. Then abbreviated greetings. Then conversational openers that still contained a request. At some point it felt like I was manually drawing a decision boundary using rules. The question was no longer "does this query contain a greeting?" but rather "does this query require retrieval?" and that started looking much more like a classification problem than a pattern-matching problem.
Classification over pattern matching
I was already leaning in that direction, but it took a teammate pointing it out during a discussion to connect the dots. A lightweight classification model felt like a textbook case for traditional machine learning. The challenge then became whether I could train a classifier that reliably distinguishes between queries that require retrieval and those that do not.
The training data was already hiding inside the document itself. Most technical documents come with a reasonably structured table of contents. A PDF parser can usually extract this hierarchy as a tree where each entry has an associated depth. At the highest level you have broad topic families. Under those families sit sections, commands, methodologies, processes, configurations, explanations, and everything else the document contains. Instead of treating the PDF as an unstructured blob of text, the hierarchy itself becomes useful information.
Pipeline Optimization: Standard vs Classifier-Gated
Training on the document hierarchy
One issue quickly appeared. Technical documents are rarely balanced. Some topic families may span hundreds of pages while others occupy only a few. If samples are selected randomly, the resulting dataset becomes dominated by whichever family happens to contain the most content. Rather than generating synthetic samples to compensate for this imbalance, I experimented with family-based sampling. Sections were grouped according to their top-level topic family and then sampled evenly across families. The goal was not to maximize the number of samples but to maximize coverage of the document's vocabulary and concepts.
Once the topic families were available, I used Ollama to generate synthetic user queries. For every topic, the model generated a mixture of direct questions, explanatory requests, procedural questions, and goal-oriented queries. I also asked it to generate variants that began with greetings. Queries such as "Hey, can you explain X?" or "Hi, how do I configure Y?" became useful positive examples because they forced the classifier to learn that a greeting at the beginning of a message does not automatically make the message itself a greeting.
Even then, people are remarkably creative when it comes to saying hello. To make the model more robust, I generated additional greeting-task combinations by prepending diverse greeting patterns to randomly selected task queries while preserving the original task labels. Over time, the classifier stopped treating the greeting as the signal and started focusing on whether there was an actual task hidden inside the message.
The real optimization
The final model itself was not particularly fancy. TF-IDF features with unigrams and bigrams, sublinear term frequency scaling, and logistic regression with balanced class weights were enough to achieve a surprisingly strong F1 score. The interesting part was not the model. It was realizing that while building a system around an LLM, one of the most useful components ended up being a fairly traditional machine learning classifier whose only purpose was deciding when the LLM was actually unnecessary.
The most interesting optimization in the pipeline turned out not to be retrieval, prompting, or generation.
It was deciding when none of them were needed.