Why Accuracy Lied: Lessons from Building an NLP System

For the past year I've been building and dogfooding Velar — a personal finance app that parses UPI and SMS transactions and automatically categorizes them. The first version was embarrassingly simple: a lookup table. Swiggy maps to Food. Netflix maps to Entertainment. Done.

It worked until I started accumulating twelve months of real transaction data. Real Indian UPI messages don't play by rules. They look like this:

"₹1200 paid"
"txn id 48291 success"
"Netflix electricity bill ₹1500"

No merchant. No context. Conflicting signals. The rule-based system collapsed outside its happy path immediately. So I decided to go full ML — and that's where things got interesting, then humbling, then finally useful.

Stage 1 — 97% and a False Sense of Progress

I engineered a dataset of 3,000+ samples from scratch — clean patterns, noisy variants, and adversarial inputs that deliberately tried to trip the model. I trained a TF-IDF + Logistic Regression classifier. I ran evaluation.

97% accuracy. I shipped it mentally. Celebrated internally. Then I fed it the messy inputs I'd accumulated over the past year.

The model was confident every single time. And wrong every single time.

The deeper problem was the Other class — transactions that are ambiguous or genuinely unclassifiable. It had 0% precision and 0% recall. The model never predicted it, not once. It had such a strong bias toward known merchant categories that it couldn't admit when it didn't know. That's a dangerous property in a finance app.

The root cause was label leakage baked into the dataset. "Swiggy" always meant Food. "Netflix" always meant Entertainment. The model wasn't learning intent — it was memorizing a lookup table with extra steps. Remove the merchant token and prediction collapses entirely.

Stage 2 — Hiding the Problem More Cleverly

I went back to the dataset. Added weak signals, semantic anchors, and hard samples designed to break merchant memorization. The numbers looked perfect again.

The model looked fixed. It wasn't. The dataset refinement had just made it better at hiding the failure — the implicit mappings were still there, just more subtle. I hadn't solved the problem. I'd made it harder to see.

Stage 2.5 — Forcing Honesty

I stripped everything deterministic. Probabilistic labels. Real noise injection. True ambiguity. Inputs that genuinely could belong to multiple categories. The model had nowhere to hide.

48% wasn't a regression — it was the truth finally surfacing. These were the actual failure modes, made visible:

Stage 3 — BERT Didn't Save Me

The obvious next move was to upgrade the model. I swapped TF-IDF for BERT — a state-of-the-art transformer pre-trained on billions of tokens.

Two percentage points. That result was the most important finding of the entire project.

The bottleneck wasn't the model. It was the information in the input.

"₹1200 paid" doesn't contain enough signal for any model to classify reliably. That's not an architecture problem — it's a data problem. No model, no matter how capable, breaks through that ceiling. Swapping in a better architecture when the inputs are information-poor is like buying a better calculator to solve an equation with missing variables.

The Redesign — From Model to System

I stopped asking "how do I improve the model?" and started asking "how do I build a reliable system?" Those are fundamentally different questions, and the second one has a better answer.

The insight was that not all inputs are created equal. Some transactions are highly deterministic — Swiggy in the merchant field is nearly always Food. Others are genuinely ambiguous and no classifier will ever get them right with certainty. Treating all inputs the same and routing everything through a single model was the wrong architecture from the start.

The answer was a hybrid pipeline:

That last step is the most important. The system now knows when to stay silent. A transaction with insufficient signal gets returned as unclassified rather than a confident wrong answer. In a finance app, silence is always better than a confidently incorrect category.

The Feedback Loop — Making It a Living System

After deploying the pipeline as a standalone inference service (integrated with a Node.js backend), the real unlock came from what followed deployment.

Every time a user recategorizes a transaction, that correction is stored. Those corrections become future training data. The model retrains on real usage patterns instead of synthetic datasets engineered to look realistic.

This is the shift from a model to a living system. The synthetic dataset I built in Stage 1 was a reasonable starting point — but the real dataset is the one users generate through actual behavior. No amount of adversarial engineering fully replicates the variety of 12 months of real transactions. Closing that loop was the most architecturally important decision in the whole project.

What This Project Actually Taught Me

On data

High performance on clean data is a warning, not a signal. If your test set doesn't reflect the full ugliness of production inputs — the noise, the ambiguity, the inputs that genuinely don't have a right answer — your metrics are lying. I spent two stages learning this the slow way.

On models

Swapping in a better model doesn't fix an information problem. BERT scoring 50% when TF-IDF scored 48% is not a failure of BERT — it's a ceiling set by the inputs themselves. Understanding which problem you're actually solving before reaching for a better architecture saves a lot of time.

On systems

A classifier that outputs nothing on uncertain inputs is more valuable than one that outputs confidently wrong answers. Confidence filtering isn't a weakness or an admission of failure — it's a deliberate design decision that makes the overall system more trustworthy.

On iteration

The real dataset isn't the one you engineer before launch. It's the one your users generate through real behavior. Build the correction loop from day one, not as an afterthought. Every user interaction is a labeled training example.

The Metric That Actually Matters

The shift that mattered most in this entire project wasn't architectural — it was conceptual. Moving from "does this model score well on my benchmark" to "does this system behave correctly under real conditions, including knowing when to stay silent" changed every decision that followed.

Accuracy is easy to optimize for. Reliability is harder. But reliability is the thing that makes a system actually useful to the person who depends on it.

High accuracy on synthetic data is easy. Reliable systems under ambiguity are hard. The difference is worth understanding before you ship.