From Query Understanding to Knowledge-Driven Recommendations

In 2022, I wrote a technical proposal for a new tax and finance product. It covered query understanding, recommendation systems, and the fragmentation of domain knowledge. Like many architecture documents of its time, it also contained a long inventory of techniques: tokenization, intent classification, collaborative filtering, ranking models, entity extraction, and knowledge graphs.

Most of those model names are no longer the interesting part. What has survived is a priority decision I made at the beginning:

Structured domain knowledge > query understanding > recommendation ranking.

Ranking is not unimportant. The point is that in a regulated, time-sensitive domain, a system's ceiling is first determined by whether it can represent the knowledge correctly, then by whether it can understand the user's actual situation, and only then by how well it orders the candidates.

If the knowledge base is incomplete, perfect query understanding still retrieves incomplete answers. If the system misses jurisdiction, entity type, or effective date, a more confident model can become more dangerous. If those foundations are weak, a sophisticated ranker merely amplifies errors with greater precision.

Why tax and finance search is unusually difficult

General search often competes on relevance. Tax and finance search must satisfy several constraints at once.

Queries are underspecified. “Can a small business deduct this?” is not a complete business question. The legal entity, jurisdiction, transaction type, invoice type, and effective date may all change the answer.

Domain language is dense. A concept may have a statutory name, a professional abbreviation, an old name, and several everyday expressions. Words that look similar in general language may carry sharply different accounting or legal meanings.

Answers expire. Tax knowledge is not a static encyclopedia. Rules are issued, amended, superseded, and interpreted differently across jurisdictions. A relevant but expired result is not merely low quality; it is wrong.

Ranking is multi-objective. Users need authoritative source material, understandable explanations, and actionable guidance. The system must balance authority, relevance, readability, recency, and context.

Every conclusion needs provenance. A useful system may say that it lacks enough information. It should not produce a definitive answer that cannot identify its evidence and scope.

This is why a domain search product is not simply a vector database with an LLM on top. It is a decision system spanning knowledge governance, query interpretation, retrieval, ranking, and risk control.

Query understanding should produce a contract

Query understanding is traditionally divided into normalization, tokenization, correction, expansion, entity recognition, and intent classification. These capabilities still matter, but a production system needs a stable, auditable output that every downstream component can consume.

For example:

{
  "rawQuery": "How does a small business in Chongqing file VAT this year?",
  "normalizedQuery": "Chongqing small business 2026 VAT filing",
  "intent": "tax_policy_and_procedure",
  "entities": {
    "region": "Chongqing",
    "taxType": "VAT",
    "businessType": "small_business"
  },
  "constraints": {
    "effectiveAt": "2026-08-06",
    "authorityLevel": ["national", "municipal"]
  },
  "rewrites": [
    "Chongqing VAT relief for small businesses",
    "Chongqing VAT filing procedure for small businesses"
  ],
  "confidence": 0.91,
  "needsClarification": false
}

This object is more useful than an isolated label such as policy_lookup. Retrieval, filters, rerankers, answer generation, logging, and evaluation can share the same contract. When the system fails, the contract also helps reveal where it failed.

If a decisive constraint is missing, the right behavior is not to guess. A question about deducting the tax on a company vehicle may require the entity type, business purpose, invoice type, and transaction date. Asking one good clarifying question is safer than generating a polished answer to the wrong problem.

Replace algorithm inventories with an observable pipeline

My 2022 design used a Pipe-Filter architecture: a stable core moved data through plug-in algorithm components. The idea remains useful, but each stage now needs an explicit contract, evaluation method, trace, and fallback.

A production pipeline can be organized as follows:

1. Normalize input: characters, units, dates, regulation identifiers, jurisdiction aliases, and entity types.
2. Detect safety and scope: sensitive input, unauthorized requests, and questions the product cannot answer.
3. Parse intent and entities: task, business entities, temporal constraints, and jurisdiction.
4. Rewrite and decompose: correct errors, map terminology, and split complex questions into verifiable subqueries.
5. Retrieve through multiple channels: lexical, semantic, rule-based, graph-based, and behavioral retrieval.
6. Fuse and deduplicate: combine channels without letting one scoring system dominate.
7. Rerank under business constraints: semantic relevance, authority, validity, and user context.
8. Validate confidence: evidence coverage, policy status, citation consistency, and conflicting sources.
9. Present the result: answer, evidence, applicable scope, uncertainty, and next action.

Every stage should record its version, latency, input and output summaries, and failure reason. When a model times out, the system should degrade to lexical search and deterministic filters rather than taking the entire experience down.

Do not choose between lexical and vector retrieval

Domain search is naturally hybrid.

Lexical retrieval is excellent for regulation identifiers, technical terms, exact phrases, and numeric conditions. BM25 remains a strong baseline; embeddings do not make exact matching obsolete.

Vector retrieval handles conversational language and semantic similarity. A user may ask whether a hotel invoice from an employee trip is deductible while the source document discusses “input VAT deduction for accommodation services.” The expressions differ, but the underlying intent is close.

Rule-based retrieval enforces constraints that must not be approximated: jurisdiction, effective date, entity eligibility, and policy status.

Knowledge-relation retrieval follows connections such as policy, tax type, legal entity, transaction, required document, and filing procedure.

Behavioral retrieval becomes valuable only after sufficient, compliant, and de-biased feedback exists. Clicks are not proof of correctness; long dwell time may simply mean that the content is hard to understand.

The channels should not be concatenated blindly. A stable rank-fusion method can combine candidates before a reranker handles the top set. Fusion optimizes for coverage, reranking for order, and deterministic business rules protect boundaries that relevance must never override.

Recommendation is a decision process, not a model

The older proposal summarized industrial recommendation as “algorithm + recall + rank.” I would now make the stages explicit:

Candidate generation
  → eligibility filtering
  → deduplication and diversity
  → coarse ranking
  → fine reranking
  → policy constraints
  → explanation and presentation

Candidate generation answers “what might apply?” Filtering answers “what definitely does not apply?” Ranking decides “what should the user see first?” Explanation answers “why is this being shown?”

For tax and finance content, a conceptual score should include more than semantic relevance:

FinalScore =
  SemanticRelevance
  + AuthorityWeight
  + FreshnessWeight
  + ContextMatch
  + EvidenceCoverage
  - ExpirationPenalty
  - ConflictPenalty

This is not a universal formula. It is a design reminder: a semantically perfect result must still be rejected if it is expired, inapplicable to the jurisdiction, or unsupported by an authoritative source.

The knowledge layer sets the long-term ceiling

Knowledge fragmentation was the highest-priority problem in the original proposal. It matters even more now.

Acquire with provenance

Ingest regulations, official interpretations, procedural guides, cases, and internal material while preserving the source, publisher, publication date, effective period, and canonical URL.

Structure only what affects decisions

Do not begin with an enormous ontology. Start with the fields that determine applicability:

Policy → issuing authority
Policy → effective and expiration dates
Policy → jurisdiction
Policy → eligible entity
Policy → tax type
Policy → business event
Policy → supersedes or cites another policy

Resolve identity and conflict

The same policy may be mirrored, summarized, or incorrectly interpreted by multiple sources. The system must align entities, merge versions, apply authority rules, and expose conflicts instead of passing duplicate text to a model and hoping it decides correctly.

Evaluate and update incrementally

Knowledge needs a computable health state: source authority, current validity, field completeness, conflicts, and last review time. A policy update should trigger incremental ingestion and reevaluation of affected answers.

Where LLMs belong

LLMs have dramatically improved query interpretation, semantic rewriting, decomposition, and answer composition. They should not become an unconstrained center of the architecture.

Good uses include:

• extracting structured intent and entities from conversational queries;
• generating alternative retrieval expressions;
• decomposing complex questions into verifiable subproblems;
• semantically reranking retrieved evidence;
• composing an answer from confirmed sources;
• identifying conflicts and escalating them for review.

LLMs should not be the sole mechanism for:

• determining whether a regulation is currently valid;
• enforcing access control and data isolation;
• authorizing high-risk business actions;
• inventing missing facts when evidence is absent;
• replacing deterministic jurisdiction, date, and eligibility filters.

The working rule is simple: let models handle ambiguity and systems handle certainty. Let models propose judgments, and let evidence and rules constrain them.

Evaluation compounds faster than model selection

Without an evaluation set, architecture work eventually becomes subjective tuning against a few impressive demo queries.

Build a layered evaluation set from real user questions:

The set must include ambiguous language, typos, multiple intents, cross-jurisdiction conflicts, expired rules, and contradictory sources. These cases—not the clean demos—determine whether a system is ready for production.

How I would start today

I would not begin by training a model or drawing an architecture with every known algorithm. I would proceed in this order:

1. Select one bounded tax task and define correct outcomes and unacceptable errors.
2. Build a minimal knowledge model around provenance, time, jurisdiction, entity, and policy relationships.
3. Establish a hybrid BM25 and vector retrieval baseline.
4. parse each query into a stable contract and clarify low-confidence cases.
5. Add reranking, citations, and deterministic validation.
6. Turn real failures into an evaluation set with weekly regression runs.
7. Add knowledge graphs, personalization, or multi-agent orchestration only when the baseline reveals a concrete bottleneck.

Looking back at the 2022 design, my conclusion is not that the old algorithms became useless. It is that architecture decisions should outlive model names.

A trustworthy domain system knows where its knowledge came from, to whom it applies, and when it is valid. It understands the problem the user is trying to solve. And when the evidence is insufficient, it knows when to stop.

Those properties still matter more than the name of the model.

Further reading

• Elasticsearch: Hybrid Search
• Google Cloud: About hybrid search