When you search applications and profiles in TechPrufer, you expect one coherent answer: which resume or role matches this keyword. We had the opposite — two pages, two tokenizers, shallow base-profile matching, and a debounced API call that only scanned tailored JSON. The fix was not “add embeddings” or “spin up pgvector.” It was a small, structure-aware lexical ranker we call BM25F-lite.
The problem
Our applications page and profiles page share a search box, but they did not share a brain. The client split queries on spaces with OR semantics; the server deep-search used plainto_tsquery with phrase/AND semantics. The main list on one page got deep resume matching; the sidebar on the other did — but not vice versa.
- Base profiles were searched via a thin
extractSearchableTermshelper that ignored bullet text, headlines, and summaries. - JD facets came from a hardcoded list (~18 languages, ~23 skills). Add
LangGraphorpgvectorto your profile and search would not see it. - Every keystroke on the list pages could trigger a network round-trip to scan tailored JSON in Postgres.
The solution track
Resumes in TechPrufer are already structured JSON (our Canonical Profile Model). That JSON is the document tree — we do not need PageIndex-style LLM navigation per query. Instead:
- On save, harvest typed terms from skills, tech stacks, bullets, boosted keywords, and metadata. Persist
searchTerms[]andsearchDocjsonb onmaster_profilesandapplications. - On search, score in-memory with one shared tokenizer and BM25F-lite field weights. Same engine powers the main list and the opposite sidebar on both pages.
- Facets (Languages, Frameworks, Cloud…) are built from each user's own skill categories — not a global hardcoded dictionary.
score(doc) = Σ IDF(term) × fieldWeight × matchQuality × evidenceBoost IDF(term) = ln(1 + (N − n_term + 0.5) / (n_term + 0.5))
Why generic Postgres FTS fails here
We already had a GIN index on to_tsvector('simple', tailoredJson::text). It is fast — but wrong for tech resumes:
| Term in profile | Generic tsvector | Problem |
|---|---|---|
| C++ | c | Collides with C language |
| Node.js | node, js | “js” matches everything |
| AWS (S3, SQS) | aws, s3, sqs | Loses parent/child grouping |
| A/B Testing | a, b, testing | Garbage tokens |
Our custom tokenizer preserves +, ., / inside tokens, expands parentheticals into nested searchable children, and applies a small alias map (js↔javascript, k8s↔kubernetes, golang↔go).
The ranking list — what actually matters
Classic BM25 assumes long prose: term frequency and document length matter. Resumes are the opposite — a skill listed once is as strong as listed twice. We use BM25F (fielded BM25): only which field a term lives in and how rare it is across the user's library.
| Tier | Weight | Source |
|---|---|---|
| S | 1.00 | Skills · Languages |
| A | 0.85 | Other skills, experience/project tech, bullet tags |
| B | 0.60 | Boosted keywords (from tailoring) |
| B | 0.55 | Headline, role, company, project names |
| C | 0.35 | Bullet text, summary, education |
Two multipliers refine quality: matchQuality (exact 1.0 → prefix 0.7 → trigram-fuzzy 0.5) and evidenceBoost (×1.15 when a skill is both listed and used in experience bullets or tech stacks).
A search that improves itself
There is no ML training step. The vocabulary is harvested from each user's own profileJson every time they save. Fill in a new skill category, add Structured Streaming or Bun, and the next save updates searchTerms, the facet tree, and ranking — automatically.
- No LLM at query time (or index time).
- No embeddings — deterministic lexical matching is enough at ≤35 docs per user.
- No hardcoded skill list — facets grow from real profile data.
- Better data → better search — the loop is: curate your resume JSON, search gets smarter.
Where it runs
Write path: buildSearchIndex(profile) runs in profiles.service and applications.service on create/update. Results land in Postgres as searchTerms text[] and searchDoc jsonb.
Read path: dashboard server components load persisted docs, build per-user corpus IDF stats once, and pass them to client components. Scoring is in-memory TypeScript — sub-millisecond for a typical library.
The tailor flow's /api/applications/search-resumes endpoint is unchanged for now; list pages no longer depend on it. One engine, one tokenizer, both pages, both sidebars.