Query & Embedding Intelligence Lab

1. The "Whole-Caption Dilution" Problem

If you feed a 250-word bird description into an embedding model to get one single vector, that vector represents the average meaning of the whole text. Specific visual details—like golden crown patch or rusty barbed wire—get diluted by the remaining 200 words talking about habitat, lighting, and wings. Searches for specific field marks end up scoring poorly.

2. Fine-Grained Semantic Chunking

Instead of 1 vector per photo, the pipeline decomposes each bird photo description into 10 to 19 distinct chunks (e.g. head, beak, wings, action, setting, lighting, color_profile). Each chunk gets its own 384-dimensional vector coordinate.

3. Section-Level MaxSim Scoring

When a search query is submitted, the engine computes the cosine similarity between the query vector and every chunk vector of a photo:

If the photo's setting chunk matches rusty barbed wire with a 0.88 similarity score, that single high match catapults the photo to the top of the search results!

4. Int8 Quantization & In-Memory Binary Buffer

With ~2,134 photos producing ~60,000 chunk vectors, storing 384 raw 32-bit floats would consume ~92 MB of RAM. The script quantizes every float vector into signed 8-bit integers (int8) and packs them into a single 7.64 MB binary buffer (chunk_vectors.bin). The Node.js server scans all 60,000 vectors in under 15 milliseconds with zero garbage collection overhead.

5. Curation & Embedding Diversity (≤ 0.72)

When mining discovery queries, raw LLM outputs often produce near-duplicate paraphrases (e.g. standing on a rock vs perched on a rock, cosine similarity 0.98). By calculating pairwise cosine distances with all-MiniLM-L6-v2 and enforcing a strict ≤ 0.72 ceiling, the engine guarantees every discovery chip reveals a truly distinct visual mark, posture, or environment.

6. Solving the Bi-Encoder "Color-Attribute Binding" Flaw (Stage 2 ONNX Cross-Encoder)

Pure bi-encoder models (all-MiniLM-L6-v2) embed the query and document chunks independently into fixed 384-d vectors. On compositional queries like red wings and body, high-frequency anatomical nouns (wings, body) dominate the vector direction. As a result, a black-and-white Bufflehead with detailed wing/body descriptions scored 0.618 in Stage 1 MaxSim, beating truly red birds like Northern Cardinal (0.565, Rank #14) and Scarlet Tanager (0.545, Rank #20).

To eliminate this flaw without sacrificing speed, the engine runs a Two-Stage Retrieval & Cross-Attention Reranking pipeline in ~15–28ms:

• Stage 1 Recall + Section Binding Check (evaluateSectionBinding): Scans all ~60,000 Int8 chunk vectors in ~5ms and checks whether the query pairs color words (red, yellow, black, etc.) with structured VLM anatomy keys (head, beak, body, wings, tail). Photos matching all queried color-section pairs receive a recall bonus (+0.25) so they are guaranteed to enter the Top-16 rerank pool, while mismatches receive a demotion multiplier (sectionMultiplier = 0.12x or 0.72x).
• Stage 2 ONNX Cross-Encoder (ms-marco-MiniLM-L-6-v2 / bge-reranker-base): Concatenates [CLS] Query [SEP] Passage [SEP] across the Top-16 candidates and performs full token-by-token cross-attention, promoting Northern Cardinal (+13 ranks to #1, score 0.865) and Scarlet Tanager (+18 ranks to #2, score 0.860) while crushing Bufflehead to 0.057.

7. Adaptive Passage Routing & Signed Int8 Fix: Why General Queries Like "prominent ear tufts" Require Dynamic Windowing

Because the ONNX Cross-Encoder runs with a compact 52-token sequence window to maintain ~15–25ms CPU latency, how the candidate passage is assembled in buildCrossEncoderPassage() is critical. Three subtle pitfalls were identified and solved:

8. Live Benchmark Verification Across Both Query Regimes

Click any row's Inspect Live ➔ button below to load that query into the Playground and inspect its live Stage 1 MaxSim, Stage 2 Cross-Encoder logits, and evaluated 52-token passages: