By Ryze AI Team
This tutorial adds meaning-based search to a static site. Users can find pages even when their query does not match the exact keywords.
At build time, run each page through an embedding model such as all-MiniLM-L6-v2 and store the vectors. In Python:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode([page_text for page_text in pages])
# save embeddings and page metadata to a JSON file
For small sites, store vectors in a JSON file shipped with the site. For larger sites, use a vector database or Supabase pgvector and query via an API. Keep the embedding model consistent between indexing and querying.
Load the same embedding model in the browser with transformers.js, embed the query, and compute cosine similarity against stored vectors:
import { pipeline } from '@xenova/transformers';
const embedder = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
const q = await embedder(query, { pooling: 'mean', normalize: true });
// cosine similarity = dot product of normalized vectors
Combine semantic results with keyword search using a library like Fuse.js. Merge the two result lists with reciprocal rank fusion. This covers exact terms that embeddings miss, such as product names and codes.
Debounce input by 200 to 300 milliseconds. Show results with highlighted snippets. Display a "no results" state. Cache the model and embeddings so repeat searches are instant. For large corpora, query a backend instead of shipping all vectors to the browser.
Resources: Supabase pgvector, Fuse.js.