Add semantic search to a website

By Ryze AI Team

This tutorial adds meaning-based search to a static site. Users can find pages even when their query does not match the exact keywords.

1. Embed your content

At build time, run each page through an embedding model such as all-MiniLM-L6-v2 and store the vectors. In Python:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode([page_text for page_text in pages])
# save embeddings and page metadata to a JSON file

2. Store and ship

For small sites, store vectors in a JSON file shipped with the site. For larger sites, use a vector database or Supabase pgvector and query via an API. Keep the embedding model consistent between indexing and querying.

3. Query in the browser

Load the same embedding model in the browser with transformers.js, embed the query, and compute cosine similarity against stored vectors:

import { pipeline } from '@xenova/transformers';
const embedder = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
const q = await embedder(query, { pooling: 'mean', normalize: true });
// cosine similarity = dot product of normalized vectors

4. Hybrid search

Combine semantic results with keyword search using a library like Fuse.js. Merge the two result lists with reciprocal rank fusion. This covers exact terms that embeddings miss, such as product names and codes.

5. UI and performance

Debounce input by 200 to 300 milliseconds. Show results with highlighted snippets. Display a "no results" state. Cache the model and embeddings so repeat searches are instant. For large corpora, query a backend instead of shipping all vectors to the browser.

Resources: Supabase pgvector, Fuse.js.