> ## Documentation Index > Fetch the complete documentation index at: https://sonamu.cartanova.ai/llms.txt > Use this file to discover all available pages before exploring further. # Hybrid Search > Combining vector search and keyword search in Sonamu for improved accuracy ## Vector Search Alone Isn't Enough You've built a product search API with Sonamu: ```typescript theme={null} @api({ httpMethod: 'POST' }) async searchProducts(query: string) { const embedding = await Embedding.embedOne(query, 'voyage', 'query'); const results = await this.getPuri().raw(` SELECT name, description, 1 - (embedding <=> ?) AS similarity FROM products ORDER BY embedding <=> ? LIMIT 10 `, [ JSON.stringify(embedding.embedding), JSON.stringify(embedding.embedding), ]); return results.rows; } ``` **Problem encountered**: When a user searches for "MacBook Pro 14": * "MacBook" - Found (semantically similar) * "MacBook Pro" - Found (Korean also works) * "**MBP14**" - Not found (**exact model name**) * "**SKU-12345**" - Not found (**product code**) **Limitations of vector search**: * Weak with exact product names, model names * Can't find unique identifiers like product codes, SKUs * Vulnerable to technical terms, abbreviations ## What is Hybrid Search? Combines **vector search** (semantics) + **full-text search/FTS** (keywords). ```mermaid theme={null} flowchart TD Q["Query: MacBook Pro 14"] Q --> V["Vector Search
(semantic understanding)"] Q --> F["Full-Text Search FTS
(keyword matching)"] V --> R1["1. MacBook Air: 0.85
2. MacBook Pro: 0.82
3. iPad Pro: 0.70"] F --> R2["1. MacBook Pro 14: 1.0
2. MacBook Pro 16: 0.95
3. Pro Display: 0.50"] R1 --> H["Hybrid Combination
(weighted average)"] R2 --> H H --> Final["Final Results:
1. MacBook Pro 14
2. MacBook Pro 16
3. MacBook Air"] ``` **Advantages**: * Vector: Semantic understanding, synonyms, typos * FTS: Exact keywords, partial matching * **Combined**: Best accuracy ## Implementation in Sonamu ### 1. PostgreSQL FTS Setup First, prepare full-text search (FTS). #### Add tsvector Column ```typescript theme={null} // migrations/20240101_add_fts.ts export async function up(knex: Knex): Promise { await knex.schema.table('products', (table) => { table.specificType('search_vector', 'tsvector'); }); // Generate initial data await knex.raw(` UPDATE products SET search_vector = setweight(to_tsvector('simple', coalesce(name, '')), 'A') || setweight(to_tsvector('simple', coalesce(description, '')), 'B') `); // GIN index await knex.raw(` CREATE INDEX idx_products_search ON products USING GIN (search_vector) `); } ``` **Key points**: * `'simple'`: Handles Korean + English * `setweight`: Higher weight for title (A) * GIN index: Fast search #### Auto-Update Trigger ```typescript theme={null} export async function up(knex: Knex): Promise { // Trigger function await knex.raw(` CREATE FUNCTION products_search_trigger() RETURNS trigger AS $$ BEGIN NEW.search_vector := setweight(to_tsvector('simple', coalesce(NEW.name, '')), 'A') || setweight(to_tsvector('simple', coalesce(NEW.description, '')), 'B'); RETURN NEW; END; $$ LANGUAGE plpgsql; `); // Create trigger await knex.raw(` CREATE TRIGGER tsvector_update BEFORE INSERT OR UPDATE ON products FOR EACH ROW EXECUTE FUNCTION products_search_trigger(); `); } ``` Now search\_vector is automatically updated when products are added/modified. ### 2. Hybrid Search in Sonamu Model ```typescript theme={null} import { BaseModelClass, api } from "sonamu"; import { Embedding } from "sonamu/vector"; class ProductModelClass extends BaseModelClass { @api({ httpMethod: 'POST' }) async hybridSearch( query: string, vectorWeight: number = 0.7, ftsWeight: number = 0.3, limit: number = 10 ) { // 1. Query embedding const embedding = await Embedding.embedOne(query, 'voyage', 'query'); // 2. Hybrid search SQL const results = await this.getPuri().raw(` WITH vector_results AS ( SELECT id, 1 - (embedding <=> ?) AS vector_score FROM products WHERE embedding IS NOT NULL ), fts_results AS ( SELECT id, ts_rank(search_vector, plainto_tsquery('simple', ?)) AS fts_score FROM products WHERE search_vector @@ plainto_tsquery('simple', ?) ) SELECT p.id, p.name, p.description, p.price, COALESCE(v.vector_score, 0) AS vector_score, COALESCE(f.fts_score, 0) AS fts_score, (COALESCE(v.vector_score, 0) * ? + COALESCE(f.fts_score, 0) * ?) AS hybrid_score FROM products p LEFT JOIN vector_results v ON p.id = v.id LEFT JOIN fts_results f ON p.id = f.id WHERE v.id IS NOT NULL OR f.id IS NOT NULL ORDER BY hybrid_score DESC LIMIT ? `, [ JSON.stringify(embedding.embedding), // vector query, // fts (1) query, // fts (2) vectorWeight, // 0.7 ftsWeight, // 0.3 limit, ]); return results.rows.map(row => ({ id: row.id, name: row.name, price: row.price, vectorScore: parseFloat(row.vector_score.toFixed(3)), ftsScore: parseFloat(row.fts_score.toFixed(3)), hybridScore: parseFloat(row.hybrid_score.toFixed(3)), })); } } ``` **SQL explanation**: 1. `vector_results`: Calculate vector similarity 2. `fts_results`: Calculate FTS score 3. `LEFT JOIN`: Include if either matches 4. Weighted average: `(vector * 0.7) + (FTS * 0.3)` ## Weight Strategies ### When to Use Which Weights? **Balanced** (default) ```typescript theme={null} vectorWeight: 0.7, ftsWeight: 0.3 ``` * **Use case**: General search * **Examples**: Blogs, documents, knowledge bases **Semantic-focused** ```typescript theme={null} vectorWeight: 0.9, ftsWeight: 0.1 ``` * **Use case**: When semantic understanding is important * **Examples**: Q\&A, customer support, recommendations **Keyword-focused** ```typescript theme={null} vectorWeight: 0.3, ftsWeight: 0.7 ``` * **Use case**: When exact matching is important * **Examples**: Product codes, model names, technical terms ### Dynamic Adjustment in Sonamu ```typescript theme={null} @api({ httpMethod: 'POST' }) async smartSearch(query: string, limit: number = 10) { // Auto-adjust based on query length let vectorWeight = 0.7; let ftsWeight = 0.3; if (query.length < 10) { // Short query: keyword priority // e.g., "MBP", "SKU-123" vectorWeight = 0.3; ftsWeight = 0.7; } else if (query.length > 50) { // Long query: semantic priority // e.g., "Please recommend a laptop..." vectorWeight = 0.8; ftsWeight = 0.2; } return await this.hybridSearch(query, vectorWeight, ftsWeight, limit); } ``` ## Practical Scenario ### Scenario: E-commerce Product Search You're building an online store with Sonamu. **Step 1: Prepare Tables** ```typescript theme={null} // migrations/20240101_products.ts export async function up(knex: Knex): Promise { await knex.schema.createTable('products', (table) => { table.increments('id').primary(); table.string('name').notNullable(); table.text('description'); table.string('sku').unique(); table.decimal('price', 10, 2); // Vector + FTS table.specificType('embedding', 'vector(1024)'); table.specificType('search_vector', 'tsvector'); table.timestamps(true, true); }); // Indexes await knex.raw(` CREATE INDEX idx_products_embedding ON products USING hnsw (embedding vector_cosine_ops) `); await knex.raw(` CREATE INDEX idx_products_search ON products USING GIN (search_vector) `); // FTS trigger await knex.raw(` CREATE FUNCTION products_search_trigger() RETURNS trigger AS $$ BEGIN NEW.search_vector := setweight(to_tsvector('simple', coalesce(NEW.name, '')), 'A') || setweight(to_tsvector('simple', coalesce(NEW.description, '')), 'B') || setweight(to_tsvector('simple', coalesce(NEW.sku, '')), 'A'); RETURN NEW; END; $$ LANGUAGE plpgsql; CREATE TRIGGER tsvector_update BEFORE INSERT OR UPDATE ON products FOR EACH ROW EXECUTE FUNCTION products_search_trigger(); `); } ``` **Step 2: Add Product API** ```typescript theme={null} @api({ httpMethod: 'POST' }) async addProduct( name: string, description: string, sku: string, price: number ) { // Generate embedding const embedding = await Embedding.embedOne( `${name}\n\n${description}`, 'voyage', 'document' ); // Save (search_vector is auto-generated by trigger) const product = await this.saveOne({ name, description, sku, price, embedding: embedding.embedding, }); return product; } ``` **Step 3: Hybrid Search API** ```typescript theme={null} @api({ httpMethod: 'POST' }) async searchProducts( query: string, filters: { minPrice?: number; maxPrice?: number; category?: string; } = {}, limit: number = 20 ) { const embedding = await Embedding.embedOne(query, 'voyage', 'query'); // Filter conditions const conditions: string[] = []; const params: any[] = [ JSON.stringify(embedding.embedding), query, query, 0.6, // vectorWeight 0.4, // ftsWeight ]; if (filters.minPrice) { conditions.push(`p.price >= ?`); params.push(filters.minPrice); } if (filters.maxPrice) { conditions.push(`p.price <= ?`); params.push(filters.maxPrice); } if (filters.category) { conditions.push(`p.category = ?`); params.push(filters.category); } const whereClause = conditions.length > 0 ? `AND ${conditions.join(' AND ')}` : ''; params.push(limit); const results = await this.getPuri().raw(` WITH vector_results AS ( SELECT id, 1 - (embedding <=> ?) AS vector_score FROM products WHERE embedding IS NOT NULL ), fts_results AS ( SELECT id, ts_rank(search_vector, plainto_tsquery('simple', ?)) AS fts_score FROM products WHERE search_vector @@ plainto_tsquery('simple', ?) ) SELECT p.id, p.name, p.description, p.sku, p.price, COALESCE(v.vector_score, 0) AS vector_score, COALESCE(f.fts_score, 0) AS fts_score, (COALESCE(v.vector_score, 0) * ? + COALESCE(f.fts_score, 0) * ?) AS hybrid_score FROM products p LEFT JOIN vector_results v ON p.id = v.id LEFT JOIN fts_results f ON p.id = f.id WHERE (v.id IS NOT NULL OR f.id IS NOT NULL) ${whereClause} ORDER BY hybrid_score DESC LIMIT ? `, params); return results.rows; } ``` **Usage examples**: ```typescript theme={null} // General search await ProductModel.searchProducts("MacBook Pro"); // With filters await ProductModel.searchProducts("laptop", { minPrice: 1000000, maxPrice: 2000000, category: "laptop", }); ``` ## Benchmarks ### Search Accuracy Comparison Tested on an actual 1000-product DB: | Method | Accuracy (MAP\@10) | Pros | Cons | | ------------------- | ------------------ | ---------------------- | ---------------------- | | Keyword only (LIKE) | 0.45 | Fast | Can't find semantics | | FTS only | 0.68 | Partial matching | Weak on synonyms | | Vector only | 0.72 | Semantic understanding | Weak on exact matching | | **Hybrid** | **0.85** | **Best of both** | Complex | **Conclusion**: Hybrid is 15-20% more accurate. ## Cautions **Cautions for hybrid search in Sonamu**: 1. **Both indexes required**: Vector + FTS ```sql theme={null} CREATE INDEX ... USING hnsw (embedding vector_cosine_ops); CREATE INDEX ... USING GIN (search_vector); ``` 2. **tsvector update**: Automate with trigger ```sql theme={null} CREATE TRIGGER tsvector_update ... ``` 3. **Weight sum = 1**: Normalization ```typescript theme={null} const total = vectorWeight + ftsWeight; vectorWeight = vectorWeight / total; ftsWeight = ftsWeight / total; ``` 4. **NULL handling**: Use COALESCE ```sql theme={null} COALESCE(v.vector_score, 0) ``` 5. **LEFT JOIN**: OK if only one side matches ```sql theme={null} WHERE v.id IS NOT NULL OR f.id IS NOT NULL ``` 6. **Use 'simple' for Korean**: FTS language setting ```sql theme={null} to_tsvector('simple', text) ``` 7. **Performance monitoring**: EXPLAIN ANALYZE ```sql theme={null} EXPLAIN ANALYZE [hybrid query] ``` ## When to Use Hybrid Search? ### Recommend Hybrid * **E-commerce product search** * Semantics + model names, SKUs * **Technical documentation search** * Concepts + function names, code * **Customer support** * Problem descriptions + exact terms ### Vector Only is Sufficient * **Recommendation systems** * Keywords not needed * **Image search** * No text keywords * **Finding similar documents** * Only semantics matter ## Next Steps Basic vector search implementation Splitting long documents