Elastic Vector Search
What is Vector Search?
Vector search leverages machine learning (ML) to capture the meaning and context of unstructured data, including text and images, transforming it into a numeric representation. Frequently used for semantic search, vector search finds similar data using approximate nearest neighbor (ANN) algorithms. Compared to traditional keyword search, vector search yields more relevant results and executes faster.
Why is Vector Search Important?
Vector search overcomes this limitation, allowing you to search by what you mean. It can quickly deliver answers to queries based on similarity search. That's because vector embedding captures the unstructured data beyond text, such as videos, images, and audio. You can enhance the search experience by combining vector search with filtering and aggregations to optimize relevance by implementing a hybrid search and combining it with traditional scoring.
How Does a vector search engine work?
Key Differences:
| Aspect | Sparse Vector Search | Traditional Term/Match |
|---|---|---|
| Search Type | Semantic similarity using embeddings | Exact/fuzzy matches on specific fields |
| Fields Used | Embedding fields (*_embedding) | Text fields (e.g., customerfacilityname) |
| Model Involvement | Leverages a pre-trained ELSER model | No external model involved |
| Flexibility | Flexible, meaning-based matches | Rigid, value-based or fuzzy criteria |
| Performance | May be slower due to model inference | Faster, based on direct field matching |
| Complexity | Requires embedding fields and ELSER configuration | Works with standard text fields and queries |
SETUP :
ELASTIC QUERY vs ELSER QUERY :
| GET customerentity/_search?pretty { "query": { "bool": { "should": [ { "bool": { "must": [ { "term": { "customerentity.customerdetails.customercoreinformation.customeraddress.country.isocountrycode.keyword": { "value": "CN", "boost": 1 } } }, { "function_score": { "query": { "match": { "customerentity.customerdetails.customercoreinformation.customeraddress.singlelineaddress": { "query": "1 WENCHUAN ROAD, ", "operator": "OR", "analyzer": "addressstopwords", "fuzziness": "AUTO", "prefix_length": 0, "max_expansions": 50, "minimum_should_match": "25%", "fuzzy_transpositions": true, "lenient": false, "zero_terms_query": "NONE", "auto_generate_synonyms_phrase_query": true, "boost": 1 } } }, "functions": [], "score_mode": "multiply", "max_boost": 3.4028235e+38, "min_score": 0.2, "boost": 1 } }, { "match_phrase": { "customerentity.customerdetails.customercoreinformation.tradingname": { "query": "ZHEJIANG SHANGHE PLASTIC RUBBER MATERIAL C", "slop": 0, "zero_terms_query": "NONE", "boost": 1 } } }, { "function_score": { "query": { "match": { "customerentity.customerdetails.customercoreinformation.tradingname": { "query": "GUANGZHOU KELIER HOUSEWARE CO. LTD", "operator": "OR", "analyzer": "customerstopwords", "fuzziness": "AUTO", "prefix_length": 0, "max_expansions": 50, "minimum_should_match": "1<50% 4<40%", "fuzzy_transpositions": true, "lenient": false, "zero_terms_query": "NONE", "auto_generate_synonyms_phrase_query": true, "boost": 1 } } }, "functions": [], "score_mode": "multiply", "max_boost": 3.4028235e+38, "min_score": 0.2, "boost": 1 } }, { "function_score": { "query": { "match": { "customerentity.customerdetails.customercoreinformation.customeraddress.cityname": { "query": "HUZHOU", "operator": "OR", "fuzziness": "AUTO", "prefix_length": 0, "max_expansions": 50, "fuzzy_transpositions": true, "lenient": false, "zero_terms_query": "NONE", "auto_generate_synonyms_phrase_query": true, "boost": 1 } } }, "functions": [], "score_mode": "multiply", "max_boost": 3.4028235e+38, "min_score": 0.2, "boost": 1 } } ], "adjust_pure_negative": true, "boost": 1 } } ], "adjust_pure_negative": true, "boost": 1 } }, "_source": true } | GET prod_customer_replica/_search { "_source": true, "min_score": 50, "explain": true, "query": { "bool": { "must": [ { "sparse_vector": { "field": "tradingname_embedding", "inference_id": ".elser_model_2_linux-x86_64", "query": "EL CO IMPORT" } }, { "sparse_vector": { "field": "singlelineaddress_embedding", "inference_id": ".elser_model_2_linux-x86_64", "query": "1 SHEBIN EL-KOUM ST" } }, { "term": { "customeraddress.country.isocountrycode": { "value": "EG" } } }, { "term": { "customeraddress.cityname.keyword": { "value": "ISMAILEYA" } } } ] } } } |
|---|
KEY CHANGES :
- ELSER EMBEDDINGS ARE USED
- MODEL INFERENCE IS REQUIRED
- MATCH TYPE IS SPARSE VECTOR
CUSTOMER MATCH STATS
(Same Customer Data Set was used to find potential matche count using Elastic and Elser query)
SAMPLE RESULT WITH THE DATA SET
(ELSER MATCH STATS)
(EXISTING ELASTIC MATCH STATS)