> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/avnlp/vectordb/llms.txt
> Use this file to discover all available pages before exploring further.

# Sparse search

> Keyword and lexical matching with SPLADE or BM25 for exact terminology

Sparse search uses SPLADE models to create sparse vectors that emphasize specific terms, similar to traditional BM25 but with learned term importance. This excels when exact terminology matters—legal documents, product SKUs, or technical specifications.

## How it works

Sparse search creates term-focused embeddings where non-zero dimensions correspond to important keywords, enabling precise lexical matching.

### Search process

1. **Sparse embedding** - Convert query text to sparse vector using SPLADE or BM25
2. **Keyword matching** - Match based on term importance and frequency
3. **Result ranking** - Rank documents by sparse similarity scores
4. **Optional RAG** - Generate answer using retrieved documents

### SPLADE vs BM25

**SPLADE (Sparse Lexical and Expansion)**

* Neural sparse encoder with learned term importance
* Expands query with related terms
* Better generalization than traditional BM25
* Requires model inference

**BM25 (Best Matching 25)**

* Classic TF-IDF based ranking function
* Fast, no model required
* Purely statistical term weighting
* Native support in Weaviate

## Key features

* Supports SPLADE-based or BM25-style sparse encoders
* Weaviate uses native BM25 without external embeddings
* Works alongside dense search or as standalone retrieval method
* Excels at exact terminology and keyword precision

## Implementation

<CodeGroup>
  ```python LangChain theme={null}
  from vectordb.langchain.sparse_indexing import PineconeSparseSearchPipeline

  pipeline = PineconeSparseSearchPipeline("config.yaml")
  results = pipeline.search(
      query="HIPAA compliance requirements",
      top_k=10,
      filters={"category": "legal"},
  )

  for doc in results["documents"]:
      print(doc.page_content[:100])
  ```

  ```python Haystack theme={null}
  from vectordb.haystack.sparse_indexing import WeaviateSparseSearchPipeline

  # Weaviate uses native BM25
  pipeline = WeaviateSparseSearchPipeline("config.yaml")
  results = pipeline.search(
      query="product SKU XR-2024",
      top_k=5,
  )

  for doc in results["documents"]:
      print(f"BM25 Score: {doc.score:.3f} - {doc.content}")
  ```
</CodeGroup>

## Configuration

### Required settings

<ParamField path="pinecone.api_key" type="string" required>
  Pinecone API authentication key
</ParamField>

<ParamField path="pinecone.index_name" type="string" required>
  Target index name for sparse search
</ParamField>

### Optional settings

<ParamField path="pinecone.namespace" type="string">
  Namespace within the index
</ParamField>

<ParamField path="pinecone.dimension" type="integer" default={384}>
  Dimension for placeholder dense vector (sparse search only)
</ParamField>

<ParamField path="sparse.model" type="string">
  Sparse encoder model (e.g., SPLADE)
</ParamField>

### Example configuration

```yaml theme={null}
pinecone:
  api_key: "${PINECONE_API_KEY}"
  index_name: "sparse-search"
  namespace: "default"
  dimension: 384

sparse:
  enabled: true
  model: "splade-cocondenser-ensembledistil"
```

## Search parameters

<ParamField path="query" type="string" required>
  Search query text to encode with sparse embedder
</ParamField>

<ParamField path="top_k" type="integer" default={10}>
  Number of results to return
</ParamField>

<ParamField path="filters" type="dict">
  Optional metadata filters for pre-filtering
</ParamField>

## When to use sparse search

Sparse search is ideal when:

**Exact terminology matters**

* Legal documents with specific clauses
* Medical records with precise diagnoses
* Technical documentation with exact API names
* Product catalogs with SKUs or model numbers

**Query contains specific identifiers**

* Document IDs or reference numbers
* Proper nouns and acronyms
* Version numbers or dates
* Industry-specific jargon

### Example scenarios

```python theme={null}
# Legal document search
results = pipeline.search(
    query="GDPR Article 17 right to erasure",
    top_k=5,
    filters={"jurisdiction": "EU"},
)
# Sparse search excels at matching exact article numbers

# Product SKU lookup
results = pipeline.search(
    query="laptop model XPS-9520",
    top_k=3,
)
# Exact match on model number "XPS-9520"

# Technical API search
results = pipeline.search(
    query="boto3 s3.upload_file method",
    top_k=5,
)
# Precise matching on "boto3" and "upload_file"
```

## Sparse vector representation

Sparse embeddings are dictionaries mapping token indices to weights:

```python theme={null}
# Example sparse embedding
{
  "indices": [42, 156, 892, 1024, 2048],
  "values": [0.95, 0.78, 0.62, 0.43, 0.31]
}
```

Only non-zero dimensions are stored, making sparse vectors memory-efficient despite high dimensionality (often 30k+ dimensions).

## Database-specific implementations

### Pinecone

Uses `sparse_values` field alongside placeholder dense vectors. Requires explicit sparse embedding generation.

```python theme={null}
documents = pipeline.db.query_with_sparse(
    vector=[0.0] * dimension,  # Placeholder dense
    sparse_vector=query_embedding,
    top_k=top_k,
    filter=filters,
    namespace=namespace,
)
```

### Weaviate

Native BM25 support without external sparse embeddings. Uses built-in keyword search:

```python theme={null}
results = collection.query.bm25(
    query="search terms",
    limit=top_k,
    where=filters,
)
```

### Qdrant

Supports sparse vectors with payload-based filtering and optimized indexing.

### Milvus

Sparse vector fields with partition-key isolation for multi-tenant scenarios.

### Chroma

Flexible sparse search with document and collection scoping.

## Combining with dense search

<Tip>
  For best results, use sparse search as part of a hybrid retrieval strategy. Combine with dense semantic search for both conceptual understanding and keyword precision.
</Tip>

See [Hybrid search](/features/hybrid-search) for fusion strategies.

## Performance considerations

* Sparse search is typically faster than dense search (fewer dimensions to compare)
* BM25 requires no model inference, just term statistics
* SPLADE models add inference latency but provide better generalization
* Memory footprint is low due to sparse representation

## Evaluation metrics

Sparse search performance metrics:

* **Precision\@k** - Fraction of retrieved docs that are relevant
* **Recall\@k** - Fraction of relevant docs that are retrieved
* **MRR** - Mean reciprocal rank of first relevant result
* **NDCG\@k** - Normalized discounted cumulative gain

## Related features

<CardGroup cols={2}>
  <Card title="Hybrid search" icon="merge" href="/features/hybrid-search">
    Combine dense and sparse retrieval
  </Card>

  <Card title="Semantic search" icon="magnifying-glass" href="/features/semantic-search">
    Dense vector similarity search
  </Card>

  <Card title="Reranking" icon="arrow-down-1-9" href="/features/reranking">
    Improve results with cross-encoders
  </Card>

  <Card title="Metadata filtering" icon="filter" href="/features/metadata-filtering">
    Structured filtering on document fields
  </Card>
</CardGroup>