> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/avnlp/vectordb/llms.txt
> Use this file to discover all available pages before exploring further.

# Diversity filtering

> Post-retrieval redundancy reduction using MMR or clustering

Diversity filtering post-processes search results to ensure the returned documents cover different aspects of the query, reducing redundancy and improving information coverage. When search returns many near-duplicates, diversity filtering selects representative documents to maximize information coverage.

## How it works

Diversity filtering over-fetches candidates from the vector database, then applies post-processing to select diverse results.

### Pipeline architecture

1. **Query embedding** - Convert query text to dense vector
2. **Over-fetch** - Retrieve 3x top\_k candidates from database
3. **Re-embedding** - Generate embeddings for retrieved documents
4. **Diversity filtering** - Apply MMR or clustering method
5. **Limit** - Return top\_k diverse documents
6. **Optional RAG** - Generate answer using diverse documents

### Why diversity matters

Standard semantic search returns the k most similar documents, which often results in redundant information (e.g., 5 similar paragraphs from the same source). Diversity filtering ensures results cover different perspectives, sources, or aspects of the query topic.

## Diversity methods

### MMR (Maximal Marginal Relevance)

<Accordion title="MMR - Default method">
  **How it works**: Balances query relevance with inter-document diversity using lambda parameter.

  **Formula**: `MMR(d) = λ × sim(d, query) - (1-λ) × max_sim(d, selected)`

  **Configuration**:

  * `max_documents` - Maximum documents to return
  * `lambda_param` - Relevance-diversity trade-off (default: 0.5)

  **Best for**: Retrieval where both relevance and diversity matter

  **Speed**: Fast (greedy algorithm)
</Accordion>

### Clustering-based

<Accordion title="Clustering - Topic coverage method">
  **How it works**: Groups retrieved documents into N clusters using embeddings, then samples M documents from each cluster.

  **Configuration**:

  * `num_clusters` - Number of topic clusters (default: 3)
  * `samples_per_cluster` - Docs per cluster (default: 2)

  **Best for**: Ensuring coverage of distinct topic areas

  **Speed**: Moderate (K-means clustering)
</Accordion>

## Key features

* Two diversity methods: MMR and clustering-based selection
* Over-fetching with configurable multiplier (default 3x)
* Re-embedding ensures consistent similarity calculations
* Works with all vector databases
* Optional RAG integration

## Implementation

<CodeGroup>
  ```python LangChain MMR theme={null}
  from vectordb.langchain.diversity_filtering import PineconeDiversityFilteringSearchPipeline

  pipeline = PineconeDiversityFilteringSearchPipeline("config.yaml")
  results = pipeline.search(
      query="machine learning applications",
      top_k=5,
  )

  for doc in results["documents"]:
      print(f"Diverse result: {doc.page_content[:100]}...")
  ```

  ```python LangChain Clustering theme={null}
  from vectordb.langchain.diversity_filtering import QdrantDiversityFilteringSearchPipeline

  pipeline = QdrantDiversityFilteringSearchPipeline({
      "qdrant": {"url": "...", "collection_name": "..."},
      "diversity": {
          "method": "clustering",
          "num_clusters": 3,
          "samples_per_cluster": 2,
      },
  })

  results = pipeline.search(query="AI ethics", top_k=6)
  ```

  ```python Haystack theme={null}
  from vectordb.haystack.diversity_filtering import WeaviateDiversityFilteringSearchPipeline

  pipeline = WeaviateDiversityFilteringSearchPipeline("config.yaml")
  results = pipeline.search(
      query="renewable energy technologies",
      top_k=10,
  )
  ```
</CodeGroup>

## Configuration

### Required settings

<ParamField path="pinecone.api_key" type="string" required>
  Vector database API authentication
</ParamField>

<ParamField path="pinecone.index_name" type="string" required>
  Target index name for search
</ParamField>

### Diversity configuration

<ParamField path="diversity.method" type="string" default="mmr">
  Diversity method: `"mmr"` or `"clustering"`
</ParamField>

<ParamField path="diversity.candidate_multiplier" type="integer" default={3}>
  Over-fetch multiplier (retrieves top\_k × multiplier candidates)
</ParamField>

#### MMR-specific

<ParamField path="diversity.max_documents" type="integer" default={10}>
  Maximum documents to return for MMR method
</ParamField>

<ParamField path="diversity.lambda_param" type="float" default={0.5}>
  Relevance-diversity trade-off (0.0-1.0)

  * 1.0 = pure relevance
  * 0.5 = balanced
  * 0.0 = pure diversity
</ParamField>

#### Clustering-specific

<ParamField path="diversity.num_clusters" type="integer" default={3}>
  Number of clusters for clustering method
</ParamField>

<ParamField path="diversity.samples_per_cluster" type="integer" default={2}>
  Documents to sample from each cluster
</ParamField>

### Example configurations

<CodeGroup>
  ```yaml MMR method theme={null}
  pinecone:
    api_key: "${PINECONE_API_KEY}"
    index_name: "diversity-search"
    namespace: "production"

  embedder:
    model_name: "all-MiniLM-L6-v2"

  diversity:
    method: "mmr"
    lambda_param: 0.5
    max_documents: 10
    candidate_multiplier: 3

  rag:
    enabled: true
    generator_model: "gpt-4o-mini"
  ```

  ```yaml Clustering method theme={null}
  pinecone:
    api_key: "${PINECONE_API_KEY}"
    index_name: "diversity-search"

  embedder:
    model_name: "all-MiniLM-L6-v2"

  diversity:
    method: "clustering"
    num_clusters: 4
    samples_per_cluster: 2
    candidate_multiplier: 4
  ```
</CodeGroup>

## Search parameters

<ParamField path="query" type="string" required>
  Search query text to embed and match against documents
</ParamField>

<ParamField path="top_k" type="integer" default={10}>
  Number of diverse documents to return. Pipeline retrieves 3x this amount for diversity selection.
</ParamField>

<ParamField path="filters" type="dict">
  Optional metadata filters to apply during retrieval
</ParamField>

## Use cases

### Exploratory search

When users need to see different perspectives:

```python theme={null}
results = pipeline.search(
    query="climate change impacts",
    top_k=10,
)
# Returns diverse perspectives: economic, environmental, social, etc.
```

### Multi-document summarization

Provide diverse context to LLMs:

```python theme={null}
pipeline = PineconeDiversityFilteringSearchPipeline({
    "pinecone": {"api_key": "...", "index_name": "..."},
    "diversity": {"method": "mmr", "lambda_param": 0.4},
    "rag": {"enabled": True},
})

results = pipeline.search(
    query="summarize AI safety research",
    top_k=5,
)
print(results["answer"])  # Summary based on diverse sources
```

### News aggregation

Show articles from different sources:

```python theme={null}
results = pipeline.search(
    query="latest quantum computing breakthroughs",
    top_k=8,
    filters={"content_type": "news"},
)
# Returns articles from diverse sources, avoiding duplicate coverage
```

### Research literature review

Cover different research approaches:

```python theme={null}
pipeline = QdrantDiversityFilteringSearchPipeline({
    "diversity": {
        "method": "clustering",
        "num_clusters": 5,
        "samples_per_cluster": 2,
    },
})

results = pipeline.search(
    query="neural architecture search methods",
    top_k=10,
)
# Returns papers covering different NAS approaches
```

## Method comparison

| Aspect               | MMR                           | Clustering                           |
| -------------------- | ----------------------------- | ------------------------------------ |
| **Query awareness**  | Yes, uses query similarity    | No, only inter-doc similarity        |
| **Speed**            | Fast (greedy)                 | Moderate (K-means)                   |
| **Parameters**       | lambda\_param, max\_documents | num\_clusters, samples\_per\_cluster |
| **Best for**         | Relevance + diversity balance | Topic coverage                       |
| **Deterministic**    | Yes                           | No (K-means random init)             |
| **Interpretability** | High (clear trade-off)        | Medium (cluster interpretation)      |

## Choosing a method

<Steps>
  <Step title="Default: Use MMR">
    MMR is query-aware and provides explicit relevance-diversity control. Start with `lambda_param=0.5`.
  </Step>

  <Step title="Topic coverage: Use clustering">
    When you need guaranteed coverage of N distinct topics, use clustering with `num_clusters=N`.
  </Step>

  <Step title="Tune parameters">
    * MMR: Adjust lambda (↑ relevance, ↓ diversity)
    * Clustering: Adjust num\_clusters and samples\_per\_cluster
  </Step>

  <Step title="Evaluate">
    Measure diversity with metrics like average pairwise similarity or topic coverage.
  </Step>
</Steps>

## Diversity helpers

The diversity filtering pipeline uses helper methods that can be used independently:

```python theme={null}
from vectordb.langchain.diversity_filtering.helpers import DiversityFilteringHelper

# MMR diversification
diverse_docs = DiversityFilteringHelper.mmr_diversify(
    documents=retrieved_docs,
    embeddings=doc_embeddings,
    query_embedding=query_embedding,
    max_documents=10,
    lambda_param=0.5,
)

# Clustering diversification
diverse_docs = DiversityFilteringHelper.clustering_diversify(
    documents=retrieved_docs,
    embeddings=doc_embeddings,
    num_clusters=3,
    samples_per_cluster=2,
)
```

## Over-fetching strategy

<Tip>
  Over-fetching provides the diversity algorithm with more options for selecting diverse results. A 3x multiplier is recommended - it provides enough candidates without excessive latency.
</Tip>

**Example**:

```python theme={null}
top_k = 10
candidate_multiplier = 3
retrieved_count = 30  # 10 × 3

# Retrieve 30 candidates, select 10 diverse ones
```

**Trade-offs**:

* **Higher multiplier** - More diversity options, higher latency
* **Lower multiplier** - Faster, but limited diversity options

## Performance considerations

### Time complexity

* **MMR**: O(k × n) where k=top\_k, n=candidates
* **Clustering**: O(n × d × iterations) for K-means

### Optimization tips

1. **Cache embeddings** - Store document embeddings to avoid recomputation
2. **Limit over-fetch** - Balance diversity quality with latency (3-5x multiplier)
3. **Use metadata filters** - Reduce candidate pool before diversity filtering
4. **Batch processing** - Process multiple queries together for efficiency

## Evaluation metrics

Measure diversity effectiveness:

### Average pairwise similarity

```python theme={null}
import numpy as np
from vectordb.langchain.diversity_filtering.helpers import DiversityFilteringHelper

# Calculate average cosine similarity between all pairs
similarities = []
for i in range(len(embeddings)):
    for j in range(i+1, len(embeddings)):
        sim = DiversityFilteringHelper.cosine_similarity(
            embeddings[i], embeddings[j]
        )
        similarities.append(sim)

avg_similarity = np.mean(similarities)
print(f"Average pairwise similarity: {avg_similarity:.3f}")
# Lower is more diverse
```

### Topic coverage

Count distinct topics/sources in results:

```python theme={null}
topics = set(doc.metadata.get("topic") for doc in results["documents"])
print(f"Topic coverage: {len(topics)} distinct topics")
```

## Related features

<CardGroup cols={2}>
  <Card title="MMR" icon="chart-scatter" href="/features/mmr">
    Maximal marginal relevance algorithm details
  </Card>

  <Card title="Semantic search" icon="magnifying-glass" href="/features/semantic-search">
    Initial retrieval before diversity filtering
  </Card>

  <Card title="Hybrid search" icon="merge" href="/features/hybrid-search">
    Dense + sparse retrieval
  </Card>

  <Card title="Reranking" icon="arrow-down-1-9" href="/features/reranking">
    Cross-encoder second-stage scoring
  </Card>
</CardGroup>