Metadata filtering

Filter search results using structured document attributes

Metadata filtering applies structured constraints to vector search results, enabling you to narrow results by fields like category, date, author, or custom attributes while still ranking by semantic relevance.

Overview

Vector search finds semantically similar content but cannot filter by structured properties. Metadata filtering adds boolean predicates (equals, range, contains) to retrieve precisely targeted documents.

Metadata filtering is essential for production RAG applications when you need queries like “find documents about ML published in 2024” or “retrieve technical articles by author X”.

How it works

The metadata filtering pipeline follows these steps:

1. Query embedding - Convert query text to vector representation
2. Filtered vector search - Execute similarity search with metadata constraints
3. Post-processing - Apply additional client-side filters if configured
4. RAG generation - Generate answer using filtered documents (optional)

Supported operators

VectorDB supports the following filter operators across all databases:

Operator	Description	Example
equals	Exact match	category = "electronics"
not_equals	Not equal	status != "archived"
gt	Greater than	price > 100
gte	Greater than or equal	date >= "2024-01-01"
lt	Less than	score < 0.5
lte	Less than or equal	rating <= 4.5
in	Value in list	category in ["tech", "science"]
not_in	Value not in list	author not in ["user1", "user2"]
contains	Substring match (case-insensitive)	title contains "machine"
startswith	Prefix match (case-insensitive)	name startswith "Dr"
endswith	Suffix match (case-insensitive)	filename endswith ".pdf"

String operators (contains, startswith, endswith) are case-insensitive for consistent behavior across databases.

Database-specific syntax

Each database uses its own native filter format:

Pinecone

# Pinecone uses JSON filter syntax
filters = {
    "$and": [
        {"category": {"$eq": "technical"}},
        {"date": {"$gte": "2024-01-01"}}
    ]
}

Configuration

Define metadata filters in your pipeline configuration:

filters:
  conditions:
    - field: "category"
      value: "technical"
      operator: "equals"
    - field: "price"
      value: 500
      operator: "lt"

Usage example

LangChain

from vectordb.langchain.metadata_filtering.search.pinecone import (
    PineconeMetadataFilteringSearchPipeline,
)

pipeline = PineconeMetadataFilteringSearchPipeline("config.yaml")

# Search with filters
results = pipeline.search(
    "machine learning frameworks",
    top_k=10,
    filters={"category": {"$eq": "technical"}},
)

print(f"Found {len(results['documents'])} documents")
for doc in results["documents"]:
    print(f"- {doc.metadata['title']} (category: {doc.metadata['category']})")

Performance optimization

Selectivity analysis

Filter order matters for query performance. VectorDB includes selectivity analysis to optimize filter execution:

# High selectivity filters (fewer matches) should run first
filters = [
    {"user_id": "specific-user"},  # High selectivity
    {"category": "news"},          # Lower selectivity
]

Pre-filter vs post-filter

Databases apply filters at different stages:

• Pre-filter - Filter before vector search (faster, smaller search space)
• Post-filter - Filter after vector search (preserves ranking quality)

Use pre-filtering for highly selective filters (user_id, tenant_id) and post-filtering for broader criteria (category, date ranges).

Timing metrics

Track filter performance with built-in timing metrics:

results = pipeline.search(query, filters=filters)

print(f"Filter time: {results['metrics']['filter_time_ms']}ms")
print(f"Search time: {results['metrics']['search_time_ms']}ms")
print(f"Total time: {results['metrics']['total_time_ms']}ms")

Related features

JSON indexing

Filter by nested JSON paths

Namespaces

Logical data partitioning

Multi-tenancy

Tenant-isolated retrieval

Semantic search

Vector similarity search