Skip to main content
The dataloaders module provides a unified interface for loading benchmark datasets, normalizing their formats, and converting them to framework-specific document objects.

​
Overview

Dataloaders abstract away dataset-specific formats and provide:
  • Catalog-based loader creation - Factory pattern for consistent instantiation
  • Normalized record format - All datasets produce DatasetRecord objects
  • Framework conversion - Automatic conversion to Haystack or LangChain documents
  • Evaluation queries - Ground-truth QA pairs for retrieval benchmarking
  • Streaming support - Memory-efficient iteration over large datasets

​
Supported datasets

DatasetTypeRecordsQueriesDescription
TriviaQA (triviaqa)Open-domain QA~500 index~100 evalTrivia questions with evidence documents
ARC (arc)Science QA~1000 index~200 evalAI2 Reasoning Challenge questions
PopQA (popqa)Entity-centric QA~500 index~100 evalEntity-focused questions from Wikipedia
FActScore (factscore)Factuality QA~500 index~100 evalFactuality-focused evaluation dataset
Earnings Calls (earnings_calls)Financial QA~300 index~50 evalFinancial QA from earnings call transcripts

​
Architecture

​
Core components

dataloaders/
├── catalog.py          # Factory for creating loaders
├── base.py             # Abstract base class defining the contract
├── types.py            # Shared types (DatasetRecord, EvaluationQuery)
├── converters.py       # Framework document conversion
├── dataset.py          # LoadedDataset wrapper
├── evaluation.py       # Evaluation query extraction
└── datasets/           # Per-dataset implementations
    ├── triviaqa.py
    ├── arc.py
    ├── popqa.py
    ├── factscore.py
    └── earnings_calls.py

​
Class hierarchy

BaseDatasetLoader (ABC)
    ├── TriviaQALoader
    ├── ARCLoader
    ├── PopQALoader
    ├── FactScoreLoader
    └── EarningsCallsLoader

​
Basic usage

​
Creating a loader

Use the DataloaderCatalog factory to create loaders: 
from vectordb.dataloaders import DataloaderCatalog

loader = DataloaderCatalog.create(
    name="triviaqa",
    split="test",
    limit=500
)

​
Loading datasets

# Load normalized records
dataset = loader.load()

print(f"Loaded {len(dataset.records)} records")
print(f"Dataset type: {dataset.dataset_type}")

​
Converting to framework documents

from vectordb.dataloaders.converters import DocumentConverter, records_to_items

# Convert records to normalized items
items = records_to_items(dataset.records)

# Convert to Haystack documents
haystack_docs = DocumentConverter.to_haystack(items)

# Convert to LangChain documents
langchain_docs = DocumentConverter.to_langchain(items)

​
Data structures

​
DatasetRecord

Normalized document record with text and metadata: 
@dataclass(frozen=True, slots=True)
class DatasetRecord:
    text: str                    # Document content to index
    metadata: dict[str, Any]     # Dataset-specific metadata
Example: 
DatasetRecord(
    text="The Great Wall of China was built over several centuries...",
    metadata={
        "id": "doc_001",
        "source": "triviaqa",
        "title": "Great Wall of China"
    }
)

​
EvaluationQuery

Evaluation query with ground-truth answers and relevant document IDs: 
@dataclass(frozen=True, slots=True)
class EvaluationQuery:
    query: str                      # User/evaluation question
    answers: list[str]              # Ground-truth answers
    relevant_doc_ids: list[str]     # IDs of known relevant docs
    metadata: dict[str, Any]        # Additional metadata
Example: 
EvaluationQuery(
    query="When was the Great Wall of China built?",
    answers=["over several centuries", "7th century BC"],
    relevant_doc_ids=["doc_001", "doc_045"],
    metadata={"difficulty": "easy", "category": "history"}
)

​
LoadedDataset

Wrapper containing loaded records and metadata: 
class LoadedDataset:
    dataset_type: DatasetType      # "triviaqa", "arc", etc.
    records: list[DatasetRecord]   # Normalized documents

​
Dataset implementations

​
TriviaQA

Dataset ID: triviaqa
HuggingFace: trivia_qa
Structure: Questions with multiple evidence documents 
loader = DataloaderCatalog.create("triviaqa", split="test", limit=500)
dataset = loader.load()
Record format:
  • text: Evidence document content
  • metadata.id: Document identifier
  • metadata.title: Document title
  • metadata.question: Associated question

​
ARC (AI2 Reasoning Challenge)

Dataset ID: arc
HuggingFace: ai2_arc
Structure: Science questions with multiple-choice answers 
loader = DataloaderCatalog.create("arc", split="test", limit=1000)
dataset = loader.load()
Record format:
  • text: Question + answer choices
  • metadata.id: Question identifier
  • metadata.question: Question text
  • metadata.answerKey: Correct answer

​
PopQA

Dataset ID: popqa
HuggingFace: akariasai/PopQA
Structure: Entity-centric questions from Wikipedia 
loader = DataloaderCatalog.create("popqa", split="test", limit=500)
dataset = loader.load()
Record format:
  • text: Wikipedia passage
  • metadata.id: Passage identifier
  • metadata.entity: Entity mention
  • metadata.question: Associated question

​
FActScore

Dataset ID: factscore
HuggingFace: dskar/FActScore
Structure: Factuality-focused QA pairs 
loader = DataloaderCatalog.create("factscore", split="test", limit=500)
dataset = loader.load()
Record format:
  • text: Factual statement
  • metadata.id: Statement identifier
  • metadata.topic: Topic/category

​
Earnings Calls

Dataset ID: earnings_calls
HuggingFace: lamini/earnings-calls-qa
Structure: Financial QA from earnings call transcripts 
loader = DataloaderCatalog.create("earnings_calls", split="train", limit=300)
dataset = loader.load()
Record format:
  • text: Transcript excerpt
  • metadata.id: Excerpt identifier
  • metadata.company: Company name
  • metadata.quarter: Reporting quarter

​
Base loader interface

All loaders implement the BaseDatasetLoader abstract class: 
class BaseDatasetLoader(ABC):
    def __init__(
        self,
        dataset_name: str,
        split: str,
        limit: int | None = None,
        streaming: bool = True,
    ) -> None:
        """Initialize the loader with dataset configuration."""

    @property
    @abstractmethod
    def dataset_type(self) -> DatasetType:
        """Return the supported dataset type identifier."""

    @abstractmethod
    def _load_dataset_iterable(self) -> Iterable[Mapping[str, Any]]:
        """Return the raw dataset rows as an iterable."""

    @abstractmethod
    def _parse_row(self, row: Mapping[str, Any]) -> list[DatasetRecord]:
        """Parse a dataset row into normalized records."""

    def load(self) -> LoadedDataset:
        """Load the dataset and return normalized records."""

​
Document conversion

The DocumentConverter class provides framework-specific conversion:

​
Haystack conversion

from vectordb.dataloaders.converters import DocumentConverter
from haystack import Document

items = [{"text": "content", "metadata": {"id": "1"}}]
haystack_docs = DocumentConverter.to_haystack(items)

# Result: List[Document]
# Document(content="content", meta={"id": "1"})

​
LangChain conversion

from vectordb.dataloaders.converters import DocumentConverter
from langchain_core.documents import Document

items = [{"text": "content", "metadata": {"id": "1"}}]
langchain_docs = DocumentConverter.to_langchain(items)

# Result: List[Document]
# Document(page_content="content", metadata={"id": "1"})

​
Configuration

Dataloaders integrate with YAML configuration: 
dataloader:
  dataset: "triviaqa"    # Dataset identifier
  split: "test"          # Dataset split
  limit: 500             # Optional record limit
Load configuration and create loader: 
from vectordb.utils.config import load_config
from vectordb.dataloaders import DataloaderCatalog

config = load_config("config.yaml")
dl_config = config["dataloader"]

loader = DataloaderCatalog.create(
    name=dl_config["dataset"],
    split=dl_config["split"],
    limit=dl_config.get("limit")
)

​
Streaming mode

By default, loaders use streaming to handle large datasets efficiently: 
loader = DataloaderCatalog.create(
    name="triviaqa",
    split="test",
    limit=None  # No limit
)

# Streams dataset without loading all records into memory
dataset = loader.load()

​
Custom loaders

Implement custom loaders by extending BaseDatasetLoader: 
from vectordb.dataloaders.base import BaseDatasetLoader
from vectordb.dataloaders.types import DatasetRecord, DatasetType

class CustomLoader(BaseDatasetLoader):
    @property
    def dataset_type(self) -> DatasetType:
        return "custom"

    def _load_dataset_iterable(self):
        # Load raw dataset rows
        from datasets import load_dataset
        dataset = load_dataset(
            self.dataset_name,
            split=self.split,
            streaming=self.streaming
        )
        return dataset

    def _parse_row(self, row):
        # Parse row into DatasetRecord objects
        return [
            DatasetRecord(
                text=row["content"],
                metadata={"id": row["id"]}
            )
        ]

​
Error handling

Dataloaders raise specific exceptions for different error conditions: 
from vectordb.dataloaders.types import (
    UnsupportedDatasetError,
    DatasetLoadError,
    DatasetValidationError
)

try:
    loader = DataloaderCatalog.create("invalid_dataset")
except UnsupportedDatasetError as e:
    print(f"Dataset not supported: {e}")

try:
    dataset = loader.load()
except DatasetLoadError as e:
    print(f"Failed to load dataset: {e}")
except DatasetValidationError as e:
    print(f"Dataset validation failed: {e}")

​
Best practices

Use the catalog

Always use DataloaderCatalog.create() instead of instantiating loaders directly for consistency

Set limits during development

Use limit= parameter during prototyping to avoid loading full datasets

Convert once

Convert to framework documents once during indexing, not repeatedly during queries

Handle exceptions

Catch DatasetLoadError and DatasetValidationError for robust pipelines