> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/avnlp/vectordb/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataloaders

> Dataset loading and conversion utilities for evaluation and indexing

Dataloader APIs for loading benchmark datasets and converting them to Haystack or LangChain document formats.

## DataloaderCatalog

Factory class for creating dataset loaders by name.

### Methods

#### create

Create a dataset loader instance.

```python theme={null}
DataloaderCatalog.create(
    name: Literal["triviaqa", "arc", "popqa", "factscore", "earnings_calls"],
    split: str = "test",
    limit: int | None = None,
    dataset_id: str | None = None
) -> BaseDatasetLoader
```

<ParamField path="name" type="Literal" required>
  Dataset type name. Options: "triviaqa", "arc", "popqa", "factscore", "earnings\_calls"
</ParamField>

<ParamField path="split" type="str" default="test">
  Dataset split to load (e.g., "train", "test", "validation")
</ParamField>

<ParamField path="limit" type="int" optional>
  Optional limit on record count to load
</ParamField>

<ParamField path="dataset_id" type="str" optional>
  Optional HuggingFace dataset ID override
</ParamField>

<ResponseField name="loader" type="BaseDatasetLoader">
  Configured dataset loader instance ready to load data
</ResponseField>

#### supported\_datasets

Return list of supported dataset identifiers.

```python theme={null}
DataloaderCatalog.supported_datasets() -> tuple[DatasetType, ...]
```

<ResponseField name="datasets" type="tuple[DatasetType, ...]">
  Tuple of supported dataset type names
</ResponseField>

***

## LoadedDataset

Wrapper for normalized dataset records with conversion methods.

### Constructor

```python theme={null}
LoadedDataset(
    dataset_type: DatasetType,
    records: list[DatasetRecord]
)
```

<ParamField path="dataset_type" type="DatasetType" required>
  Identifier of the dataset (e.g., "triviaqa", "arc")
</ParamField>

<ParamField path="records" type="list[DatasetRecord]" required>
  Normalized dataset records
</ParamField>

### Methods

#### records

Return normalized dataset records.

```python theme={null}
records() -> list[DatasetRecord]
```

<ResponseField name="records" type="list[DatasetRecord]">
  List of normalized dataset records
</ResponseField>

#### to\_dict\_items

Convert normalized records to dictionary items.

```python theme={null}
to_dict_items() -> list[dict[str, Any]]
```

<ResponseField name="items" type="list[dict[str, Any]]">
  List of dictionary representations of records
</ResponseField>

#### to\_haystack

Convert records to Haystack documents.

```python theme={null}
to_haystack() -> list[HaystackDocument]
```

<ResponseField name="documents" type="list[HaystackDocument]">
  List of Haystack Document objects ready for indexing
</ResponseField>

#### to\_langchain

Convert records to LangChain documents.

```python theme={null}
to_langchain() -> list[LangChainDocument]
```

<ResponseField name="documents" type="list[LangChainDocument]">
  List of LangChain Document objects ready for indexing
</ResponseField>

#### evaluation\_queries

Extract evaluation queries from records.

```python theme={null}
evaluation_queries(limit: int | None = None) -> list[EvaluationQuery]
```

<ParamField path="limit" type="int" optional>
  Optional limit applied after deduplication
</ParamField>

<ResponseField name="queries" type="list[EvaluationQuery]">
  List of evaluation queries with ground truth answers for retrieval testing
</ResponseField>

***

## BaseDatasetLoader

Base class for dataset loaders. All specific dataset loaders inherit from this class.

### Supported Datasets

* **TriviaQALoader**: TriviaQA question-answering dataset
* **ARCLoader**: AI2 Reasoning Challenge (ARC) dataset
* **PopQALoader**: PopQA popularity-based question-answering dataset
* **FactScoreLoader**: FactScore factual consistency dataset
* **EarningsCallsLoader**: Earnings calls transcripts dataset

***

## Usage Examples

### Load a dataset

```python theme={null}
from vectordb.dataloaders import DataloaderCatalog

# Create loader
loader = DataloaderCatalog.create(
    name="triviaqa",
    split="test",
    limit=100
)

# Load dataset
dataset = loader.load()

# Convert to Haystack documents
haystack_docs = dataset.to_haystack()

# Convert to LangChain documents
langchain_docs = dataset.to_langchain()

# Get evaluation queries
queries = dataset.evaluation_queries(limit=50)
```

### List supported datasets

```python theme={null}
from vectordb.dataloaders import DataloaderCatalog

# Get all supported datasets
datasets = DataloaderCatalog.supported_datasets()
print(datasets)
# ('triviaqa', 'arc', 'popqa', 'factscore', 'earnings_calls')
```