Hugging Face Datasets: Developer Reference

A Hugging Face dataset is a versioned, structured data collection stored on the Hugging Face Hub and consumed through the datasets Python library. Each dataset exposes one or moresplits (typically train, validation, and test), a typedfeatures schema describing every column, and metadata covering license, task category, and language tags. The library handles downloading, caching, and format conversion so you almost never deal with raw files directly.

Finding Datasets on the Hub

The primary discovery surface is huggingface.co/datasets. Filters available in the UI:

• Task — text classification, question answering, image segmentation, translation, summarization, and more
• Language — ISO 639-1 codes (en, zh, fr, de, …)
• License — Apache 2.0, MIT, CC-BY, CC0, OpenRAIL, etc.
• Size category — under 1K rows up to over 1B rows
• Modality — text, image, audio, video, tabular, multimodal

You can also search programmatically via the Hub Python client:

from huggingface_hub import list_datasets

results = list_datasets(filter='task_categories:text-classification', limit=20)
for ds in results:
    print(ds.id, ds.downloads)

The Datasets Server REST API exposes a /valid endpoint that lists all datasets with precomputed Parquet exports, enabling fast previews and row sampling without downloading a full dataset.

Installation

Thedatasets library runs on Python 3.8+ and is platform-agnostic.

pip install datasets

For image and audio support, install the relevant extras:

pip install datasets[vision]# Pillow
pip install datasets    # librosa, soundfile

To authenticate for private datasets or to push data to the Hub:

pip install huggingface_hub
huggingface-cli login          # prompts for your HF access token

Your token is stored in ~/.cache/huggingface/token on macOS/Linux, or %USERPROFILE%.cachehuggingfacetoken on Windows. Alternatively, set the HF_TOKEN environment variable directly.

Loading a Dataset

The primary entry point is load_dataset(). Without asplit argument it returns a DatasetDict containing all available splits:

from datasets import load_dataset

ds = load_dataset('stanfordnlp/imdb')
print(ds)
# DatasetDict({
#     train: Dataset({features: ['text', 'label'], num_rows: 25000})
#     test:  Dataset({features: ['text', 'label'], num_rows: 25000})
# })

train = ds['train']
print(train.features)
# {'text': Value(dtype='string'), 'label': ClassLabel(names=['neg', 'pos'])}

print(train[0]['text'][:120])

Loading a Single Split

train = load_dataset('stanfordnlp/imdb', split='train')
# Returns a Dataset directly, not a DatasetDict

Loading a Named Configuration

Many datasets define named configs for language variants, domain subsets, or schema versions. Pass the config name as the second positional argument:

ds = load_dataset('Helsinki-NLP/opus_books', 'en-fr')

To list all available configs for a dataset before loading:

from datasets import get_dataset_config_names
print(get_dataset_config_names('Helsinki-NLP/opus_books'))

Loading from Local Files

Pass a format name and file path instead of a Hub dataset ID. Supported formats include CSV, JSON/JSONL, Parquet, Arrow, plain text, and the ImageFolder/AudioFolder conventions.

ds = load_dataset('csv', data_files='my_data.csv')
ds = load_dataset('json', data_files={'train': 'train.jsonl', 'test': 'test.jsonl'})
ds = load_dataset('imagefolder', data_dir='./photos/')

Dataset Structure: Splits and Features

Check which splits a dataset has before loading:

from datasets import get_dataset_split_names
print(get_dataset_split_names('stanfordnlp/imdb'))
# ['train', 'test', 'unsupervised']

Thefeatures dict maps column names to typed descriptors. Common feature types:

Streaming Large Datasets

For datasets too large to download — Common Crawl, The Pile, LAION-5B — pass streaming=True. Data is fetched and decoded on the fly without filling your disk:

ds = load_dataset('allenai/c4', 'en', split='train', streaming=True)

for example in ds.take(1000):
    print(example['text'][:80])

Streaming returns anIterableDataset rather than a Dataset. It supports.map(), .filter(), .shuffle(buffer_size=N), and .take(N), but not random indexing or len(). To get a fixed slice without streaming the whole dataset:

ds = load_dataset('allenai/c4', 'en', split='train[:50000]')

Split slicing accepts absolute row counts ([:50000]), percentages ([:10%]), and stepped ranges ([10%:20%]).

Filtering and Processing

All operations run in Apache Arrow and use multiprocessing by default. The result is cached on disk; re-running the same .map() on the same data returns the cache instantly.

# Filter rows
short = train.filter(lambda x: len(x['text']) < 500)

# Batched map — much faster for tokenization
def tokenize(batch):
    return tokenizer(batch['text'], truncation=True, padding='max_length')

tokenized = train.map(tokenize, batched=True, batch_size=256, num_proc=4)

# Column operations
tokenized = tokenized.remove_columns(['text'])
ds = ds.rename_column('label', 'labels')

# Shuffle and select
ds = ds.shuffle(seed=42).select(range(10000))

Converting to Other Formats

Pushing Your Own Dataset to the Hub

After running huggingface-cli login, push anyDataset or DatasetDict object directly:

from datasets import Dataset, DatasetDict
import pandas as pd

df = pd.read_csv('my_data.csv')
ds = Dataset.from_pandas(df)

ds.push_to_hub('your-username/my-dataset', private=False)

To push train and test splits together:

split = ds.train_test_split(test_size=0.1)
DatasetDict({'train': split['train'], 'test': split['test']}).push_to_hub('your-username/my-dataset')

The Hub stores datasets as sharded Parquet files and auto-generates a dataset preview viewer. Add a README.md (Dataset Card) with YAML front-matter to make your dataset filterable by task, language, and license in the Hub search UI.

Platform Notes

Cache Paths

Windows

Windows uses thespawn start method for multiprocessing, which requires your script entry point to be inside a if __name__ == '__main__': guard. Without this,.map(num_proc=4) will either hang or raise a RuntimeError. If you’re running in a Jupyter notebook, either use num_proc=1 or installmultiprocess alongside datasets, which the library will prefer over the standard-library multiprocessing module.

Disk Space

Large datasets (Common Crawl, RedPajama, LAION) consume hundreds of gigabytes when fully cached. Use streaming=True to avoid downloads. To see what’s in your cache, run python -c "from datasets import inspect_dataset; print(inspect_dataset.__doc__)" or browse the cache directory directly. Cached datasets are stored as Arrow files organized by dataset name and hash; delete subfolders manually to reclaim space.

Using Datasets for Fine-Tuning and Evaluation

The standard fine-tuning pipeline is: load dataset → tokenize with.map(batched=True) → set format to 'torch' → pass to aTrainer or a custom training loop. The Hugging Face transformers library’s Trainer class accepts a Dataset object directly for its train_dataset and eval_dataset arguments.

When evaluating models against benchmark datasets, the LLM leaderboard provides scores across common evaluation sets — useful context when deciding which dataset to target for your own benchmark. If you’re weighing whether to fine-tune and self-host versus calling an API, the self-hosting vs API break-even calculator can model the cost crossover by request volume. For raw per-token API cost across providers, use the API cost calculator. And if you plan to run a fine-tuned model locally, VRAM is the hard constraint — the VRAM calculator estimates GPU memory requirements from model size and quantization precision.

Frequently Asked Questions

What is the difference between a Dataset and a DatasetDict?

A Dataset is a single split — one table of rows and columns. A DatasetDict is a dict-like container holding multiple splits, and is the default return type of load_dataset() when you omit the split argument. Access individual splits by key: ds['train'], ds['test'], etc. If you passsplit='train', you get a bare Dataset directly.

How do I load a private dataset from the Hub?

Authenticate first with huggingface-cli login, or set the HF_TOKEN environment variable to your access token. Then call load_dataset('org/private-dataset', token=True). The token=True flag tells the library to use the cached or environment-variable credential. For CI/CD pipelines, set HF_TOKEN as a secret and omit the interactive login step.

Why does load_dataset() take a long time on the first call?

The first call downloads raw data files, converts them to Apache Arrow format, and writes the cache to disk. For large datasets this can take minutes or longer. Subsequent calls on the same machine return the cached Arrow files almost instantly. If you’re on a slow connection or have limited disk space, pass streaming=True to process data on the fly without caching it locally.

Can I use Hugging Face datasets without an internet connection?

Yes. Once a dataset is cached, set the environment variable HF_DATASETS_OFFLINE=1 andload_dataset() reads from the local cache without making any network requests. This is useful for air-gapped servers, reproducible offline runs, or HPC clusters where worker nodes lack internet access but share a network file system with the cache directory.

What file format does the Hub use internally?

Datasets stored on the Hub are served as Apache Parquet files, split into shards. The datasets library downloads these shards and converts them to Apache Arrow (.arrow) files for local caching. You can bypass the library entirely and access the Parquet shards directly via the Hub file browser or by using huggingface_hub.hf_hub_download().

How large can a Hugging Face dataset be?

There is no enforced size limit, and multi-terabyte datasets exist on the Hub (LAION-5B image-text pairs, large Common Crawl snapshots). For datasets above a few gigabytes, the Hub stores data as multiple Parquet shards rather than a single file. Use streaming=True in the datasets library to work with these datasets without downloading them in full, or download specific shards via the data_files argument.