birgermoell/oellm-dpo-metadataset

OpenEuroLLM DPO metadataset

A lightweight, versioned source of truth for building preference-training data for OpenEuroLLM. It contains metadata and planning decisions—not copies of upstream training examples.

The catalogue pins each upstream revision and records its license, size, language coverage, pair schema, overlap family, decision, risks, and required transformations. Upstream licenses and terms still apply. The Apache-2.0 license in this repository covers only the original catalogue metadata, documentation, and build code.

Start here

Config	Purpose	Rows
sources	Candidate datasets, pinned revisions, decisions, risks, and required actions	16
recommended_mix	A compute-conscious 250k-pair pilot recipe	7
language_coverage	One row for every official EU language	24
quality_gates	Blocking and warning checks for materialization	18

from datasets import load_dataset

sources = load_dataset(
    "birgermoell/oellm-dpo-metadataset", "sources", split="train"
)
pilot = load_dataset(
    "birgermoell/oellm-dpo-metadataset", "recommended_mix", split="train"
)

ready = sources.filter(lambda row: row["decision"] == "recommended")
print(ready.to_pandas()[["source_id", "revision", "required_actions"]])

Recommended pilot

The proposed first materialization is 250,000 unique preference pairs, not a blind concatenation:

Component	Weight	Pairs	State
Translated multilingual core	35%	87,500	Rejudge before use
Deduplicated English anchors	25%	62,500	Ready after filters
Multi-attribute quality	12%	30,000	Transform and validate labels
Native multilingual	10%	25,000	Reconstruct ranked trees
Human votes	8%	20,000	Access/transform/decontaminate
On-policy math and code	5%	12,500	Generate rollouts and verify
EU safety and civic	5%	12,500	Blocking source gap

The percentages are hypotheses to test, not immutable truth. Publish every materialized mix with source counts, language counts, hashes, judge versions, and evaluation results.

Selection policy

The decision field is deliberately strict:

• recommended: useful in the first pilot after its listed quality actions.
• conditional: promising, but blocked on reconstruction, access, licensing, decontamination, or overlap handling.
• prompt_seed_only: not DPO data; suitable for generating checkpoint-specific rollouts that are then verified or judged.
• exclude: known incompatible for the intended broadly reusable flagship mix.
• protected_eval: must never enter training.

Two especially important rules:

1. Treat Tülu/UltraFeedback derivatives as one overlap family. Selecting several without cross-source deduplication inflates apparent diversity and overweights the same prompts.
2. Treat translated preference labels as untrusted until both answers have been independently rejudged in the target language. Fluent translation does not guarantee preservation of the original preference margin.

Why on-policy pairs matter

The strongest next tranche is small, verified, checkpoint-specific data rather than another large generic aggregate. For math and code:

1. Sample several responses from the exact post-SFT OpenEuroLLM checkpoint.
2. Evaluate math with canonical-answer or symbolic verifiers and code in an isolated test sandbox with hidden tests.
3. Form pairs only when outcomes are decisive. Prefer a correct response over an incorrect one; use a quality judge only to break ties between equally correct responses.
4. Keep answer keys, tests, verifier traces, and judge rationales outside model-visible fields.
5. Split by problem family and translation group before rollout generation to prevent leakage.

The prompt seeds currently listed are birgermoell/oellm-math-rlvr and birgermoell/oellm-code-rlvr. They are not counted as preference pairs.

Materialization contract

A downstream DPO release should normalize to:

pair_id, messages, chosen, rejected, language, domain,
source_id, source_revision, source_row_id, source_license,
preference_origin, judge_or_verifier, preference_margin,
rollout_model, transformation_log, overlap_family, split_group

At minimum, every chosen/rejected pair must have the same prompt context, a non-tied preference, an allowed-use provenance chain, a protected-evaluation decontamination result, and a stable split group. Use the quality_gates config as the release checklist.

OpenEuroLLM-specific findings

• openeurollm/Dolci-Instruct-DPO-translated is the largest directly aligned multilingual source, but its 3.26M rows should be sampled by language and rejudged rather than consumed wholesale.
• Existing OpenEuroLLM multilingual SFT work indicates that source quality and explicit target-language data matter; multilingual ability should not be assumed to transfer reliably from English replay alone.
• openeurollm/battle-annotations and openeurollm/ArenaHard-EU-v0 are reserved here as evaluation/judge-calibration assets, not training sources.
• The current coverage map has no direct pair source for Estonian, Irish, Croatian, Lithuanian, Latvian, Maltese, or Slovenian. These languages should receive native prompt collection and on-policy generation rather than translation-only backfilling.

Evidence and methodology

Dataset revisions, licenses, configurations, and sizes were verified against the Hugging Face dataset API and datasets server on 2026-09-14. Important methodological references:

• Tülu 3: careful prompt curation, on/off-policy DPO, and decontamination
• OpenEuroLLM post-training repository and TRL DPO configuration
• OpenEuroLLM multilingual SFT experiment discussion
• OpenAssistant OASST2 ranked message-tree structure
• Chatbot Arena human preference data

This is a planning catalogue, not legal advice or an endorsement of every source. Recheck upstream revisions and terms before each materialization.

Rebuild and validate

python scripts/build_catalogue.py
python scripts/build_catalogue.py --check
python -m unittest discover -s tests -v

manifest.json records the catalogue version and SHA-256 checksum of every generated Parquet file.