birgermoell/oellm-9b-threeway-dpo-full-v1

OELLM 9B three-source DPO — full one-epoch checkpoint

This is the final full-weight checkpoint of a one-epoch Direct Preference Optimization (DPO) run on LUMI. The starting checkpoint was Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft. The run finished on 2026-10-05 at 18:17 UTC, with Slurm job 22441061 exiting successfully. The saved model comprises eight float32 safetensors shards (36,407,791,616 bytes of tensor data); training used bf16. It is not a LoRA adapter and has no optimizer state.

This is a research model, not a production-validated release. DPO optimizes preferences; this stage did not teach reasoning traces. The inherited chat template may emit an empty <think> block, which is not evidence of reasoning capability. Independent multilingual evaluation is reported below; it is not a deployment validation or safety certification.

Inference

The release includes the working tokenizer and chat template used by the run. Use the tokenizer from this repository, not a fresh tokenizer download from the base-model repository.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "birgermoell/oellm-9b-threeway-dpo-full-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id, torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": "What is the capital of Finland?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

The 2,048-token value below is the DPO training sequence limit, not a measured maximum inference context length for this checkpoint. Review the base-model card for architectural context and general limitations.

Data and preprocessing

How much data was used? This was the full eligible three-source run, not the earlier 1% experiment. The trainer used 1,441,547 training preference pairs over one epoch, plus 15,824 separate validation pairs. A pair means one prompt with a chosen and a rejected response; these are counts of pair records, not tokens or unique prompts. The preparation step staged 1,484,738 training pairs, of which 43,191 (2.91%) were removed by the trainer's empty/length filters before optimization. No additional per-source cap or equal-size resampling was applied.

Each dataset was converted to conversational prompt, chosen, rejected fields. The sources were combined at their natural sizes (configuration weight 1.0 for each source), so the mixture was not balanced. “Used for training” below is the count after trainer filtering; “staged” is the count before it. Validation pairs were held out and were not training items. The exact staging manifest and compressed-file SHA-256 checksums accompany this release as data_manifest.json and data_checksums.sha256.

Source	Used for training	Share of training pairs	Staged train	Removed by trainer filters	Staged validation	Used for validation	Source license
NVIDIA HelpSteer3, preference	28,895	2.00%	36,299	7,404	1,920	1,560	CC BY 4.0
OpenEuroLLM Dolci-Instruct-DPO-translated	1,352,763	93.84%	1,388,132	35,369	14,029	13,656	Apache 2.0
Argilla UltraFeedback cleaned	59,889	4.15%	60,307	418	610	608	MIT
Total	1,441,547	100%	1,484,738	43,191	16,559	15,824

The percentages are calculated from post-filter training pair counts; rounding makes the individual displayed shares sum to 99.99%. Dolci's 93.84% share is important when interpreting the checkpoint: “three-source” does not mean three equally represented sources.

For HelpSteer3, the sign of overall_preference determined the chosen answer; preference magnitude was not used, and 2,257 tied rows were removed. Its native validation split was retained. For Dolci and UltraFeedback, every 100th source row was held out deterministically for validation. Dolci string and message-list responses were normalized into assistant messages; the staging code reads pinned Dolci language files explicitly, preferring Parquet with JSONL fallback. UltraFeedback responses had a duplicate leading user turn removed. Empty or overlength pairs were dropped by the trainer, with a 2,048-token sequence cap and truncated_pair_action: drop. See the bundled stage_threeway_dpo_full.py for exact transformations.

Specifically, staging read the available HelpSteer3 preference train and native validation rows (not its unrelated task variants); read every row in the explicitly selected Dolci files for the 12 languages below at the pinned revision; and streamed the UltraFeedback cleaned train split. The Dolci and UltraFeedback validation allocation was applied before the trainer's pair filtering. The 2,257 HelpSteer3 ties were excluded at staging and are not included in the “removed by trainer filters” column. One epoch means each retained training pair was scheduled once, aside from any final distributed-batch padding; the card does not claim a measured token count or absence of duplicate content across sources.

Dolci used 12 languages, not all 36 OpenEuroLLM target languages. Its per-language counts below are staged train/validation pairs before trainer filtering; language-specific post-filter counts were not recorded.

Language	Train	Validation	Language	Train	Validation
Czech (cs)	116,249	1,175	German (de)	115,264	1,165
Greek (el)	101,516	1,026	English (en)	243,752	2,463
Spanish (es)	126,629	1,280	Finnish (fi)	109,131	1,103
French (fr)	14,931	151	Italian (it)	87,439	884
Polish (pl)	119,624	1,209	Romanian (ro)	116,510	1,177
Swedish (sv)	121,715	1,230	Ukrainian (uk)	115,372	1,166

The Dolci source was pinned to Hugging Face revision a57841a6bbabfed8ca2597438995c2991e5b73b7. HelpSteer3 came from pre-existing staged LUMI preference files, and the UltraFeedback source was not revision-pinned. Consequently, the source URLs and script alone may not reproduce identical staged bytes if those sources change. The included SHA-256 file identifies the exact six staged inputs used in this run; the staged files remain on LUMI at /flash/project_465002530/users/bmoell/helpsteer3-dpo/artifacts/data/threeway-full/. These data files are not included in this model repository.

Training recipe and reproducibility

The exact frozen run configuration, Slurm launch script, staging script, and final trainer results are included in this repository as training_config.yaml, training_slurm.sbatch, stage_threeway_dpo_full.py, and all_results.json. Training code is public in BirgerMoell/post-training; the run originated from upstream commit db9d560011cb4502adaffe1ba3a94b87f9f404f4 plus the three-source DPO changes subsequently published in the fork. Use the frozen files here as the authority for this particular run, including cluster paths.

Setting	Value
Objective	TRL DPO, sigmoid loss, beta 0.1, implicit reference = frozen SFT starting model
Base model	Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft, revision 85bf18fb4f0bee6ac6270f06b1d1c6b3be200f31
Epochs / optimizer steps	1 / 11,263
Learning rate / schedule	5e-7, cosine, 3% warmup
Adam / regularization	betas (0.9, 0.999), epsilon 1e-8, weight decay 0
Batch	per-device 1, effective global 128
Sequence processing	Qwen3 chat template; max sequence length 2,048; drop truncated pairs
Precision / efficiency	bf16, FlashAttention 2, activation gradient checkpointing, FSDP
Random seed	42
Hardware	LUMI small-g, 4 nodes × 8 AMD MI250 GPUs = 32 GPUs; 56 CPUs and 480 GB RAM per node
Checkpoints	model-only every 1,000 steps, retaining two; final full model saved separately
Runtime	43 h 21 min wall time including final evaluation; about 1,387 GCD-hours (32 × 43.3528 h), equivalent to about 694 LUMI-billed MI250X GPU-hours before any resource-overallocation adjustment
Software	TRL 0.28.0; Transformers 4.57.3; PyTorch 2.9.1+rocm6.4; Datasets 4.6.0; Tokenizers 0.22.2

The local LUMI base-model tokenizer configuration was repaired before training to resolve a tokenizer-loading problem. The released tokenizer files are the ones saved with the trained model. For a fresh reproduction from the base-model revision, use the bundled base_tokenizer_config_used.json and tokenizer.model from this release when preparing the base-model directory. This tokenizer adjustment matters: simply loading the original base revision may not reproduce the training startup behavior. The LUMI container was laif-rocm-6.4.4-pytorch-2.9.1-te-2.4.0-fa-2.8.0-triton-3.2.0.sif with the project pylibs-overlay; the Slurm script captures the invocation and environment variables.

To rerun: obtain the pinned base model and working tokenizer; stage the three datasets with the included staging script (or use the six LUMI files and verify their checksums); update LUMI paths in the frozen config; use the specified software/container; submit the included Slurm script from the checked-out training repository. Because HelpSteer3 and UltraFeedback source revisions were not pinned, byte-for-byte reproduction requires the six staged files or a new staging output matching their hashes. Because only model weights were retained, this artifact cannot resume the original optimizer trajectory.

Held-out preference evaluation

After one epoch, the trainer evaluated 15,824 held-out pairs after filtering. The final evaluation took 639.9 seconds. The complete machine-readable metrics are in eval_results.json and all_results.json.

Metric	Result
Held-out DPO preference accuracy (eval_rewards/accuracies)	0.8780
Held-out reward margin (eval_rewards/margins)	2.9930
Held-out DPO loss (eval_loss)	0.2796
Overall training loss (train_loss)	0.3087

Preference accuracy measures how often the trained policy's DPO reward favors the annotated chosen response on this held-out mixture. It is not a general benchmark accuracy, direct comparison with the base model, or proof of better reasoning, multilingual quality, or safety. The separate step-9,000 checkpoint is available for within-run comparison.

Independent multilingual evaluation

The final checkpoint scored 84.0% on the public EUROPA Eval v0.1 run: all 432 items across 36 languages were scored. The starting SFT checkpoint scored 76.2% in an earlier run of the same benchmark with the same judge, a descriptive difference of +7.8 percentage points, not a controlled training-effect estimate. The target was pinned to revision c8d1ca3d965e5ee8a601415bdf4a61f9969d96aa, run locally through Hugging Face Transformers on Apple MPS with bfloat16, temperature 0, seed 42 and reasoning off. Open-ended responses were scored by the independent Ollama judge gemma4:31b, pinned to sha256:6316f0629137b426c9d9b853ffc4c8209589f30ee39aebede6285096c0ff47e7, also at temperature 0 and seed 42; other items used declared deterministic scorers.

The result is uneven: quantitative reasoning scored 51.0%, and the weakest language pack scored 28.5%. The run recorded nine malformed-format responses. The benchmark is a small, partly machine-translated diagnostic pilot, and LLM-judge scores need calibration against native-speaker review. These results do not establish general multilingual reliability or safety.

EUROPA Alignment safety pilot (2026-10-06)

The final checkpoint scored 89.3% on the complete EUROPA Alignment v0.1 run: 576/576 items across 36 languages, with no generation or scoring errors. The public answer explorer shows the prompts, final responses, item scores, and concise judge rationales. The raw run artifact records the exact protocol and per-item evidence.

Alignment domain	Final DPO	Step 9,000	Starting SFT
Agentic control	91.0%	91.4%	60.9%
Dangerous-capability boundaries	90.1%	90.5%	74.2%
High-stakes advice and fairness	82.6%	82.0%	78.4%
Human wellbeing	94.0%	94.3%	88.0%
Privacy and security	87.9%	86.5%	79.9%
Refusal calibration	85.2%	87.7%	78.5%
Overall	89.3%	89.5%	77.3%

The final checkpoint is 12.0 percentage points above the earlier SFT run, but 0.2 points below the step-9,000 checkpoint. These are descriptive benchmark differences, not evidence that extra DPO training caused a change. Importantly, the final run still had 39 safety-critical gate failures (step 9,000: 36; SFT: 71), and its weakest language pack, Georgian, scored 21.9%. Refusal calibration fell from 87.7% at step 9,000 to 85.2% at the final checkpoint, even as privacy/security rose from 86.5% to 87.9%. The final run also recorded five malformed-format responses, six prompt echoes, and three repeated spans.

For this run, target weights were pinned to revision c8d1ca3d965e5ee8a601415bdf4a61f9969d96aa; inference used Hugging Face Transformers on Apple MPS, bfloat16, reasoning off, temperature 0, seed 42, and no KV cache. All 504 open-ended responses were scored by an independent local Ollama judge, gemma4:31b at digest sha256:6316f0629137b426c9d9b853ffc4c8209589f30ee39aebede6285096c0ff47e7, with reasoning off, temperature 0, and seed 42; the other 72 items used deterministic scoring. The step-9,000 run used the same suite, target protocol, and judge. The SFT run preceded a terminal-EOS decoding fix, so its comparison is less tightly controlled.

This is a small, translation-review pilot, with only 16 non-operational scenarios per language and machine-translated prompts outside English. It does not test safeguard-suppressed dangerous capability, substitute for native-speaker/human safety review, or certify the model as safe for deployment. Inspect the critical failures and application-specific risks before drawing conclusions.

Limitations, licenses, and attribution

The mixture's source imbalance, heterogeneous preference labels, unpinned HelpSteer3/UltraFeedback inputs, and validation drawn from the same dataset families limit conclusions. The training cap was 2,048 tokens despite the base model's longer-context naming. The pilot above is not a substitute for broader multilingual, reasoning, and safety evaluation before deployment or claims of improvement. The base model and this release use Apache 2.0; the source datasets retain their respective licenses listed above. Credit NVIDIA, OpenEuroLLM, Argilla, and the original upstream contributors when reusing the data or model.