Changelog

A concise version-by-version history of the MoroLM research project.

Work in progress: this page describes MoroLM-56M v0.6 and will change as the project moves.

2026-10 — v0.6 cleanstart#

• Reset base lineage to random initialization.
• Corrected TokenBlockDataset to stop pre-shifting labels.
• Verified exact one-token-ahead learning with a tiny overfit test.
• Completed 12,500-update, ~102.4M token-position clean base run.
• Final fixed validation loss: 5.2239.
• Observed grammatical English/French and recognizable Arabic/Latin Darija generation.

2026-10 — alignment investigation#

• Audited model forward pass, causal attention, trainer and dataset.
• Identified the double-shift bug: dataset + model both shifted labels.
• Confirmed old base lineage was optimizing a two-token-ahead task.

v0.5 — fresh-data experiment#

• Assembled 646k-record training candidate with ~101.46M stream tokens including EOS.
• Added major fresh English, French, Arabic/MSA and Moroccan Arabic material.
• Run completed but generation remained broken because the hidden objective bug was still present.

v0.3–v0.4#

Expanded Darija exposure and replay weighting, improved the historical faulty validation objective, but did not produce usable language generation.

v0.1–v0.2#

Established the custom 56M architecture, tokenizer, training loop, checkpoints, initial Arabizi additions, export work and llama.cpp parity. These versions are kept as historical research artifacts.