MoroLMData & provenance
The v0.6 corpus, held-out sets, licensing discipline and known repetition issues.
Work in progress: this page describes MoroLM-56M v0.6 and will change as the project moves.
v0.5 training candidate#
The v0.6 cleanstart reused the existing v0.5 training candidate because the corpus encoding was verified as healthy UTF-8; the critical failure was in label alignment, not the stored text.
Approximate text mixture#
| Component | Approx. share |
|---|---|
| English | 39.68% |
| French | 19.84% |
| Arabic / MSA | 14.88% |
| Fresh Moroccan Arabic wiki material | 4.77% |
| Replay Darija Arabic | 7.94% |
| Latin Darija / Arabizi | 9.92% |
| Code-switch replay | 2.98% |
Fixed holdouts#
Historical v0.1 data was separated into clean training, validation and test snapshots. External evaluation sets were also created for Moroccan code-switching and Morisco Arabizi gold examples. These holdouts should never be mixed back into training.
Source policy#
The project tracks dataset provenance and does not treat a dataset card as a blanket legal guarantee. Sources with unclear, conflicting or non-commercial terms are held out of the commercial-capable lineage until provenance is resolved.
Template concentration#
Generation audits show that many Moroccan locality records share very similar article templates. v0.6 can recombine pieces of those templates rather than always copying one record verbatim. This is useful evidence of learned structure, but it also reveals a data-distribution problem: repeated template families can dominate a 56M model.