MoroLMA small language model learning Moroccan Darija.
MoroLM is built from scratch: its own tokenizer, model and training code. It handles Arabic script, Arabizi, French and English, and it's designed to run on your own hardware.
It's early. The current model, v0.6, is a raw base model with 56.6M parameters. It can write recognizable Darija, but it isn't a chatbot and shouldn't be trusted for facts.
prompt
Hello, my name is
what it does
Grammatical English structure, but it falls into repeating the same idea.
Where it stands
Some parts work, some are only partly done and some don't exist yet. This is the honest version.
The pipeline
MoroLM is a full language-model pipeline, not a wrapper around a hosted model. Here is how far each stage has come.
How well it's learning
Validation loss on the fixed set fell from 7.25 to 5.22 and is flattening. A perplexity of 185 is a long way from fluent, which matches what the generations show.
Validation loss across the v0.6 run. Losses from v0.1 to v0.5 aren't comparable, because they used the faulty objective.
Why the numbers look rough
The model saw about 102M token positions, which is one pass over a corpus of roughly 101M tokens. That's small for a 56.6M-parameter model that covers four languages and scripts.
Greedy decoding exposes repetition loops. Sampling with temperature 0.8, top-k 50 and top-p 0.9 shows a richer distribution. That doesn't make it smarter.
The bug that reset everything
Earlier checkpoints looked better by loss but still generated badly. An audit found the labels were shifted twice, once in the dataset and again in the model, so the model learned to predict two tokens ahead. v0.6 starts over from random weights with the fix.
Before: two tokens ahead
dataset: x = A B C D
dataset: y = B C D E
model shift: logits A B C vs labels C D E
A → C B → D C → E ✗After: one token ahead
dataset: x = A B C D
dataset: y = A B C D
model shift: logits A B C vs labels B C D
A → B B → C C → D ✓What it can't do yet
- Follow instructions or chat. It has no instruction tuning.
- Give reliable facts. It hallucinates entities and answers.
- Stay on topic from a short prompt.
- Match its Arabic-script Darija in Latin Darija.
- Be used for legal, medical or financial advice, or customer support.
What's next
Why build it
Darija is spoken every day across Morocco but is underrepresented in language-model tools, especially when Arabic and Latin scripts mix. MoroLM asks whether a compact model that runs locally can become useful for Moroccan institutions, education and private on-premise applications.