Custom tokenizer
SentencePiece Unigram tokenizer, frozen at 24k vocabulary, built for Arabic, Darija, Latin Darija, English and mixed text.
Current research milestone: MoroLM-56M Base v0.6
A small language model built for Moroccan Darija.
MoroLM is a from-scratch language-model research project focused on Moroccan Darija, Arabic script, Arabizi, code-switching, English and French — designed to run locally and eventually power privacy-first Moroccan applications.
56.6Mparameters
24,000custom tokenizer vocabulary
102.4Mv0.6 token-position exposures
What exists today
The project is no longer just an idea. A custom tokenizer, Llama-like model implementation, training pipeline, fixed evaluation sets, local inference path and a clean base-model lineage now exist.
From-scratch base model
MoroLM-56M Base v0.6 was initialized from random weights and trained using the corrected next-token causal objective.
Local-first runtime
The target stack is browser UI → Go backend → llama.cpp → GGUF, with no cloud inference required.
Project map
The working system is being built as a full language-model pipeline rather than a wrapper around another hosted model.
Raw sourcesDarija + EN + FR + AR
Provenancelicense + holdouts
Tokenizer24k Unigram
Packingtoken blocks
Pretrainingcausal LM
Evaluationheld-out + generation
RuntimeGGUF / llama.cpp
Research status. MoroLM-56M v0.6 is a base model, not a production assistant. It can produce recognizable Darija, English and French structures, but it is still repetitive, factually unreliable and not instruction-tuned.
The bug that reset the lineage
Earlier base checkpoints appeared to improve by loss while generation stayed broken. An audit found a double label shift: the dataset shifted targets once, and the model shifted them again. The effective task was two-token-ahead prediction. v0.6 starts over from random weights with the corrected one-token-ahead objective.
Before
dataset: x = A B C D
dataset: y = B C D E
model shift: logits A B C vs labels C D E
A → C B → D C → E ✗After
dataset: x = A B C D
dataset: y = A B C D
model shift: logits A B C vs labels B C D
A → B B → C C → D ✓Why MoroLM exists
Darija is widely used in everyday Moroccan life but underrepresented in language-model tooling, especially in mixed Arabic/Latin writing and code-switching. MoroLM explores whether a compact local model can become genuinely useful for Moroccan institutions, education and private on-premise applications.