MoroLMTokenizer
The frozen 24,000-token SentencePiece Unigram tokenizer used by MoroLM.
Work in progress: this page describes MoroLM-56M v0.6 and will change as the project moves.
Tokenizer v0.1#
The tokenizer is a custom SentencePiece Unigram model with an exact vocabulary size of 24,000. It is frozen and should not be casually retrained because every checkpoint depends on its token IDs.
| Property | Value |
|---|---|
| Model type | Unigram |
| Vocabulary size | 24,000 |
| Character coverage | 1.0 |
| Byte fallback | Enabled |
| split_digits | False |
| pad / bos / eos / unk | 0 / 1 / 2 / 3 |
Special tokens#
<|bos|>
<|eos|>
<|pad|>
<|system|>
<|user|>
<|assistant|>
<|memory|>
<|eot|>Canonical chat representation#
<|bos|><|system|>...<|eot|>
<|user|>...<|eot|>
<|assistant|>...<|eot|>Language markers are not required from users. The tokenizer is intended to let the model infer language/script from normal input.
Verified behavior#
Round-trip tests covered mixed Arabizi and Arabic text, including strings such as wach 3lach ma jitich خويا?. Tokenizer parity with llama.cpp was also verified during the earlier export work.