ASTERIZER Tokenizer

One byte-level BPE tokenizer, frozen once and reused for ~10 years across every model from 100M → 1T params. South-Indian-first (Kannada / Tamil / Telugu / Malayalam), plus code, math, and broad multilingual coverage. Built only from open, license-audited data.

Everything lives in a single file: asterizer.py (config + all stages + CLI).

Quick start

python3 -m pip install tokenizers datasets "huggingface_hub[hf_transfer]" tiktoken transformers sentencepiece

# interactive (asks vocab size + GB):
python3 asterizer.py

export HF_TOKEN=...  # gated permissive code (starcoderdata) - accept the dataset terms on HF first

# -- STEP 1 — instant config/plan check, NO download (seconds) --
python3 asterizer.py --vocab 8K --gb 1 --dry-run --yes --allow-license-review

# -- STEP 2 — tiny END-TO-END test (~1 GB, a few min): proves download -> clean -> corpus -> train ->
#             eval -> smoke all work and write a real tokenizer dir, with NO errors --
python3 asterizer.py --vocab 8K --gb 1 --yes --allow-license-review
#  → if a bucket under-fills on the tiny sample, just bump --gb (e.g. --gb 3) and re-run.

# -- STEP 3 — pin source versions, then the PRODUCTION FREEZE (rented high-RAM CPU box): 32K+64K+128K
#             from ONE shared corpus, all gates on (~14–24 h, ≥256–512 GB RAM, no GPU) --
python3 asterizer.py --lock
python3 asterizer.py --vocab 32K,64K,128K --gb 240 --yes --require-lock --allow-license-review

# validate / inspect an existing tokenizer:
python3 asterizer.py --smoke runs/128K_240gb/tokenizer/asterizer_128K_v1.json

# FULL held-out fertility + compression vs cl100k / o200k / Llama-3 / Qwen / Gemma / Sarvam:
python3 asterizer.py --eval runs/128K_240gb/tokenizer/asterizer_128K_v1.json

# post-freeze edits:
python3 asterizer.py --promote tok.json --names "<|new_ctrl|>,<|tool2|>" --out tok2.json # size FIXED, sidecars auto-synced
python3 asterizer.py --extend tok.json --new-data new.jsonl --num-new 2000 --out tok2.json # GROWS vocab

Each build writes a complete HuggingFace directory: tokenizer.json + asterizer_<v>_v1.json + tokenizer_config.json + special_tokens_map.json

• spec_hash.txt + fertility_report.json , with the shared corpus provenance in manifest.json .

Frozen design (current spec)

Vocab profiles ( vocab_size = 256 bytes + 32 named + reserved + learned ):

Freeze policy (per the review): use 64K for 1B–4B, 128K for 7B+. At d_model=2048 a 131,072 vocab is ~268M tied-embedding params vs ~134M for 64K — a heavy tax on a 1B model.

Corpus (40% South-Indian, scales to --gb ): South-Indian 40% · code 17% (StarCoderData, 15 languages, license review-tagged) · english 10% · other-Indic 8% (hi/bn/gu/mr/pa/or) · european 5% · more-european 3.5% (el/tr/pl/nl/sv/ka/hy) · SEA 3.5% (th/vi/id/km/my/lo) · cjk 4.5% · semitic 3.8% (ar/fa/ur/he) · cyrillic 2.5% · african 1.5% (sw/am/yo/ha) · coverage 0.5% (bo/si) · math 1.5%. 26 scripts, ~40 languages. Every bucket is coverage-gated. --gb N scales to exactly N GB.

Professor review (2026-06-26) — resolution

Professor review #3 (2026-06-26) — final hardening

Professor review #4 (2026-06-26) — resume safety, exact sizing, honest license

CHANGELOG (newest first — always appended)

2026-06-26 — Professor-loop pre-freeze review (gpt-5.5 reviewer, auto-logged)

A two-brain refinement loop hardens asterizer.py round by round: worker = the build author, reviewer = gpt-5.5 via test1/auto_refine_loop.py (ARM proxy). Each round sends the summary + full source, gets a JSON verdict, and applies fixes. The reviewer's #1 blocker every round is "no built artifacts exist", so APPROVED requires actually running the 240 GB build — the code/gate hardening below is what the loop produced. test1/professor_review.py appends each round to the table below automatically (logged here in the CHANGELOG, no separate log files).

Fixes applied across rounds (all in asterizer.py , validated offline):

• R1: license_gate (--allow-license-review); config-fingerprint checkpoints; exact _gb_tag run-dir keys; shuffled sampling + _sample_digest.json ; hash-based held-out split written to raw/heldout/*.jsonl ; smoke decodes with skip_special_tokens=False + special-literal cases; --extend gated behind ASTERIZER_EXPERIMENTAL .
• R2 (real bugs caught): per-bucket .done markers store the fingerprint (stale corpus can't be reused); the post-clean gate runs even on resume (was bypassable). Plus per-source coverage floor, eval-threshold freeze gate, whole-dir SHA256SUMS.txt , manifest env + NOTICE.txt .
• R3: eval-gate-bypass-on-resume fixed (freeze folded into the tokenizer fingerprint); run_stage validates expected outputs exist; vocab-conformance gate; lock inventory check; provenance copied into the tokenizer dir; promote writes promotions.json + spec_hash before checksums.
• R4 (open): remaining asks are stricter-enforcement/operational — per-file SPDX, corpus-hash checkpoint verification, holdout-after-cleaning, dependency-version refusal, full source-tuple lock match, and (always) building the real artifacts.

2026-06-26 (round 4) — Resume safety + exact sizing + honest license (review #4)

1. Coverage gate no longer trappable on resume

• Before: each bucket's .done marker was written before the coverage gate ran, so an under-filled (e.g. gated/failed) bucket got checkpointed and was skipped on every resume while the gate kept failing — a permanent fail loop.
• After: the floor is checked before the marker, so an under-filled bucket is never checkpointed; the final gate also clears the marker of any bucket under floor so a resume re-downloads it. Validated with a stubbed streamer (the bad bucket re-downloads and passes on the second run).

2. Exact corpus sizing

• Before: REFERENCE_TOTAL_GB = 60 but the source quotas summed to 60.5 (Urdu added +0.5), so --gb 240 actually scaled to ~242 GB.
• After: REFERENCE_TOTAL_GB is recomputed as the exact sum of all source quotas, so --gb N scales to exactly N GB and never drifts when sources change.

3. Honest code license tag

• Before: StarCoderData sources were tagged permissive — an upstream claim, not a filter we ran.
• After: tagged permissive(bigcode-filtered; review) ; the bucket comment documents that a hard commercial freeze should accept the upstream filter explicitly or filter on a per-file license column (e.g. the-stack-dedup). Surfaced in manifest.json + sources.lock.json .

2026-06-26 (round 3) — Final hardening (professor review #3)

1. Pre-resolved source lockfile ( --lock → sources.lock.json )

• Before: dataset commit SHAs were resolved into manifest.json after the build.
• After: --lock resolves every source's HF commit SHA + license decision into sources.lock.json before download; the build pins load_dataset(revision=...) from it and --require-lock aborts unless it is fully pinned. The 240 GB pull is now reproducible by construction.

2. Coverage floors on EVERY bucket

• Before: only the South-Indian bucket was gated, at 50% of target.
• After: every bucket must hit a floor (default 60%, required 85%, low-resource buckets a smaller explicit min_frac ) or the build aborts — a silent source/auth failure can't gut any script.

3. Permissive-only code (white-label clean)

• Before: codeparrot-clean (Python, mixed GitHub licenses, tagged mixed-github(review) ).
• After: bigcode/starcoderdata (already license-filtered to permissive; 15 languages; gated → HF_TOKEN ). codeparrot retained as a clearly-commented dev fallback. the-stack-v2-dedup stores content as S3 pointers → not plain-streamable, so StarCoderData is the practical pick.

4. stage_promote keeps the whole HF dir in lockstep

• Before: it edited the tokenizer JSON and told you to update sidecars by hand.
• After: it auto-updates tokenizer_config.json + special_tokens_map.json and re-saves the canonical tokenizer.json — IDs and vocab size still unchanged. Validated 311→311, sidecars synced.

5. Real eval harness ( --eval )

• Before: fertility_report.json = 23 one-sentence probes.
• After: per-language held-out corpora (fresh dataset streams, or local --evaldir files) + domain probes (legal/biomed/finance/yaml/url/diff/ZWJ-emoji/noisy-web), measured as bytes/token + tokens/word head-to-head against cl100k, o200k, Llama-3, Qwen, Gemma, Sarvam. Runs after each freeze build (non-fatal) and standalone via --eval .

6. Urdu + ZWJ coverage

• Added urd_Arab to the corpus (semitic bucket) and an Urdu probe; added a real ZWJ-emoji family ( 👨‍👩‍👧 ) to smoke + eval so the preserved U+200D is actually exercised.

2026-06-26 (round 2) — Freeze-gate hardening (professor review #2)

1. Smoke gate now refuses stale / incomplete artifacts

• Concept: a freeze gate that passes a stale artifact is worse than none.
• Before: the old 64K artifact still smoke-PASSED despite GPT-2 pretokenization, 24 specials, 1000 reserved, and no sidecars.
• After: stage_smoke fails if any HF sidecar is missing, a STALE_* marker exists in the dir/parent, named specials ≠ 32, or reserved count ≠ the profile's. Verified: old 64K now FAILS with all 6 reasons; fresh builds still PASS.

2. Required-bucket coverage gate

• Concept: a silently-failed source must not gut the priority data.
• Before: source failures were non-fatal → coverage could collapse unnoticed.
• After: South-Indian bucket is required ; stage_download records actual per-source bytes ( raw/_download_stats.json ) and aborts if a required bucket is under MIN_REQUIRED_FRAC (50%) of target.

3. Manifest records actual bytes + resolved dataset commit SHAs

• Before: revision plumbing existed but all sources were None → no real pinning; no actual bytes.
• After: write_manifest records actual_bytes per source and resolves each dataset's commit SHA via HfApi into resolved_revision . Copy resolved_revision → a source revision to hard-pin a rebuild. Corpus sha256 remains the primary anchor.

4. Code license (open decision, not a hard blocker)

• codeparrot-clean is Python + mixed GitHub licenses. For a tokenizer (a statistical merge table, not redistributed code/weights) mixed-license training data is widely treated as non-infringing (GPT-4/Llama did the same). For maximum white-label cleanliness, set HF_TOKEN and swap the code bucket to a permissive subset ( bigcode/the-stack-v2-dedup ). The license is tagged mixed-github(review) and surfaced in manifest.json so it's a conscious choice.

2026-06-26 — Freeze-readiness (professor review)

1. Indic ZWJ/ZWNJ preservation

• Concept: zero-width joiners are semantic, not noise.
• Before: ZERO_WIDTH stripped U+200C (ZWNJ) and U+200D (ZWJ).
• After: both preserved; only ZWSP/BOM/WordJoiner/SoftHyphen stripped.
• Why: ZWNJ/ZWJ control Indic conjunct formation, Persian/Urdu orthography, and emoji ZWJ-sequences. Stripping them silently corrupts spelling and breaks emoji.

2. Smoke test is a HARD freeze gate

• Concept: a failed validation must stop the freeze.
• Before: lambda: (stage_smoke(...) or True) → stage marked done even on FAIL.
• After: _smoke_gate() calls sys.exit(...) on failure; stage is not checkpointed.
• Why: a frozen-10-year artifact cannot be allowed to ship if round-trip/compat fails.

3. Reserved-slot promotion vs vocab growth (split)

• Concept: the reserved-slot promise = add control tokens WITHOUT changing IDs or size.
• Before: stage_extend appended tokens (grew vocab_size ), violating the promise.
• After: new stage_promote renames free <|reserved_i|> slots in place (IDs + size UNCHANGED); stage_extend kept but loudly labeled as vocab-growth (needs embedding resize + continue-pretrain). New --promote / --extend CLI.
• Why: promoting a reserved slot must never shift a learned-merge ID. Validated: 907 → 907.

4. Dataset revision pinning + license tags + manifest

• Concept: reproducibility and commercial-safety must be auditable, not asserted.
• Before: load_dataset streamed by name (no revision); no license metadata.
• After: per-source license + optional revision ; manifest.json records every source, license, revision, seed, and the corpus sha256 (the true reproducibility anchor).
• Why: a frozen artifact needs provenance; re-training from the same corpus.jsonl is deterministic, so its hash pins the build.

5. Corpus universality (12 → 26 scripts)

• Concept: "best in every language" needs every major script present so merges form.
• Before: 9 buckets, ~12 scripts.
• After: 13 buckets, 45 sources, 26 scripts — added Gujarati/Marathi/Punjabi/Odia, Thai, Vietnamese, Indonesian, Khmer, Burmese, Lao, Greek, Turkish, Polish, Dutch, Swedish, Georgian, Armenian, Swahili, Amharic, Yoruba, Hausa, Tibetan, Sinhala. South-Indian stays 40%.
• Why: byte-level guarantees no UNK, but good fertility needs the script in the training corpus. Missing-config sources fail non-fatally (logged), so this is safe.

6. Per-language/script/domain fertility report

• Concept: freeze evidence must be measurable per language and domain.
• Before: only an aggregate corpus fertility number.
• After: fertility_report.json with tokens, bytes/token, tokens/word for 23 probes (all scripts + legal/biomed/finance/social-code-mixed/code-diff).
• Why: the review requires held-out fertility per language/script/domain before freeze.

7. FORGE (cross-repo) exactness

• Concept: vocab size must be identical across tokenizer, configs, packer, model cards.
• Before: configs vocab_size = 64000/128000; packer uint16 if vocab < 65536 (so the real 65536 vocab wasted uint32 ).
• After: all 8 configs → 65536 / 131072; packer condition <= 65536 (65535 + uint16 max, fits).
• Why: a 64000 vs 65536 mismatch corrupts embedding sizing; uint32 at 64K doubles token-bin storage for nothing.

8. Stale artifacts quarantined

• Before: runs/32K_6gb and runs/64K_12GB looked freezable but predate every change.
• After: STALE_DO_NOT_FREEZE.txt in each; real freeze rebuilds on the rented box.

2026-06-26 — SOTA audit (pre-freeze quality)

A. o200k Indic-aware pre-tokenizer (the critical fix)

• Before: ByteLevel(use_regex=True) = GPT-2 regex; \p{L}+ excludes \p{M}+ , so Indic marks split off the consonant at the pre-token boundary → BPE cannot merge syllables.
• After: GPT-4o o200k regex (letter classes include \p{M} ) via Split(Regex) + ByteLevel .
• Why / proof: ಕನ್ನಡ 3→1 pre-token, తెలుగు 6→1, ಅತ್ಯಾಧುನಿಕ 9→1; tiny-build fertility Kannada 4.33→1.17, Tamil 7.0→1.80, Telugu 6.6→1.80, Malayalam 8.0→2.50.

B. Special tokens 24 → 32 (append-only IDs 24–31)

• Added <|im_start|>/<|im_end|> (ChatML), <|endofprompt|> , <|repo_name|>/<|file_sep|> (repo-level code), <|image_start|>/<|image_end|>/<|image_pad|> (vision patches).
• Reserved reduced by 8 per profile so named+reserved stays a power of two → learned merges unchanged (128K still 128,768).

C. Complete HF directory + ByteLevel post-processor

• Emits tokenizer.json (canonical name AutoTokenizer needs), tokenizer_config.json , special_tokens_map.json with a ChatML chat_template ; unk_token=None , bos/eos/pad declared.
• Compat test now loads the dir via AutoTokenizer.from_pretrained and renders the chat template; gates PASS/FAIL.

2026-06-25 — Shared-corpus refactor + UX

• Download/clean/corpus run once into runs/_data_<gb>gb/ ; 64K and 128K train from the same corpus. --vocab 64K,128K comma list. Download progress bar with %/rate/ETA.
• Saturation analysis: tokenizer quality plateaus ~150–240 GB; >300 GB is wasted spend.

Build it (rented high-RAM CPU box, no GPU)

The full 240 GB build needs ~150–200 GB RAM peak (BPE holds word counts in RAM). Rent a CPU box with ≥ 256–386 GB RAM, ~1 TB disk. Then:

export HF_TOKEN=...                  # gated permissive code (starcoderdata)
python3 asterizer.py --lock          # pin source SHAs -> sources.lock.json (commit it)
python3 asterizer.py --vocab 64K,128K --gb 240 --yes --require-lock

Freeze checklist before shipping: sources.lock.json fully pinned · every bucket passed its coverage floor · smoke PASS (gate) · fertility_report.json reviewed per language and vs baselines · manifest.json licenses audited (all permissive/open) · spec_hash.txt recorded · FORGE vocab_size matches tokenizer.get_vocab_size().