perf(js): speed up Tiktoken.encode

[jeet-dhandha]

Tiktoken.encode does three things on the hot path that are cheap to avoid. Output is byte-for-byte unchanged.

• Rank-map keys. Keys were built with Array.from(bytes).join(",") on every lookup. Mapping each byte to one latin1 code unit (String.fromCharCode) is injective over byte sequences and avoids the intermediate array + comma string. The constructor uses the same helper, so keys stay consistent.
• Regex compilation. encode() compiled the token pattern and the special-token regex on every call. Both depend only on values fixed at construction, so they're compiled once in the constructor.
• BPE merges. The merge loop re-sliced and re-ranked every adjacent pair each round (O(n²) per round). Ranks are now cached, and a merge only refreshes the two neighbours it affects — the algorithm tiktoken's reference implementation uses.

Correctness

Checked the patched encoder against the current release across o200k_base, cl100k_base, p50k_base, r50k_base on a diverse corpus (prose, code, CJK, emoji, whitespace runs, URLs, edge cases) — identical token output on every sample, matching decode round-trips, and identical special-token throw/allow behaviour.

Performance

~2.4x faster on large inputs (100KB encode, o200k_base, Apple M1). No API or memory-model change.