TypoFormer Banner

TypoFormer (Stage 2 Open Weights)

High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction
_{Created & Authored by Md. Fahim Sarker Mridul}

Overview

TypoFormer is a purpose-engineered, character-aware neural backbone designed from the ground up to solve two conflicting natural language problems inside a single unified 94M parameter trunk:

Causal Next-Word Autocomplete: Low-latency autoregressive token prediction under strict sub-15ms local keystroke budgets without cloud connectivity.
Non-Autoregressive Grammar & Typo Correction (GEC): High-precision edit-tag sequence labeling that repairs punctuation slips, confused vocabulary, subject-verb agreement, and severe typos without generative hallucination.

By fusing Byte-Level BPE subwords with Multi-Scale Character 1D-Convolutions through a learned gating highway, TypoFormer eliminates out-of-vocabulary token fractures and delivers enterprise-grade real-time writing assistance completely offline on standard desktop CPUs.

This repository publishes both:

Trainable PyTorch Safetensors Checkpoints with architecture source code (modeling_typoformer.py) and training runner (finetune.py) for community researchers to continue training or fine-tune on custom domain datasets.
Quantized INT8 ONNX Runtimes for instant edge deployment in desktop environments.

Model Architecture

TypoFormer Full Internal Neural Architecture Diagram

Figure 1: TypoFormer Full Internal Neural Architecture: Fusing byte-level subwords with multi-scale character convolutions, Pre-RMSNorm grouped-query attention (GQA), rotary embeddings (RoPE), SwiGLU feed-forward networks, and multi-task dual heads for offline autocomplete and grammar error correction (Visualized in NVIDIA Emerald Green #76B900).
[View High-Resolution 300 DPI Architecture Diagram (PNG)]

1. Dual-Channel Gated Input Highway

Standard tokenizers fail when words are corrupted: mistyping "definitely" as "definatly" converts a single semantic token into multiple rare subword fragments. TypoFormer prevents this tokenization distortion through a hybrid input representation:

Byte-Level BPE: Words are split into standard BPE tokens from a compact 16,000-vocabulary dictionary ($x_{\text{tok}} \to \mathbf{E}{\text{tok}} \in \mathbb{R}^{d{\text{model}}}$).
Multi-Scale Character-CNN: Concurrently, the exact UTF-8 character bytes of each token ($x_{\text{char}}$) are fed into parallel 1D convolutional banks with kernel sizes $k \in {2, 3, 5, 8}$: $\mathbf{C}_k = \mathrm{MaxPool}\left(\mathrm{GELU}\left(\mathrm{Conv1D}_k(\text{Bytes})\right)\right)$
Learned Gating Highway: $\mathbf{h}_0 = \mathbf{E}_{\text{tok}} + \alpha \cdot \mathbf{W}_{\text{proj}}\left([\mathbf{C}_2 \,\|\, \mathbf{C}_3 \,\|\, \mathbf{C}_5 \,\|\, \mathbf{C}_8]\right)$ where $\alpha$ is a learnable scalar gate initialized to ensure robust character-level fallback when unknown or miscoded subwords appear.

2. Transformer Trunk (12 Blocks, $d=768$, $d_{\text{ffn}}=2048$)

Attention Mechanism: Grouped-Query Attention (GQA) with 12 Query heads and 4 Key-Value heads (3:1 ratio), cutting KV-cache memory traffic by 66% during sequence generation.
Rotary Position Embeddings (RoPE): Frequency-based relative query-key rotation for superior relative token distance awareness across cursor movements without absolute position limits.
Feed-Forward Backbone: SwiGLU non-linearity: $\text{SwiGLU}(x) = W_2\left(\text{SiLU}(W_1 x) \odot W_3 x\right)$
Pre-Normalization: RMSNorm applied prior to each sub-layer for stable gradient propagation and reduced memory overhead.

3. Multi-Task Dual Output Heads

Task 1: Causal LM Autocomplete Head: Linear projection ($d_{\text{model}} \to 16{,}000$, weight-tied with token embeddings) yielding next-token probability distribution $P(w_{t+1} \mid w_{\le t})$.
Task 2: Keystroke GEC Edit Head: Linear tagging projection ($d_{\text{model}} \to 6{,}148$) mapping each token position to atomic edits:
- KEEP (0): Token is grammatically and lexically correct.
- DELETE (1): Redundant or erroneous token removal.
- CASE_CAP (2) / CASE_LOWER (3): Casing adjustments.
- REPLACE (4 .. 4099): Atomic swap with one of 4,096 high-frequency correction tokens.
- APPEND (4100 .. 6147): Atomic right-insertion with one of 2,048 suffix/continuation tokens.

Training Checkpoints and Resumption

For developers and researchers wanting to resume training where we left off or fine-tune TypoFormer on custom domain datasets, this repository includes the complete Stage 2 trained checkpoints:

Checkpoint File	Format	Parameters	Training Step	Description
`gec_stage2_model.safetensors`	Safetensors	93.88M	Step 2,500	Stage 2 Wine Fine-Tuned PyTorch weights for GEC Edit Tagging
`lm_stage2_model.safetensors`	Safetensors	93.88M	Step 1,000	Stage 2 Wine Fine-Tuned PyTorch weights for Causal LM Autocomplete
`gec_trainer_state.json`	JSON	N/A	Step 2,500	Training metadata, samples seen (320,000), and learning rate state
`lm_trainer_state.json`	JSON	N/A	Step 1,000	Training metadata, samples seen (64,000), and learning rate state
`modeling_typoformer.py`	Python	N/A	Standalone	Complete PyTorch architecture definition with `from_checkpoint` loader
`finetune.py`	Python	N/A	Standalone	PyTorch training runner for domain adaptation and continual learning

Loading Checkpoints in PyTorch

import torch
from modeling_typoformer import TypoFormer

# 1. Load GEC Model from checkpoint
gec_model = TypoFormer.from_checkpoint("gec_stage2_model.safetensors")
gec_model.eval()

# 2. Load Causal LM Model from checkpoint
lm_model = TypoFormer.from_checkpoint("lm_stage2_model.safetensors")
lm_model.eval()

print("Both models loaded successfully with exact Stage 2 weights!")

Fine-Tuning Guide

You can fine-tune TypoFormer on your own text data using finetune.py:

# 1. Fine-tune Autocomplete on custom text
python finetune.py \
    --task lm \
    --checkpoint lm_stage2_model.safetensors \
    --tokenizer tokenizer.json \
    --output typoformer_lm_finetuned.safetensors \
    --epochs 3 \
    --batch_size 4 \
    --lr 1e-4

# 2. Fine-tune Grammar & Typo Correction
python finetune.py \
    --task gec \
    --checkpoint gec_stage2_model.safetensors \
    --tokenizer tokenizer.json \
    --output typoformer_gec_finetuned.safetensors \
    --epochs 3 \
    --batch_size 4 \
    --lr 8e-5

Stage 2 Hyperparameters

Optimizer: AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay $0.1$)
Precision: BF16 / Mixed Precision
Peak Learning Rate: $6 \times 10^{-4}$ (pre-training) down to $8 \times 10^{-5}$ (Stage 2 fine-tuning)
Gradient Clipping: $1.0$
GEC Loss Weighting: Cross-Entropy with keep_downweight=0.15 to prioritize correcting rare typos over majority KEEP tokens.

Key Highlights

Ultra-Low Keystroke Latency: Mean end-to-end CPU inference of 9.97 ms (P95: 12.12 ms), fitting comfortably inside the 16.6 ms 60 FPS display frame deadline.
Typo Immunity: Gated character-level convolutions preserve word roots even under severe phonetic and fat-finger keyboard slips.
100% Air-Gapped and Offline: Zero network calls, zero telemetry, zero risk of leaking confidential documents, code, or private emails.
Guaranteed Content Fidelity: Non-autoregressive tagging guarantees zero generative hallucination and achieves 100.0% clean negative sentence preservation.
Dual Runtime Availability: Safetensors for PyTorch training and INT8 ONNX for sub-10ms edge execution.

Architectural Comparison: TypoFormer vs. BERT vs. Cloud LLMs

Architectural Dimension	Classical BERT-Base (110M)	Cloud Generative LLMs (7B - 70B)	TypoFormer (94M)	TypoFormer Advantage
Generative Autocomplete	Incapable (Bidirectional MLM only)	Yes (High quality autoregression)	Native Causal Mode (Tied causal head)	Unified Trunk: Single model serves both autocomplete & GEC.
Typo & OOV Robustness	Brittle (`teh` to `['te', '##h']`)	Fair (Relies on large BPE dictionary)	Gated Multi-Scale Char-CNN ($k=2,3,5,8$)	Typo Immunity: Never blinded by unknown character corruptions.
Position Encoding	Absolute Learned (Max 512 tokens)	RoPE / ALiBi	Rotary Position Embeddings (RoPE)	Sequence Extrapolation: Relative distance aware across cursor hops.
Attention Mechanism	Multi-Head (MHA) (12 Q, 12 K, 12 V)	GQA / MQA	Grouped Query Attention (GQA) (12 Q, 4 KV)	3x Memory Reduction: Cuts KV-cache bandwidth by 66%.
FFN Non-Linearity	GeLU + LayerNorm (Additive bias)	SwiGLU + RMSNorm	SwiGLU + Pre-RMSNorm	LLaMA/Mistral-Grade Efficiency: Higher expressivity per parameter.
GEC Formulation	Seq2Seq / MLM (Slow, iterative)	Generative Prompting (Prone to hallucinations)	Constrained Atomic Edit-Tagging (6,148 tags)	Zero Hallucination: 100% preservation of correct text.
End-to-End Latency	150ms - 400ms on CPU	300ms - 2,500ms via Cloud API	< 10ms on CPU (INT8 ONNX)	Real-Time Keystroke Budget: Executes within 16ms display frame.
Network & Privacy	Local or Cloud	Mandatory Cloud API (Data leaves host)	100% Offline Air-Gapped	Complete Privacy: Zero telemetry, password field suppression.

Benchmark and Evaluation Metrics

Live CPU benchmarks measured on standard Intel/AMD consumer hardware (scripts/evaluate_gec_live.py):

Metric	Measured Value	Benchmark Description
Mean CPU Latency	`9.97 ms`	Per-sentence end-to-end inference (INT8 ONNX)
P95 CPU Latency	`12.12 ms`	95th percentile latency (well below 16.6 ms display frame)
P99 CPU Latency	`14.80 ms`	99th percentile worst-case latency
GEC Stage 2 Validation Loss	`0.2597`	Down from 4.1067 (93.7% loss reduction across training)
LM Stage 2 Loss	`1.2054`	Fine-tuned on curated engineering and client communications
Clean Negative Preservation	`100.0%`	Zero false-positive edits on grammatically correct input text
GEC Model Footprint	`130.5 MB` (INT8) / `424.7 MB` (Safetensors)	Symmetric INT8 ONNX graph vs FP32 PyTorch weights
LM Model Footprint	`138.1 MB` (INT8) / `424.7 MB` (Safetensors)	Symmetric INT8 ONNX graph vs FP32 PyTorch weights
Active Working Set RAM	`< 250 MB`	Total memory footprint during active inference

Repository Contents

File	Size	Format	Description
`gec_stage2_model.safetensors`	424.7 MB	Safetensors	Full Stage 2 PyTorch checkpoint for GEC training & fine-tuning
`lm_stage2_model.safetensors`	424.7 MB	Safetensors	Full Stage 2 PyTorch checkpoint for LM Autocomplete fine-tuning
`gec_stage2_int8.onnx`	130.5 MB	ONNX INT8	Quantized graph for sub-10ms CPU grammar and typo correction
`lm_stage2_int8.onnx`	138.1 MB	ONNX INT8	Quantized graph for sub-10ms CPU causal autocomplete
`modeling_typoformer.py`	11.6 KB	Python	Standalone PyTorch architecture module with `from_checkpoint` helper
`finetune.py`	6.4 KB	Python	Standalone PyTorch fine-tuning runner
`infer.py`	6.5 KB	Python	Standalone Python inference engine using ONNX Runtime
`edit_maps.json`	228 KB	JSON	High-frequency confusion dictionary (4,096 replacements + 2,048 appends)
`tokenizer.json`	1.11 MB	JSON	Byte-Level BPE tokenizer config (16,000 vocabulary)
`vocab.json` / `merges.txt`	526 KB	Text/JSON	BPE vocabulary and merge table
`config.json`	780 B	JSON	Architecture parameters & model metadata
`gec_trainer_state.json`	705 B	JSON	Stage 2 GEC training state at step 2,500
`lm_trainer_state.json`	703 B	JSON	Stage 2 LM training state at step 1,000
`typoformer_architecture.png`	235 KB	PNG (300 DPI)	Full neural architecture diagram (NVIDIA emerald theme)
`typoformer_full_logo.png`	188 KB	PNG	Official project branding logo
`TypoFormer-win-x64.zip`	221.0 MB	ZIP	Standalone Windows distribution package

Quickstart and Usage

1. Minimal Installation

pip install torch safetensors onnxruntime tokenizers numpy
git clone https://huggingface.co/fahiiim/TypoFormer
cd TypoFormer

2. Standalone Python Inference (ONNX)

from infer import TypoFormer

# Initialize the standalone inference engine (loads ONNX models locally)
model = TypoFormer(".")

# 1. Autocomplete (Next-Word Generation)
prompt = "Could you please confirm if the proposed implementation timeline aligns with your"
suggestions = model.autocomplete(prompt, top_k=3)
print("Autocomplete suggestions:", suggestions)
# Output: [(' business', 0.9978), (' goals', 0.0002), (' company', 0.0001)]

# 2. Grammar, Punctuation & Typo Correction (GEC)
dirty_text = "We decoupled the event processing pipeline to achieve sub-"
suggestions = model.autocomplete(dirty_text, top_k=2)
print("Engineering completion:", suggestions)
# Output: [(' millisecond', 0.8841), (' second', 0.0712)]

Windows Desktop Keystroke Companion (1-Line Install)

TypoFormer includes a native Windows x64 .NET 8 desktop daemon that runs silently in the system tray and renders a floating caret overlay across any active text editor (Word, Notepad, VS Code, Slack, Browser):

Run in PowerShell:

irm https://raw.githubusercontent.com/fahiiim/TyPoTaKer/main/install.ps1 | iex

Offline & Private: Zero telemetry; no keystrokes ever leave your device.
Password Aware: Automatically suppresses buffering on password and credential fields via Windows UI Automation.
Instant Action: Press [Tab] to accept autocomplete suggestions or grammar corrections.

Citation and Author Attribution

@software{typoformer2026,
  author       = {Md. Fahim Sarker Mridul},
  title        = {TypoFormer: High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/fahiiim/TypoFormer},
  howpublished = {\url{https://github.com/fahiiim/TyPoTaKer}}
}

License

This model and its associated weights are distributed under the MIT License. Authored by Md. Fahim Sarker Mridul.

Downloads last month: 737