CRATER: A Proposal for Complex Angular Attention with Ternary Complex Weights and In-Forward Retrieval

Claims, Evidence and Status

Each strong statement in this document is tagged by confidence and by the nature of the supporting evidence. The statuses below are the authoritative reference; every restatement in the body is scoped identically.

Notation

1. Introduction

Conventional Transformer architectures process natural language through real-valued vector spaces, relying heavily on scaled dot-product attention. While effective, this geometry entangles semantic confidence (vector magnitude) with relational alignment (directional similarity), requiring vast parameters and FP16/FP32 floating-point multiplications to resolve subtle contextual dependencies.

CRATER lifts a \(d_{real}\)-scalar representation into \(\mathbb{C}^{d_{real}/2}\). This parameterization makes magnitude and phase separately accessible, but their semantic roles are learned hypotheses rather than intrinsic properties of complex numbers:

• Complex State Geometry: Phase-aware normalization (Complex LayerNorm) whose multiplicative core is exactly phase-preserving (§4.2), reinforced by an explicit phase-coherence regularization aligning the phases of syntactically linked tokens. The default component-wise \(\mathrm{cGELU}\) activation is not itself phase-preserving — e.g. \(\mathrm{cGELU}(-1+i)\) rotates the phase from 135° to ≈101°, a ≈34° distortion — so “angular coherence” describes the normalization core, not the activation; modReLU (Arjovsky et al., 2016), which scales each phasor by the non-negative real gate \(\mathrm{ReLU}(|z|+b)\) and is therefore exactly phase-preserving on non-zero outputs, is added as a phase-exact activation ablation (§5.3).
• Angular Attention: Replacing scaled dot-product attention with phasor-normalized phase agreement metrics under learnable per-head temperatures and RoPE-style relative phase rotation, making attention routing invariant to magnitude noise and generalizable across context lengths.
• Integrated Active Retrieval (IRW): Incorporating an Imaginary Reflection Window directly into the forward path, allowing tokens to dynamically query external memory indices mid-computation via end-to-end trained reflection gating, at an adaptive cadence driven by context novelty.
• Ternary Linear Layers: Complex weights use a base-3 encoding averaging 3.2 bits per complex weight. With FP16 activations the inner loop is floating-point add/sub; with INT8/INT16 activations it can use INT32 signed accumulation.
• Continual Update Infrastructure: Per-expert LoRA adapters, versioned swaps, and incremental IRW writes support low-parameter-overhead specialization; they do not by themselves establish autonomous self-improvement.

This document serves as an exhaustive architectural blueprint and formal proposal, detailing the underlying mathematical operators, execution mechanics, deployment pipelines, and theoretical limits of the CRATER framework.

2. Related Work

2.1 Complex-Valued Deep Learning

Complex-valued neural networks have been studied in signal processing, acoustics, and oscillatory domains (Trabelsi et al., 2018), and complex transformer blocks have been explored by Eilers and Jiang (2023). A literature search dated 2026-07-23 across arXiv and Semantic Scholar using “complex-valued transformer quantization”, “phase-only attention”, and “ternary complex neural network” did not identify this exact combination; this provisional search statement is not a proof of novelty.

2.2 Attention Mechanics & Geometric Alignment

Standard attention scales quadratically and can be sensitive to query/key norms. CRATER projects queries and keys component-wise onto the complex unit circle and averages cosines of phase differences. A RoPE-style rotation contributes a relative-offset term; the full score still depends on content and may retain absolute-position information from the lifting path.

2.3 Retrieval-Augmented Architectures

Traditional Retrieval-Augmented Generation (RAG) models (Lewis et al., 2020; Borgeaud et al., 2022) treat retrieval as a static, pre-generation context insertion step. This creates a hard separation between parametric memory (model weights) and non-parametric knowledge. Two lines of prior work already bring retrieval inside the forward pass: kNN-LM (Khandelwal et al., 2020) interpolates a non-parametric nearest-neighbour distribution with the model's own next-token distribution at every step, and Memorizing Transformers (Wu et al., 2022) attach an external kNN memory to a middle attention layer and read from it through cross-attention. CRATER's Imaginary Reflection Window (IRW) extends this in-forward line by internalizing retrieval into every layer's residual stream and gating each layer's absorption of retrieved content through a per-token reflection gate \(g_t\), allowing continuous, gated factual updates during sequence processing rather than a single fixed injection point.

2.4 Extremely Low-Bit & 1.58-Bit Quantization

Prior work shows that ternary weights can be competitive under specific QAT recipes (Ma et al., 2024; Hubara et al., 2016; Li et al., 2016; Zhu et al., 2017). A separate line of work removes outliers before quantization by applying a fixed rotation that decorrelates channels, then quantizing in the rotated basis (QuaRot, Ashkboos et al., 2024; SpinQuant, Liu et al., 2024). CRATER's phase alphabet pulls a different lever on the same underlying problem — structuring the quantization error directly in angle space rather than pre-rotating the weights — and §4.7.1 borrows the same decorrelation goal via additive phase dither (Schuchman, 1964) instead of a learned rotation. CRATER shares one scale across real and imaginary components, yielding eight non-zero phase directions. Accuracy, geometric fidelity, and realized kernel speed remain empirical questions for this architecture.

2.5 Sparse Experts and Representation Learning

CMoE follows sparse conditional-computation work from Shazeer et al. (2017) and Switch Transformers (Fedus et al., 2021); CRATER's complex experts and per-expert LoRA remain unvalidated extensions. Its phase interpretation is a representation-learning hypothesis in the sense of Bengio et al. (2013), not an intrinsic semantic property.

3. Architecture Overview

The CRATER layout is a proposed seven-stage forward path. Internal transitions operate in \(\mathbb{C}^{d_{complex}}\); representation lifting and a real-valued readout connect it to standard data and losses.

Residual connections carry stacked real-imaginary components, guaranteeing gradient stability during deep layer traversal. The architecture map below consolidates stages 1–2 into a single complex-lift node for readability.

Interactive Companion

The scripted CRATER Tiny conversation and seven-stage operator trace are maintained on a separate page so the archival paper remains focused.

Open the CRATER Tiny chat and operator trace →

4. Mathematical Framework

4.1 Multi-Frequency Positional Encoding with Relative Phase Rotation

Unlike standard Rotary Position Embeddings (RoPE), which applies a fixed, geometrically spaced set of rotation frequencies across channel pairs, CRATER’s multi-frequency positional encoding constructs an explicit multi-scale periodic basis in the real state lifting step, related to Fourier feature mappings (Tancik et al., 2020). By parameterizing both the frequency scales \(\omega_k^{(i)}\) and phase shifts \(\delta_k^{(i)}\) as learnable parameters, the architecture dynamically adapts its positional sensitivity per representation channel.

This learnable basis prevents the position information from degenerating into a purely static signal over long contexts. The phase offsets \(\delta_k^{(i)}\) prevent a narrower, specific degeneracy: with \(\delta_k=0\) for all \(k\), every sinusoid vanishes simultaneously at \(t=0\) (\(\mathrm{PE}(0,i)=0\) for every channel \(i\)), collapsing the encoding at that one position; generic non-zero offsets avoid this coordinated zero. This is weaker than a uniqueness guarantee: a finite sum of periodic functions is itself periodic, so collisions across distinct positions cannot be ruled out by the offsets alone. A clean necessary condition for injectivity over positions \(t \in [0, T_{max}]\) is that the smallest learned frequency satisfy \(\omega_{\min} < 2\pi/T_{max}\) — its period must exceed the sequence length, or the lowest-frequency component alone repeats within the training context and the encoding must rely entirely on higher frequencies to disambiguate. Because \(\omega_k^{(i)}\) is learnable, this is a property to monitor during training (§5.5), not a guarantee established by construction.

The learnable absolute basis is paired with a RoPE-style relative rotation applied to \(Q\) and \(K\) before phasor normalization. The explicit rotational contribution depends on \(t-s\), while the full score also depends on content and may retain absolute-position information from the lifting path. This mechanism is intended to improve relative-position handling and will be tested for length extrapolation; RoPE alone does not guarantee it.

4.2 Complex State Construction and CLN

Lifting real tokens into \(\mathbb{C}^{d_{complex}}\) exposes magnitude and phase as separate coordinates; any association with confidence or relational structure must be learned and measured. CLN computes a real-valued RMS over complex magnitudes: \(\mathrm{RMS}(z_t) = \sqrt{\frac{1}{d_{complex}} \sum_j |z_{t,j}|^2}\).

Division by the positive RMS and multiplication by positive real \(\gamma_j\) preserve phase. The optional complex bias \(\beta_j\) generally changes phase. Quantitatively, for a fixed pre-bias value \(z\) and \(|\beta_j| \le |z|\),

with equality when \(\beta_j\) is tangent, as seen from the origin, to the circle of radius \(|\beta_j|\) centered at \(z\); the bound widens toward \(\pi/2\) as \(|z|\to|\beta_j|\), so low-magnitude channels are disproportionately exposed to bias-induced phase drift. Only the multiplicative normalization core is unconditionally phase-preserving; experiments must compare biased and bias-free variants, and — per this bound — should stratify the comparison by \(|z|\).

4.3 Angular Attention

Here \(V_{ts}^{(h)}\) contains coordinates for which both magnitudes exceed \(\varepsilon\), and \(m_{ts}^{(h)}=|V_{ts}^{(h)}|\); an all-invalid pair receives score zero. The stable map \(u_\varepsilon\) approaches unit phasor projection away from the origin without a singular derivative. Before multiplication by the positive temperature \(\gamma_h\), the valid-pair average is bounded in \([-1,+1]\).

When imaginary parts vanish, component-wise phasor normalization maps each non-zero coordinate to its sign. Angular attention therefore becomes signed-coordinate agreement, not standard dot-product attention and not vector-level cosine similarity. The unrotated kernel is positive definite: it is the real inner product of the toric embedding \((\cos\theta_1,\sin\theta_1,\ldots,\cos\theta_d,\sin\theta_d)\), restricted to valid coordinates. A learned per-head temperature controls concentration. RoPE contributes a relative-offset rotation, but the full score remains content-dependent (§4.1).

4.4 IRW Retrieval and Fusion

The Imaginary Reflection Window (IRW) transforms typical retrieval-augmented generation from a passive prefix lookup into an active, token-level retrieval cycle. During the forward pass, selected complex hidden states emit multiple parallel queries \(q_t^{(m)}\). These are used to search a product-quantized index (Jégou et al., 2011) containing external memory candidates.

Retrieved values are fused by a learned scalar gate \(g_t\in[0,1]\): the internal stream receives weight \(1-g_t\) and the retrieved stream receives weight \(g_t\). “Imaginary Reflection Window” is a project name, not an additional algebraic operator; the normative mechanism is simply gated retrieval fusion.

Retrieval operates at an adaptive cadence driven by novelty. The design targets a 5–10× reduction in query frequency, with the quality-cost frontier to be measured experimentally. The normative prototype permits retrieval only in layers listed by a manifest, shares one top-\(k\) result per token across eligible layers for at most \(N_{max}\) steps, uses real-valued index keys of declared dimension, blocks writes from future or evaluation targets, treats nearest-neighbour selection as non-differentiable, invalidates stale cache entries by index version, and reports network latency separately. If the index is unavailable, \(g_t=0\).

The IRW threat model includes poisoned documents, prompt-injected instructions in retrieved text, membership leakage, stale or unauthorized writes, adversarial nearest-neighbour collisions, and denial-of-service through retrieval cadence. Evaluation therefore requires signed provenance, namespace ACLs, write auditing, content sanitization, index-version rollback, query-rate limits, canary documents, and red-team measurements of attack success with IRW on/off.

4.5 Ternary Weight Quantization

Ternary quantization restricts every real and imaginary component of the weight tensors to the discrete set \(\{-1, 0, +1\}\), yielding nine joint states per complex weight. The quantizer has exactly two free quantities, and keeping them distinct removes every apparent inconsistency between sections: the scale \(\gamma_w\) (one real scalar per output channel, shared across real and imaginary parts, restoring output magnitude after accumulation) and the normalized threshold \(\Delta_\ell\) (learnable per layer, deciding which normalized components round to zero). Only their product — the absolute threshold \(\Delta_\ell \cdot \gamma_w\) — has geometric meaning.

A shared scale places each non-zero quantized weight on one of eight directions, but a generic weight has only the looser bound of Theorem 1(2), not \(\pi/8\). The \(\pi/8\) result applies only when already unit-normalized phasors are quantized directly with the calibrated threshold. The reference architecture instead quantizes \(W_q/W_k\), then normalizes their outputs; its post-matvec error is measured empirically. Zero outputs have undefined phase and form an explicit sparsity channel.

During training, each quantized weight is passed functionally to its matmul, so the clipped-tanh soft-STE is in the autograd graph. Whether it damps oscillation is an experimental hypothesis. A proposed ~10% quaternary budget uses components in \(\{-2,-1,+1,+2\}\): 16 non-zero joint Cartesian states spanning 12 distinct, non-uniform phase directions and multiple magnitudes. Its estimated weight-storage overhead is about 2.5–3% under the stated assumptions.

Quantization-sensitive components remain in FP16. Ternary weights with FP16/BF16 activations use floating-point additions/subtractions, with no general weight multiplication in a specialized inner loop. INT32 accumulation is valid only when activations are quantized to INT8/INT16. The output-channel scale is then applied in FP16/FP32 after accumulation. NumPy’s generic @ reference does not demonstrate a multiplication-free hardware kernel.

4.5.1 Threshold Conventions (single source of truth)

Three numbers appear in threshold discussions across this document — \(1/2\), \(\sin(\pi/8) \approx 0.383\), and \(2\sin(\pi/8) \approx 0.765\). They are the same geometric constraint expressed in different units, related by the identity: absolute threshold \(= \Delta_\ell \cdot \gamma_w\). The table below is the normative reference; every other mention in this paper conforms to it.

On unit phasors, placing the cardinal/diagonal boundary at \(22.5°\) requires the absolute threshold \(\sin(\pi/8)\). This fixed calibration belongs only to the optional direct phasor quantizer applied after normalization. The reference architecture instead computes a detached per-output-channel mean-modulus scale for weight matrices \(W_q/W_k\) and learns \(\Delta_\ell=\sigma(\delta_\ell)\); that threshold trades sparsity against weight fidelity and is not called calibrated. The theorem is not transferred through the matvec, and post-matvec \(q/k\) angular error is measured empirically.

4.5.2 Optimal Quantization Points, Predicted Sparsity, and Variance-Preserving Initialization

The threshold conventions of §4.5.1 fix the geometry of the ternary alphabet but leave open a statistical question: for weights approximately \(w \sim \mathcal{N}(0,\sigma^2)\) per real/imaginary component (a common empirical approximation near initialization and, loosely, during training of wide layers), what threshold and centroid minimize mean-squared quantization error? This is the classical Lloyd–Max scalar quantizer problem restricted to three levels \(\{-c, 0, +c\}\) with a symmetric decision threshold \(\pm t\).

Lloyd–Max optimality requires each decision boundary to sit at the midpoint of its two neighbouring reconstruction levels, so \(t = c/2\), and each reconstruction level to be the conditional mean of its cell, so \(c = \mathbb{E}[\,|w| \mid |w| > t\,]\). Substituting \(t=c/2\) and using the truncated-Gaussian mean gives the self-consistency equation

where \(\varphi,\Phi\) are the standard normal density and CDF. Solving numerically gives \(x^\star \approx 1.224\), i.e.

and substituting back into the mean-squared distortion \(D=\mathbb{E}[(w-Q(w))^2]\) gives \(D^\star \approx 0.190\,\sigma^2\), i.e. an optimal signal-to-quantization-noise ratio of \(10\log_{10}(\sigma^2/D^\star)\approx7.2\ \text{dB}\) — an information-theoretic ceiling for any symmetric 3-level scalar quantizer of a Gaussian source, independent of architecture.

Two consequences follow directly. (i) Scale estimator. The self-consistency condition \(c=\mathbb{E}[\,|w|\mid |w|>t\,]\) says the MMSE-optimal channel scale is the conditional mean of the surviving (non-zero) weights, not the unconditional mean-modulus \(\gamma_w=\mathbb{E}|w|\) used elsewhere in this section — this is exactly the centroid update already used by Trained Ternary Quantization (Zhu et al., 2017, already cited in §2.4), so re-estimating \(\gamma_w\) from the post-threshold survivors rather than from all weights is a one-line, drop-in refinement of the existing scale rule, not a new mechanism. (ii) The uncalibrated default is a deliberate looseness, not an error. §4.5.1's uncalibrated default (\(\gamma_w=\mathbb{E}|w|\approx0.798\sigma\), \(\Delta_\ell=1/2\), absolute threshold \(0.399\sigma\)) sits well below the MMSE-optimal \(0.612\sigma\); it trades higher distortion for a scale rule with no self-referential fixed point to solve at every training step. Whether that trade is worth it is exactly the kind of ablation §5.3 already calls for (learned weight-threshold sweeps).

The same Gaussian assumption predicts the zero-fraction directly. For a per-component absolute threshold \(\Delta_{abs}=\Delta_\ell\gamma_w\),

At the uncalibrated default (\(\Delta_{abs}\approx0.399\sigma\)) this predicts \(\approx31\%\) zeros per real/imaginary component — consistent with, and giving a first-principles derivation for, the “30–50%” range already stated in the §8 Limitations table. Because the real and imaginary components are quantized independently, the probability that both land in the zero channel (the full complex-origin state) is the square of the per-component fraction, \(\approx0.31^2\approx9.6\%\) — a falsifiable prediction against the zero-fraction metric already tracked in §5.5.

Variance-preserving initialization. Write a ternary weight as \(w=\gamma_w\cdot m\) with \(m\in\{-1,0,+1\}\) and \(\Pr(m\ne0)=1-s\) for sparsity fraction \(s\), so \(\mathbb{E}[m^2]=1-s\). For a linear layer \(y_j=\sum_{k=1}^{d_{in}} w_{jk}x_k\) with i.i.d. zero-mean inputs of variance \(\mathrm{Var}(x)\) independent of \(w\), \(\mathrm{Var}(y)=d_{in}\,\gamma_w^2\,(1-s)\,\mathrm{Var}(x)\). Setting \(\mathrm{Var}(y)=\mathrm{Var}(x)\) (the standard He/Xavier variance-preservation criterion) gives

This is a principled, shape- and sparsity-aware alternative to a single global initialization constant; the reference implementation's ComplexQuantizedLinear (§12.1) currently initializes weight at a fixed \(0.02\) standard deviation regardless of \(d_{in}\) or the eventual learned sparsity \(s\), so \(\gamma_w^\star\) above is directly substitutable there as a QAT-stability ablation (§5.3).

Bit-parity with real 1.58-bit models, made explicit. A real ternary scalar carries \(\log_2 3\approx1.585\) bits of information; CRATER's complex ternary weight is two independent real ternary scalars, so it carries \(2\log_2 3 = \log_2 9\approx3.170\) bits — already the packing target reported in §11.2 (3.2 bits realized, a \(3.170/3.2\approx99.06\%\) packing efficiency). At matched real-scalar count \(d_{real}\), a real 1.58-bit checkpoint (\(d_{real}\log_2 3\) bits total) and a CRATER checkpoint (\(d_{real}/2\) complex weights \(\times \log_2 9\) bits \(=d_{real}\log_2 3\) bits total) therefore carry exactly the same number of bits — the one-line algebraic form of the qualitative claim already made in the FAQ (§B): CRATER's advantage over a real 1.58-bit model, if any, must come from the phase structure imposed on those bits, not from a compression advantage that does not exist at matched scalar count.

4.6 Worked Numerical Example: Angular Attention Under Ternary Quantization

This constructed example isolates the optional direct quantization of already normalized phasors; it does not represent weight quantization through \(W_q/W_k\). It illustrates possible score changes but cannot establish general ranking robustness.

Step 1 — FP16 query and key phasors (after CLN + phasor normalization). Each token is represented by two unit-modulus complex numbers, so we list only the phase angle (in degrees) per component:

Step 2 — FP16 angular attention scores. This example fixes the softmax temperature to \(\tau=1\). For query token \(i\) and key token \(j\), the phase-agreement score is

Taking cat as the query and computing \(s_{\text{cat},j}\) for each key \(j\):

Step 3 — Snap all phases to the ternary 8-direction grid. Every angle is projected onto its nearest neighbour in \(\{0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°\}\) — the exact set of arguments producible by non-zero \((\pm 1, 0, \pm 1)\) real/imaginary components. This nearest-angle snapping is the angular Voronoi partition of Theorem 1(3), i.e. exactly the calibrated configuration of §4.5.1 (absolute threshold \(\sin(\pi/8) \approx 0.383\), zero-boundary at 22.5°) — not the rejected uncalibrated default whose boundary sits at 30°. The two rows below track the resulting quantized \(Q\) for cat and the quantized \(K\) for every key:

Step 4 — Re-compute the attention scores from the quantized Q, K. Using the same formula \(s_{ij} = \tfrac{1}{d_h}\sum_k \cos(\phi^{Q_T}_{i,k}-\phi^{K_T}_{j,k})\), we obtain the ternary column below and place it beside the FP16 column for direct comparison:

Step 5 — Interpretation. No strict rank inversion occurs here, although two distinct FP16 orderings collapse into ties: The/the and sat/on. Because both query and key are quantized, their relative-phase perturbation is bounded by \(|\varepsilon_Q-\varepsilon_K|\le\pi/4\). The corresponding tight per-term score-value bound is \(2\sin(\pi/8)\approx0.765\), since \(\sup_x|\cos(x+\delta)-\cos x|=2\sin(|\delta|/2)\); \(\pi/4\) remains a valid but looser Lipschitz bound. Evaluation must report Kendall \(\tau\), top-1/top-\(k\) retention, KL or Jensen–Shannon divergence, logit-collision rate, and temperature sensitivity. This example says nothing about composed weight-matvec error.

4.7 Phase Preservation: Formal Statement

4.7.1 Composition Lemma under Phase Dithering

The scope note above identifies the open problem precisely: Theorem 1 bounds one component's angular error, but says nothing about how \(d_h\) such errors combine inside the score sum of §4.3. Without further assumptions they cannot be assumed to average out — quantization error from a deterministic rounding rule is a deterministic function of the signal, so it can be arbitrarily correlated across coordinates, and the worst case is exactly the un-improved per-component bound. Dithering — adding independent random noise before rounding — is the classical fix for exactly this failure mode (Schuchman, 1964), and is the same lever, applied to weights instead of activations, that motivates rotation-based decorrelation in QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al., 2024) (§2.4). This subsection states the dithered construction, the exact assumption it requires, and the resulting composed-error bound — closing the gap conditionally, not unconditionally.

Construction. Instead of rounding a phase \(\theta\) directly to the nearest multiple of \(\pi/4\) (the calibrated 8-direction grid of Theorem 1(3)), add independent dither before rounding, once per coordinate and independently for \(Q\) and for \(K\):

Because \(\pi/8\) is exactly half the grid step \(\pi/4\), this is precisely Schuchman's non-subtractive dither condition for a uniform quantizer: the classical result gives that the net per-coordinate error \(\varepsilon = \hat\theta - \theta\) is exactly \(\mathrm{Unif}(-\pi/8,\pi/8)\) and independent of \(\theta\) — the deterministic worst case of Theorem 1(3), turned into a random variable with a fully known law instead of an adversarial unknown quantity.

Assumption (decorrelation). The dither values are drawn independently across the \(d_h\) coordinates and independently for \(Q\) and \(K\). This is a property of the quantizer implementation (an explicit pseudo-random or low-discrepancy dither stream keyed by coordinate index), not an empirical property of trained weights — it must be built, not hoped for. It is exactly the assumption the scope note above says is missing.

Under this assumption, the combined per-coordinate relative-phase error \(\delta_k = \varepsilon^Q_k - \varepsilon^K_k\) is a difference of two i.i.d. \(\mathrm{Unif}(-\pi/8,\pi/8)\) variables, hence triangularly distributed on \((-\pi/4,\pi/4)\) — consistent with, but strictly more informative than, the deterministic bound \(|\delta_k|\le\pi/4\) already used in the Theorem 1 corollary — and i.i.d. across \(k\).

Practical consequence. The bias term \(\mathrm{sinc}^2(\pi/8)\approx0.950\) is a known constant, not a nuisance: dividing dithered scores by it (equivalently, absorbing \(1/\mathrm{sinc}^2(\pi/8)\) into the learned per-head temperature \(\gamma_h\)) removes the systematic attenuation, leaving only the concentrating term. Because the toric-embedding identity of §4.3 writes the unrotated score as a real inner product \(\tfrac{1}{d_h}\langle u(Q), u(K)\rangle\), Theorem 2 is equivalently a concentration statement about that inner product under dithered rounding of the toric embedding's angles — connecting this lemma directly to the kernel view of §4.9 below.

Falsifiable predictions for §5.1. If a dithered calibrated phasor quantizer is added as an ablation arm: (i) the mean ratio \(\hat s/s_0\) over a held-out batch should match \(\mathrm{sinc}^2(\pi/8)\approx0.950\) within measurement noise; (ii) the empirical standard deviation of \(\hat s - \mathrm{sinc}^2(\pi/8)s_0\) should shrink approximately as \(d_h^{-1/2}\) across head-dimension sweeps; (iii) both should be visibly tighter than the same statistics measured on the existing dither-free default path, which this theorem does not cover. All three are additions to the angular-error measurement bullet already scheduled in §5.1.

4.7.2 Quaternary Phase Alphabet Geometry

§4.5 introduces an optional ~10% quaternary budget with components in \(\{-2,-1,+1,+2\}\) and no zero state. The same Voronoi argument used to prove Theorem 1 applies to this alphabet and yields a different, and in one respect worse, angular guarantee than calibrated ternary — worth stating precisely, since §11.6 credits the quaternary budget only with removing origin loss, not with improving grid fidelity.

Restricting to the first quadrant (both components positive; the other three follow by symmetry), the four non-zero Cartesian pairs \((1,1),(1,2),(2,1),(2,2)\) produce exactly three distinct phase directions — \((1,2)\) at \(\arctan(1/2)\approx26.57°\), \((1,1)\) and \((2,2)\) both at \(45°\), and \((2,1)\) at \(\arctan(2)\approx63.43°\) — for \(4\times3=12\) distinct directions in total, matching the count already stated in §4.5. Because \(\arctan(2)+\arctan(1/2)=90°\) identically, the two interior gaps are always equal: \(45°-26.57°=63.43°-45°=18.43°\). The gap across a cardinal axis, from \(63.43°\) to the reflected \(90°+26.57°=116.57°\), is \(53.13°\) — strictly larger than the interior gaps.

The worst-case Voronoi deviation is half of the largest gap, i.e. \(53.13°/2\approx26.57°\) — worse than calibrated ternary's \(22.5°\) (Theorem 1(3)). The quaternary Cartesian budget is therefore not a phase-fidelity upgrade over calibrated ternary; its genuine advantage is structural (no origin state, hence no phase loss at all on those channels), at the cost of a looser worst-case angular bound on the phase it does keep. §11.6 is corrected accordingly.

An alphabet that does improve worst-case phase fidelity while keeping the zero-free property is a polar design: 16 uniformly spaced directions (4 bits of angle, step \(22.5°\)) with magnitude encoded in a separate bit, giving a worst-case deviation of only \(22.5°/2=11.25°\) and, by construction, no origin state. This is a design alternative worth recording for a future revision, not a retraction of the Cartesian quaternary budget already specified; the two are a direct §5.3 ablation candidate (uniform-polar vs Cartesian quaternary heads, matched storage budget).

4.8 Attention Scaling Laws: Dilution, Initialization, and Collisions

The learnable per-head temperature \(\gamma_h\) (§4.3) has no prescribed scale in this document beyond “exp-parameterized” and “learnable”. Three elementary calculations bound the regime it must operate in, analogous to how the \(1/\sqrt{d}\) scaling of standard dot-product attention is derived from a variance argument rather than tuned by hand.

Dilution law (why temperature must grow with context length). Consider \(n-1\) unrelated keys with average score \(\bar s\) and one matching key with score \(s^\star>\bar s\) under a fixed head. The softmax weight on the matching key is

Requiring \(p_{match}\ge \tfrac12\) — the matching key retains at least half the attention mass against \(n-1\) distractors — gives \(\gamma_h(s^\star-\bar s)\ge\ln(n-1)\). For the idealized case of a perfectly aligned match (\(s^\star=1\)) against zero-similarity distractors (\(\bar s=0\)), this is

A temperature that is fixed (or grows more slowly than \(\ln n\)) increasingly dilutes the match as context length grows — not a failure of angular attention specifically, but a generic property of softmax over a growing candidate set that this architecture inherits and should track explicitly. This gives a concrete, falsifiable prediction for §5.5: \(\gamma_h\) trajectories should be monitored against \(\ln(n-1)\) at the context lengths used in training, and the passkey-retrieval stress tests of §5.1 at 4k/8k/16k tokens directly test whether the learned \(\gamma_h\) keeps pace (\(\ln(4095)\approx8.3\), \(\ln(8191)\approx9.0\), \(\ln(16383)\approx9.7\) — a slowly growing, cheap-to-satisfy target, but only if nothing actively suppresses \(\gamma_h\)'s growth during training).

Initialization law (the temperature's starting scale). At initialization, treat the per-coordinate relative phase \(\Delta\varphi_k\) between an arbitrary query/key pair as uncorrelated across \(k\) with \(\mathrm{Var}(\cos\Delta\varphi_k)=\tfrac12\) (exact for \(\Delta\varphi_k\) uniform on the circle). The raw score \(s_{ts}=\tfrac{1}{d_h}\sum_k\cos\Delta\varphi_k\) then has

Matching the standard attention-initialization criterion — keep the pre-softmax logit variance \(\mathrm{Var}(\gamma_h s_{ts})=O(1)\), so gradients through the softmax neither vanish nor explode at step zero — requires \(\gamma_h \sim 1/\sqrt{\mathrm{Var}(s_{ts})}\), i.e.

This is the angular-attention analogue of the \(1/\sqrt{d}\) scaling in standard dot-product attention, derived from the same variance-matching argument applied to the bounded cosine score rather than an unbounded dot product; it is a testable initialization default for the exp-parameterized \(\gamma_h\), and its trajectory away from \(\sqrt{2d_h}\) during early training is itself a training-stability signal worth adding to §5.5.

Collision law (why \(d_h\) should not be too small). At a single coordinate, the calibrated quantizer produces one of \(M=9\) joint states (§4.5, including the origin). Among \(n\) keys, the expected number of pairs sharing the identical quantized value at that one coordinate is exactly \(\binom{n}{2}/M\) by linearity of expectation — for \(n=100\), \(M=9\), that is \(\approx550\) colliding pairs out of \(4950\), i.e. collisions at any single coordinate are the norm, not the exception, for realistic context lengths. What prevents this from degrading the attention score is the product structure across coordinates: if per-coordinate collisions were independent across the \(d_h\) coordinates of a head, the probability that all \(d_h\) coordinates collide simultaneously for a given pair — a full quantized-vector collision — falls to \(\approx M^{-d_h}\), e.g. \(9^{-16}\approx2\times10^{-16}\) at \(d_h=16\), negligible even summed over every pair in a long context. This is the same birthday-paradox arithmetic used to size hash tables, applied here to explain, quantitatively, why single-coordinate agreement is common and harmless while whole-vector collisions are not — and why a head dimension \(d_h\) too small (few independent coordinates) is exactly the regime where this protection breaks down, worth flagging in the per-layer ablation of §5.3.

4.9 Kernel Structure, Invariances, and Information Budget

§4.3 already observes that the unrotated score is a real inner product of an explicit toric embedding. Three structural consequences of that observation are worth making explicit, since two resolve orphaned citations and the third is a genuine, provable invariance of the architecture.

Linear-attention structure. The toric embedding \(u(\varphi)=(\cos\varphi,\sin\varphi)\) is a finite-dimensional, explicitly written feature map, not an approximation: the score \(\tfrac1{d_h}\langle u(Q),u(K)\rangle\) is already exactly the \(\phi(q)^\top\phi(k)\) form used by linear attention (Katharopoulos et al., 2020) to replace the \(O(T^2)\) attention matrix with an \(O(T)\) causal running sum. Because CRATER's feature map is explicit rather than randomized, the same causal-recurrence trick applies directly to the pre-softmax angular score with no approximation error. If softmax is applied on top of \(\gamma_h\langle u(Q),u(K)\rangle\) (the architecture's default), the resulting exponentiated kernel remains positive definite — products and exponentials of PD kernels are PD — but is no longer finite-dimensional, and recovering sub-quadratic attention in that regime would need a randomized feature approximation distinct from, if similar in spirit to, Katharopoulos et al.'s construction.

SimHash limit. In the real-embedding limit (every imaginary part zero), phasor normalization degenerates to \(\mathrm{sign}(\cdot)\), and the score becomes the average sign agreement \(\tfrac1{d_h}\sum_k\mathrm{sign}(q_k)\,\mathrm{sign}(k_k)\) — exactly the bit-agreement statistic that SimHash (Charikar, 2002) uses to estimate cosine similarity from sign vectors. Angular attention and the locality-sensitive product-quantized index of §4.4 are, in this limit, built from the same geometric primitive under two different names.

4.10 End-to-End Error Budget and Lipschitz Audit

The scope note in §4.7 and the composition lemma in §4.7.1 stop at the attention score. Closing the remaining distance to a system-level statement — from one quantized component to a change in perplexity — requires tracking how error propagates through every downstream operator. No single number in this subsection should be read as a validated end-to-end bound: composing several individually-loose steps compounds looseness, and the empirical measurements already scheduled throughout §5 remain the authoritative source for actual magnitudes. What follows is the chain connecting the per-component guarantees already proven to the system-level quantities practitioners actually care about — an audit trail linking each § in this paper to the next, not a replacement for measuring any of them.

Two links proved directly. For the softmax row: writing \(p_i(x)=e^{x_i}/\sum_je^{x_j}\) and \(\delta=x'-x\),

and the same argument bounds \(p_i(x')/p_i(x)\), giving the stated ratio bound (and, by a first-order expansion, \(\|p(x')-p(x)\|_1 \lesssim 2\|\delta\|_\infty\) for small \(\|\delta\|_\infty\)). For the perplexity row: because \(\mathrm{PPL}=\exp(-\tfrac1N\sum_n\log p(\text{token}_n))\) is the exponentiated average negative log-likelihood, and the ratio bound above applies token-by-token to the assigned probability, the geometric mean of assigned probabilities — and hence \(\mathrm{PPL}\), its reciprocal — changes by at most the same factor \(e^{2\|\Delta\ell\|_\infty}\), where \(\Delta\ell\) is the vector of logit perturbations at a given step.

Reading the chain. Composing every row gives, schematically, component phase error \(\to\) score error (tight, §4.6/§4.7) \(\to\) composed score error (concentrating under dither, §4.7.1, otherwise unbounded) \(\to\) logit error (scaled by \(\gamma_h\)) \(\to\) attention-weight error (softmax-Lipschitz) \(\to\) fused-state error (IRW-contractive) \(\to\) output-logit error (scaled by \(\|w_v\|\)) \(\to\) perplexity change (exponential in the final logit error). Every arrow is individually justified above or elsewhere in this paper; no arrow has been validated in composition on trained weights, which is exactly why §5.1's angular-error measurement and §5.5's training-stability metrics remain load-bearing rather than optional. This table is the index connecting them, not a substitute for them.

Gradient convention (Wirtinger/CR-calculus). Every gradient step in this architecture updates a complex-valued weight using \(\partial \mathcal{L}/\partial\bar w\) (equivalently, independent partials with respect to \(\mathrm{Re}(w)\) and \(\mathrm{Im}(w)\)), the standard convention for real-valued losses of complex variables (Kreutz-Delgado, 2009) — the same convention the soft-STE surrogate gradient already uses component-wise in §12.1, without previously naming it. This is also why no non-trivial bounded holomorphic activation can exist for the complex state (Liouville's theorem: a bounded entire function is constant), motivating the component-wise, necessarily non-holomorphic \(\mathrm{cGELU}\)/modReLU family instead of a "nicer" complex-differentiable nonlinearity (Trabelsi et al., 2018). A minor, zero-risk implementation note in the same spirit: the reference QuaternarySoftSTE.backward (§12.1) computes (1 - tanh(x)**2).clamp(0.0, 1.0); since \(1-\tanh^2(x)\in(0,1]\) for every real \(x\), the clamp is a mathematical no-op, harmless but worth removing for clarity.

5. Evaluation Protocol

Because CRATER is an architecture proposal rather than a released checkpoint, the evaluation methodology described here is prospective: it defines the exact experimental grid, baselines, seeds, and reporting conventions that would be required to convert current hypotheses (see the Claims table) into peer-reviewable evidence. The protocol is deliberately split into three orthogonal axes — quality, efficiency, and training stability — with each axis specifying both the aggregate metric and the ablation slice that isolates a specific CRATER component.

5.1 Benchmark Suite

• Language modeling: WikiText-103 for historical comparability plus C4-validation and a contamination-audited FineWeb validation slice for modern perplexity; The Pile subset is retained for byte-level bits-per-byte.
• Question answering: Natural Questions and TriviaQA (exact-match and F1) with and without the IRW retrieval path, to isolate the parametric vs non-parametric contribution.
• Retrieval-augmented tasks: KILT-style suite (Wizard-of-Wikipedia, FEVER, T-REx) measuring both downstream metrics and retrieval precision at top-k.
• Long-context stress tests: passkey retrieval at 4k / 8k / 16k tokens and needle-in-a-haystack variants to isolate the effect of angular attention with relative phase rotation on positional binding at scale.
• Angular-error measurement: empirical histograms of post-matvec angular deviations on trained checkpoints, closing the composition gap flagged in the §4.7 scope note; when the dithered quantizer of §4.7.1 is enabled, additionally report the mean ratio \(\hat s/s_0\) against the predicted \(\mathrm{sinc}^2(\pi/8)\approx0.950\) and the error's standard deviation against the predicted \(d_h^{-1/2}\) scaling, both compared against the same statistics measured on the dither-free default path.
• Structural and multilingual coverage: dependency parsing and code syntax validity, multilingual perplexity/QA across high- and low-resource languages, and tokenization-normalized reporting.

5.2 Baselines and Matched-Parameter Controls

Every run reports \(d_{real}\), \(d_{complex}\), trainable real scalar components, logical complex parameters, active parameters per token, packed bytes, training FLOPs, inference operations, measured memory traffic, total/active MoE capacity, and IRW index/query cost. “Matched parameters” or “matched FLOPs” without these fields is not accepted.

• Matched logical channels: equal logical width/head count, with unequal scalar capacity reported explicitly.
• Matched real scalars: equal trainable real scalar components, using \(d_{complex}=d_{real}/2\) for CRATER.
• Matched deployment budget: equal active parameters/token and comparable measured stored bytes, memory traffic, and latency on the same hardware/kernel stack.

5.3 Ablation Grid

• Complex vs real under each of the three controls above.
• Ternary-complex vs ternary-real (isolates the shared-scale phase preservation on non-zero states).
• Angular attention vs cosine attention vs standard scaled dot product (isolates the phasor normalization).
• Relative phase rotation on/off (tests relative-position handling and length extrapolation).
• IRW on / off / gate-only-warmup (isolates active retrieval from passive RAG); adaptive vs fixed query cadence.
• CMoE expert-count sweep (1, 4, 8, 16) at fixed active compute; per-expert LoRA on/off under domain shift.
• Per-layer quantization sensitivity: leaving one layer in FP16 at a time; ternary-only vs mixed quaternary heads; learned weight-threshold sweeps, plus the fixed calibrated direct-phasor operator as a separate ablation (§4.5.1).
• Phase-aware objective and phase-coherence regularization on/off.
• cGELU vs modReLU (Arjovsky et al., 2016) component-wise activation, isolating whether an exactly phase-preserving activation changes the effectiveness of phase-coherence regularization (§1, §7.1).
• Dithered vs default (dither-free) calibrated phasor quantizer (§4.7.1), validating the composed-score bias \(\mathrm{sinc}^2(\pi/8)\) and concentration-rate predictions against the unconditional empirical baseline.
• Polar 16-direction quaternary heads vs the Cartesian \(\{-2,-1,+1,+2\}\) budget (§4.7.2), at matched storage, isolating worst-case angular fidelity from origin-loss removal.
• Activation precision FP16/BF16 vs INT8 vs INT16, reporting perplexity, calibration error, overflow rate, tokens/s, and joules/token.

5.4 Efficiency & Deployment Metrics

• Batch-1 CPU tokens/s on a single AVX2 core and on a single NEON core, since these are the deployment targets where the multiplier-free ternary inner loops pay off the most.
• Weight memory footprint for packed base-3 vs FP16, verified end-to-end (packing, mmap, unpack LUT), reported against both the FP16-complex baseline and a matched-real-parameter 1.58-bit baseline.
• KV-cache growth curves as a function of context length, to confirm the empirical crossover where the KV cache overtakes the packed weights as the dominant memory term.
• Post-quantization perplexity revalidation on WikiText-103 after every checkpoint export, as a gating criterion for any release.
• End-to-end cost accounting: training GPU-hours, inference joules/token, retrieval requests and bytes/token, index-build cost, and amortized storage.

Kernel-only speedup is bounded by Amdahl's law. If ternary linear layers occupy fraction \(p\) of baseline latency and accelerate by \(s\), the end-to-end ceiling is \(S=1/((1-p)+p/s)\); every hardware result reports \(p\), \(s\), and the measured total rather than extrapolating from inner-loop arithmetic.

5.5 Training-Stability Metrics

• Gradient-norm distribution per layer over training, to confirm the integrated soft-STE damps phase-gradient oscillation relative to hard STE.
• Phase-variance evolution per head, to detect phase collapse or drift toward degenerate uniform distributions.
• Zero-fraction (origin-channel size) per layer over training, to verify \(\Delta_\ell\) converges to a sparsity/fidelity balance rather than collapsing all weights to zero.
• Router entropy in CMoE, to detect expert-collapse (a single expert absorbing all tokens).
• Reflection-gate distribution over training, to verify that the ReZero warmup is effective and that the gate does not saturate to 0 or 1.
• Per-head temperature \(\gamma_h\) trajectory, checked at initialization against \(\gamma_h^{(0)}\sim\sqrt{2d_h}\) and during training against the context-length dilution requirement \(\gamma_h\gtrsim\ln(n-1)\) (§4.8).
• Logit dynamic range \(M=\max_v\|h\|\|w_v\|\) at the output head, checked against the perplexity-floor requirement \(M\gtrsim4.3\) for \(\mathrm{PPL}\le10\) at typical vocabulary sizes (§7.1).
• Batch-level mean resultant length \(\bar R=\bigl|\tfrac1N\sum_n e^{i\varphi_n}\bigr|\) of token phases, to distinguish healthy phase-coherence regularization from Kuramoto-style global collapse (§7.1).

5.6 Reporting Conventions

All headline numbers are reported as mean ± standard deviation over at least three seeds (defaults: 42, 1234, 2026). Any single-seed number is explicitly labelled as such and treated as preliminary. Efficiency numbers include the hardware description (CPU model, memory bandwidth, compiler flags) and the exact packing configuration used, so that third parties can reproduce the CPU tokens/s figure without ambiguity.

6. Discussion

CRATER tests whether explicitly accessible magnitude and phase provide a useful inductive bias for language. It encourages, but does not guarantee, magnitude to encode salience or confidence and phase to encode relational structure; probing must establish whether these roles emerge.

A naive complex multiply uses four real multiplications and two real additions; the Karatsuba-style identity computes it with three real multiplications and five additions, which can be preferable only when multiplications dominate. End-to-end layer cost still depends on scalar width, bandwidth, sparsity, and kernel implementation. Ternary weights remove general weight multiplication from a specialized inner loop, but realized advantage over a scalar-matched real baseline must be benchmarked.

Open risks include optimization, representation quality, origin-channel sparsity, and deployment cost. The soft-STE, biased versus bias-free CLN, and post-matvec angular error require ablation. The KV/weight memory crossover must be computed from each model’s layer count, scalar width, batch, context, attention-sharing scheme, and precision.

7. Integrated Mechanisms

Every improvement axis identified for the ternary CRATER line is folded directly into the core specification rather than deferred: each mechanism below is a working component of the architecture, cross-referenced to the section that defines it and to the §5 ablation that isolates it.

7.1 Phase-Aware Objective and Interpretability

• Phase-aware output objective: the Hermitian readout of Stage 7 is supervised directly — a phase-margin term on the complex vocabulary projection is added to cross-entropy. Concretely, the objective rewards a large phase alignment between the final hidden state and the target token's complex embedding relative to the distractor embeddings, so the phase geometry is optimized where it acts on the logits (before the real collapse), rather than being merely propagated and then discarded. The final collapse to real probabilities is a task requirement, not a phase leak: phase shapes the logits through the Hermitian product, and this term trains it to do so discriminatively.
• Margin rationale and a perplexity floor. Writing the readout logit as \(\ell_v=\mathrm{Re}\langle h,w_v\rangle=\|h\|\|w_v\|\cos\Delta\varphi_v\), \(\partial\ell_v/\partial\Delta\varphi_v=-\|h\|\|w_v\|\sin\Delta\varphi_v\) vanishes exactly at perfect alignment (\(\Delta\varphi_v=0\)): plain cross-entropy stops pushing for tighter phase alignment as soon as the target is merely the arg-max, with no margin. This is the same failure mode additive angular-margin losses fix in metric learning (ArcFace, Deng et al., 2019; CosFace, Wang et al., 2018; SphereFace, Liu et al., 2017) — the phase-margin term above is that idea applied to a vocabulary-sized softmax, and those three are the citation for why a margin, not just alignment, is the objective. The same Cauchy–Schwarz bound \(|\ell_v|\le M\) (for \(M=\max_v\|h\|\|w_v\|\)) also gives a hard floor on achievable cross-entropy, \(\mathrm{CE}_{\min}=\ln\bigl(1+(V-1)e^{-2M}\bigr)\), attained when the target logit sits at \(+M\) and every one of the \(V-1\) competitors sits at \(-M\); at vocabulary size \(V\approx5\times10^4\), keeping perplexity \(\le10\) requires \(M\gtrsim4.3\) — a small, concrete target worth tracking directly as a “logit dynamic range” metric (§5.5) rather than only inspecting perplexity after the fact.
• Phase-coherence regularization: a penalty aligning phases of dependency-linked tokens from declared Universal Dependencies treebanks or a versioned parser, with parser version, confidence filtering, language coverage, and licensing reported. Implemented as \(\lambda_{\text{phase}}\sum_{(i,j)\in\text{links}}\bigl(1-\cos(\varphi_i-\varphi_j)\bigr)\) rather than a literal squared angle difference (ill-defined across the \(\pm\pi\) wrap), this pairwise cosine coupling is exactly the Kuramoto model of coupled oscillators — whose well-known phase transition means a sufficiently strong or sufficiently densely linked \(\lambda_{\text{phase}}\) does not merely align related tokens, it can synchronize all of them onto a single common phase, destroying the discriminative structure the regularizer was meant to encourage. The cosine warmup already bounds \(\lambda_{\text{phase}}\) below \(10^{-3}\), but the failure mode should be monitored directly: track the batch-level mean resultant length \(\bar R=\bigl|\tfrac1N\sum_n e^{i\varphi_n}\bigr|\) (Mardia and Jupp, 2000), the standard circular-statistics collapse diagnostic, as a new §5.5 metric, and treat \(\bar R\) approaching \(1\) across a semantically diverse batch as evidence of Kuramoto-style collapse rather than of successful phase-coherence learning. \(\lambda_{\text{phase}}\) is cosine-warmed from 0 to 1e-3 over the first 10% of steps.

7.2 Quantization Robustness

• Learnable weight thresholds \(\Delta_\ell\), including for \(W_q/W_k\), trading sparsity against weight fidelity. Optional direct phasor snapping uses \(\sin(\pi/8)\) as a separate ablation.
• Post-quantization scale calibration on ~512 representative sequences (projected 0.1–0.3 PPL recovery without retraining; §10 tier 3).
• Mixed quaternary heads (~10% budget) with 16 Cartesian states, 12 phase directions, and an estimated 2.5–3% weight-storage overhead under the stated assumptions.

7.3 Continual Update and Specialization Infrastructure

• Per-expert LoRA adapters: online adaptation attached per CMoE expert rather than globally, providing low-parameter-overhead specialization whose full training and serving cost is reported.
• Drift-triggered blue-green swaps: phase-distribution entropy and routing statistics are monitored, and full retraining is triggered when drift crosses a threshold instead of on a fixed calendar; the versioned exchange container (§11.1) makes the swap atomic.
• Replay buffer of older examples during full retraining, preventing catastrophic forgetting of the phase structure learned by earlier checkpoints.
• Incremental IRW index writes (§4.4) as the fast, non-parametric channel for injecting fresh knowledge between retrains.

7.4 Training Stability

• A positive real RMS/gain core that preserves phase; optional complex bias is explicitly non-preserving (§4.2).
• ReZero-style gate warmup for IRW reflection gates (§4.4), stabilizing the first ~10k steps.
• Soft-STE (clipped tanh rather than hard sign) for quantized weights, with phase-only gradient clipping in the last training decile (§4.5).

7.5 Deployment Toolchain

• Base-3 unpacking via a 256-entry × 5-trit lookup table (~1.25 KB); any speedup is a projection to be benchmarked against unpack overhead.
• At 40% sparsity, the ideal arithmetic reduction is \(1/(1-0.4)\approx1.67\times\); realized speedup may be substantially lower and must include index traffic, gathers, branches, and SIMD utilization.
• A single versioned exchange container (packed trits + scales + \(\Delta_\ell\) + LoRA adapters + config + tokenizer) as the atomic unit of deployment and blue-green swaps (§11.1).

Genuinely long-horizon research directions — conjugate attention heads for contradiction sensitivity, phase-coherence-guided decoding, quaternion or hyperbolic state spaces, and latent reflective deliberation loops — are deliberately excluded from the core specification and carry no evidence tier above speculative (§10).

8. Limitations

CRATER is transparent about the operational cost of its architectural bets. Limitations that were addressed by mechanisms now integrated into the core (retrieval dependency, quantization gap, STE gradient bias, phase-regularization sensitivity) are documented alongside their solutions in §4 and §7; the table below retains only the limitations that are inherent to the design and cannot be engineered away, each paired with the integrated strategy that bounds its impact.

9. Conclusion

CRATER proposes complex angular attention, ternary complex weights, and in-forward retrieval as a testable architecture. Relative rotation is intended to aid positional handling, phase regularization is an interpretability hypothesis, and the retrieval/operator choices require controlled ablation.

Packed ternary weights require about 0.4 byte per complex weight, so one billion packed weights occupy about 400 MB before scales, high-precision modules, allocator state, and scratch buffers. This is a storage estimate, not a latency or feasibility result. Edge deployment and continual update infrastructure remain prospective benchmarks.

10. Evidence Framework

Because CRATER mixes established transformer methodology with novel architectural extensions, presenting a single global “confidence score” for the paper would be misleading. Instead, every claim in this document belongs to one of three evidence tiers, and readers should calibrate their acceptance to the tier of the specific claim rather than to the paper as a whole. The Claims table is the authoritative assignment; wherever a claim is restated in the body, its scope matches the table exactly (e.g. multiplier-free is always scoped to ternary inner loops, compression is always scoped to the FP16-complex baseline).

• Tier 1: established prior art or algebra. Prior operators retain the scope of their cited evidence. Signed accumulation is established algebraically for a specialized kernel, but it is not demonstrated by the generic NumPy/PyTorch reference and implies no hardware speedup by itself.
• Tier 2: controlled CRATER results. This tier is currently empty. It becomes usable only when data, seeds, checkpoints, and protocols are published.
• Tier 3: projections and hypotheses. Scaling projections, bookkeeping-derived compression, composed angular-error behaviour of full ternary matvecs, and unbenchmarked latency remain predictions. The Claims table's badges map “Established” to Tier 1 and both “Hypothesis” and “Projection” to Tier 3 until reproducible results justify Tier 2.

Claims that would fall below medium-low confidence (speculative long-horizon extensions such as quaternion or hyperbolic variants) are kept out of the main body and appear only in the closing note of §7 as excluded long-horizon directions, so that the evidence tier of every load-bearing statement is at least medium-low.

11. Deployment and Inference

Section 11 traces the path from a trained ternary CRATER checkpoint to a running inference process on real hardware, treating deployment as a first-class engineering concern rather than a downstream afterthought. Every design choice in this section — the packing format, the on-demand mmap loader, the multiplier-free ternary inner loop, the precision map, the versioned exchange container — is justified by a measured constraint of the target platforms (edge CPUs, embedded flash, and eventually FPGA/ASIC). The framing is deliberately deployment-first, because the entire economic case for the complex-valued architecture rests on the ternary weights delivering their promised footprint reduction end-to-end, not just on disk.

11.1 Deployable Artifact

A CRATER checkpoint is not a monolithic tensor blob but a versioned four-part exchange container — the atomic unit of the blue-green swaps of §7.3 — and every consumer (loader, inference engine, swap orchestrator) references the exact same manifest. The four parts have deliberately different precisions to concentrate the compression savings in the largest, most noise-tolerant components while protecting the small components that would break the model if perturbed:

• The packed ternary weights themselves, in base-3 five-trits-per-byte encoding (§11.2), plus the 2-bit-packed quaternary head weights.
• The FP16 per-output-channel scales \(\gamma_w\) (one scalar per matvec row of each weight tensor) and the learned per-layer thresholds \(\Delta_\ell\).
• The quantization-sensitive components kept in high precision: embeddings, CMoE router, reflection gates \(g_t\), CLN gains and biases, per-expert LoRA adapters, and the output head (FP16; embeddings and head may be further reduced to INT8 per §11.6).
• The tokenizer and model configuration (layer count, width, head count, quaternary-head mask, IRW parameters, packing metadata) shipped as a signed JSON manifest so that a checkpoint cannot be loaded with a mismatched runtime.

11.2 Packing Scheme

The packing scheme is chosen to minimize both on-disk size and unpacking cost. Because each real or imaginary component of a ternary weight lives in \(\{-1, 0, +1\}\), the natural encoding is base 3. Mapping trits to \(\{0, 1, 2\}\) and packing five trits per byte gives \(3^5 = 243 \le 256\) — the tightest lossless base-3-to-byte packing that still fits in a single unsigned byte, achieving 1.6 bits per trit (≈3.2 bits per complex weight). This is within ~1% of the information-theoretic lower bound of \(\log_2 9 \approx 3.17\) bits per complex weight, meaning the packing overhead is essentially negligible. Quaternary head components (\(\{-2,-1,+1,+2\}\)) pack trivially at 2 bits each; at a ~10% head budget they add under 3% to the weight footprint. For a canonical 7-billion-complex-weight model the resulting budget is:

11.3 Loading: mmap and On-the-Fly Unpacking

The packed file may be memory-mapped, but resident pages, allocator overhead, unpacked layer scratch, activations, scales, quaternary weights, embeddings, router/LoRA state, IRW caches, and KV storage must all be measured. A 256-entry × 5-trit LUT removes integer division; its end-to-end speedup is a projection to be benchmarked. The NumPy reference below validates values and packing only, not instruction-level multiplier elimination.

# Uses the canonical unpack_base3 codec from §12.2.
    def complex_ternary_matvec(Wre_p, Wim_p, gamma, x_re, x_im, shape):
    """(Wre + i*Wim)(x_re + i*x_im): pure adds/subs in the inner loop."""
    Wre = unpack_base3(Wre_p, shape[0] * shape[1]).reshape(shape).astype(np.int16)
    Wim = unpack_base3(Wim_p, shape[0] * shape[1]).reshape(shape).astype(np.int16)
    # Reference path: numpy promotes to the activation dtype for the sum.
    # Deployment kernels take int8/int16 activations and accumulate in int32,
    # THEN apply gamma once per output channel (see §11.6 and §12).
    y_re = gamma * (Wre @ x_re - Wim @ x_im)
    y_im = gamma * (Wre @ x_im + Wim @ x_re)
    return y_re, y_im

11.4 Multiplier-Free Ternary Inner Loops

The multiplier-free property — scoped, here as everywhere in this paper, to the ternary linear layers, which dominate the FLOP count, and never claimed for the FP16 side channels — is what makes CRATER an interesting target for CPU, embedded, and FPGA/ASIC deployment simultaneously. Once weights are in \(\{-1, 0, +1\}\), the output of a linear layer collapses to a signed accumulation of the input activations:

With FP16/BF16 activations, ternary weights select signs and the accumulations remain floating-point. With INT8/INT16 activations, a specialized kernel can use INT32 signed accumulation. A non-zero real or imaginary component of a complex weight contributes to two output accumulations; if both are non-zero, one complex weight produces up to four signed contributions. At 40% component sparsity, \(1.67\times\) is only an ideal arithmetic ceiling before indices, gathers, cache misses, branches, and SIMD under-utilization. The per-output scale and all FP16 side modules are measured rather than declared negligible.

11.5 Generation Loop

The autoregressive generation loop follows the standard transformer template, but two boundaries are worth calling out explicitly: (i) the transition from FP16 embeddings into the ternary hidden space happens exactly once per token, at the input projection; (ii) the transition back to FP16 happens exactly once per token, at the output head — so no per-step precision juggling is required inside the layer stack.

• Tokenize the prompt, fetch FP16 embeddings, add multi-frequency positional encoding.
• Per layer: CLN → angular attention with ternary Q/K/V/O (relative phase rotation, phasor-normalized scores stay cosine-like and cheap) → IRW window (external index queries at adaptive cadence, FP16 gate fusion) → CMoE (FP16 routing, ternary experts + FP16 LoRA where attached).
• Output head → logits → softmax → sampling (temperature, top-p); repeat autoregressively while feeding the KV cache.

11.6 Precision Map

The precision map is a deliberate risk-management document. Ternary precision is applied only to components where an aggregated numerical error averages out over many activations (the linear projections, CMoE experts, complex lift), while FP16 is retained wherever a single scalar mistake would change the compute path (router, gates), the represented word (embeddings, output head), or the normalization stability (CLN gains, \(\gamma_w\)). The KV cache is treated separately because it dominates memory at long context and therefore has its own optimization axis (§8).

11.7 Deployment Pipeline

The deployment diagram shows the integer-activation variant. Its hot stage is INT32 add/sub accumulation, not a multiply–accumulate; FP16/BF16 activation deployments use floating accumulation instead.

11.8 Hardware Targets and Pitfalls

CPU, embedded, FPGA, and ASIC benefits are projections to be benchmarked. Packed size does not translate proportionally to latency because unpacking, sparse indexing, cache behavior, SIMD utilization, normalization, softmax, retrieval, batch size, and KV traffic remain. A 1B-complex-weight packed ternary core is about 400 MB before all other state; FP16 scales and gates still require floating-point support or emulation. Area and power claims require synthesized designs and iso-throughput measurements.

Four operational pitfalls dominate in practice and every deployment integration should validate them explicitly:

• Use INT32 accumulation for INT8/INT16 activations; FP16/BF16 activations require floating accumulation. Accumulator range must be derived from scalar fan-in and activation bounds.
• Apply \(\gamma_w\) after accumulation, not before: pre-scaling the ternary values before the sum destroys the multiplier-free property and introduces per-weight FP16 multiplies that dominate the inner-loop cost.
• Report the KV/weight crossover from the actual layer count, scalar width, batch, context, GQA/MQA structure, and precisions rather than assuming “a few thousand” tokens.
• Re-validate post-quantization perplexity on WikiText-103 (or an equivalent held-out set) before every deployment. The residual quantization gap is small but non-zero and drifts with every retrain; treating it as a hard release gate is the only reliable way to prevent a silent quality regression.

12. Reference Implementation

This section provides compact PyTorch references for the core operators implemented below. Weight quantizers expose a learned normalized threshold, while the separately guarded unit-phasor operator uses the fixed geometric calibration of §4.5.1. The code prioritizes readability over kernel performance; production deployments should replace the generic matmul with a measured specialized kernel. All snippets are copy-runnable with Python 3.10+, torch >= 2.1, and numpy >= 1.24.

The reference is organized as a bottom-up build: (a) the ternary quantizer with soft straight-through estimator and explicit threshold, plus the quaternary quantizer and phase-sensitivity head selection, (b) the base-3 packing/unpacking codec, (c) Complex LayerNorm, (d) angular attention over phasor-normalized \(Q, K\) with the required learnable temperature, and (e) a one-step QAT training loop that ties everything together.

12.1 Ternary & Quaternary Quantizers (soft-STE, explicit threshold)

The quantizer separates the detached per-output-channel mean-modulus scale from the normalized threshold \(\Delta_\ell=\sigma(\delta_\ell)\). All weight matrices, including \(W_q/W_k\), learn that threshold because they are not unit phasors. DELTA_PHASOR is reserved for the guarded direct activation-phasor operator. The backward pass differentiates a smooth surrogate with respect to both the master weight and threshold. Quaternary exact-zero ties are stochastic during training and deterministic during evaluation.

import math
import torch, torch.nn as nn
from torch.autograd import Function

DELTA_WEIGHT_INIT = 0.5
DELTA_PHASOR = math.sin(math.pi / 8)  # direct unit-phasor operator only

class TernarySoftSTE(Function):
    """Ternary rounding with soft (clipped-tanh) straight-through gradient (§4.5)."""
    @staticmethod
    def forward(ctx, x, gamma, delta):
        scaled = x / gamma
        ctx.save_for_backward(scaled, gamma, delta)
        return gamma * torch.sign(scaled) * (scaled.abs() > delta).float()

    @staticmethod
    def backward(ctx, grad_out):
        scaled, gamma, delta = ctx.saved_tensors
        slope = 8.0
        gate = torch.sigmoid(slope * (scaled.abs() - delta))
        tanh_scaled = torch.tanh(2.0 * scaled)
        dx = 2.0 * (1.0 - tanh_scaled.square()) * gate
        dx = dx + tanh_scaled * slope * gate * (1.0 - gate) * scaled.sign()
        ddelta = grad_out * gamma * tanh_scaled * (-slope * gate * (1.0 - gate))
        return grad_out * dx, None, ddelta.sum_to_size(delta.shape)

def ternary_quantize(w: torch.Tensor,
                     delta: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
    """Quantize a (complex or real) weight tensor to ternary.
    Detached mean-modulus gamma_w is per output channel and shared across
    real/imag parts (phase snapped to the
    8-direction grid of Theorem 1 on non-zero states). The normalized
    threshold `delta` is the learnable Delta_l in the full model; only the
    product delta * gamma_w has geometric meaning (§4.5.1).
    Direct unit-phasor quantization is a separate guarded operator below."""
    if torch.is_complex(w):
        gamma = w.abs().mean(dim=0, keepdim=True).clamp(min=1e-8).detach()
        wr = TernarySoftSTE.apply(w.real, gamma, delta)
        wi = TernarySoftSTE.apply(w.imag, gamma, delta)
        return torch.complex(wr, wi), gamma
    gamma = w.abs().mean(dim=0, keepdim=True).clamp(min=1e-8).detach()
    return TernarySoftSTE.apply(w, gamma, delta), gamma

def quantize_unit_phasors(z: torch.Tensor) -> torch.Tensor:
    """Optional calibrated activation operator; rejects non-unit inputs."""
    if not torch.allclose(z.abs(), torch.ones_like(z.real), atol=1e-4, rtol=1e-4):
        raise ValueError("quantize_unit_phasors expects unit-modulus inputs")
    gamma = torch.ones_like(z.real)
    delta = torch.as_tensor(DELTA_PHASOR, dtype=z.real.dtype, device=z.device)
    return torch.complex(TernarySoftSTE.apply(z.real, gamma, delta),
                         TernarySoftSTE.apply(z.imag, gamma, delta))

class ComplexQuantizedLinear(nn.Module):
    """Complex linear layer whose quantized weight remains in the autograd graph."""
    def __init__(self, in_features: int, out_features: int,
                 delta_init: float = DELTA_WEIGHT_INIT):
        super().__init__()
        self.weight = nn.Parameter(
            torch.randn(in_features, out_features, dtype=torch.cfloat) * 0.02)
        delta_init = min(max(delta_init, 1e-4), 1.0 - 1e-4)
        self.delta_raw = nn.Parameter(
            torch.tensor(math.log(delta_init / (1.0 - delta_init))))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        q_weight, _ = ternary_quantize(self.weight, delta=self.delta_raw.sigmoid())
        return x @ q_weight

class QuaternarySoftSTE(Function):
    """2-bit rounding onto {-2,-1,+1,+2} (no zero state): components never
    fall into the origin channel, protecting phase-critical heads (§7.2)."""
    @staticmethod
    def forward(ctx, x, gamma, stochastic_ties):
        scaled = x / gamma
        ctx.save_for_backward(scaled)
        mag = torch.where(scaled.abs() > 1.5,
                          torch.full_like(scaled, 2.0),
                          torch.ones_like(scaled))
        if stochastic_ties:
            tie_sign = torch.empty_like(scaled).bernoulli_(0.5).mul_(2).sub_(1)
        else:
            indices = torch.arange(scaled.numel(), device=scaled.device).reshape(scaled.shape)
            tie_sign = indices.remainder(2).mul(2).sub(1).to(scaled.dtype)
        sign = torch.where(scaled != 0, torch.sign(scaled), tie_sign)
        return gamma * sign * mag

    @staticmethod
    def backward(ctx, grad_out):
        (scaled,) = ctx.saved_tensors
        surrogate = (1 - torch.tanh(scaled) ** 2).clamp(0.0, 1.0)
        return grad_out * surrogate, None, None

def quaternary_quantize(w: torch.Tensor,
                        training: bool = False) -> tuple[torch.Tensor, torch.Tensor]:
    """16 joint Cartesian states, 12 phase directions, no origin."""
    if torch.is_complex(w):
        gamma = w.abs().mean(dim=0, keepdim=True)
    else:
        gamma = w.abs().mean(dim=0, keepdim=True)
    gamma = gamma.clamp(min=1e-8).detach()
    if torch.is_complex(w):
        wr = QuaternarySoftSTE.apply(w.real, gamma, training)
        wi = QuaternarySoftSTE.apply(w.imag, gamma, training)
        return torch.complex(wr, wi), gamma
    return QuaternarySoftSTE.apply(w, gamma, training), gamma

def select_quaternary_heads(phase_grad_energy: torch.Tensor,
                            budget: float = 0.10) -> torch.Tensor:
    """Select the ~10% most phase-sensitive heads (§4.5, §7.2).
    phase_grad_energy: per-head EMA of |d loss / d phase| accumulated
    during QAT. Returns a boolean mask of heads to quantize as quaternary."""
    n_heads = phase_grad_energy.numel()
    k = max(1, int(round(budget * n_heads)))
    top = torch.topk(phase_grad_energy, k).indices
    mask = torch.zeros(n_heads, dtype=torch.bool, device=phase_grad_energy.device)
    mask[top] = True
    return mask

12.2 Base-3 Packing Codec (5 trits per byte)

Once training is complete, ternary tensors are packed as unsigned bytes using \(3^5 = 243 \le 256\). This is the on-disk format described in §11.2; every deployed CRATER checkpoint is a collection of these packed byte arrays plus the FP16 \(\gamma_w\) and \(\Delta_\ell\) vectors (quaternary head tensors pack separately at 2 bits per component).

import numpy as np

_TRIT_TO_VAL = np.array([-1, 0, 1], dtype=np.int8)
_VAL_TO_TRIT = {-1: 0, 0: 1, 1: 2}

def pack_base3(x: np.ndarray) -> np.ndarray:
    """Pack ternary values in {-1,0,+1} into bytes, 5 trits per byte."""
    trits = np.vectorize(_VAL_TO_TRIT.get)(x.astype(np.int8)).astype(np.uint16)
    pad = (-len(trits)) % 5
    if pad:
        trits = np.concatenate([trits, np.zeros(pad, dtype=np.uint16)])
    packed = np.zeros(len(trits) // 5, dtype=np.uint8)
    for k in range(5):
        packed = packed + (trits[k::5] * (3 ** k)).astype(np.uint8)
    return packed

def unpack_base3(packed: np.ndarray, n: int) -> np.ndarray:
    """Inverse of pack_base3. Returns the first n trits as int8 in {-1,0,+1}."""
    trits = np.empty(len(packed) * 5, dtype=np.uint8)
    p = packed.astype(np.uint16)
    for k in range(5):
        trits[k::5] = p % 3
        p //= 3
    return _TRIT_TO_VAL[trits[:n]]

12.3 Complex LayerNorm (magnitude-only variant)

The positive real RMS/gain core preserves phase. The optional complex bias generally does not, so the complete affine layer must not be called phase-preserving. Both parameters remain in FP16 during inference.

class ComplexLayerNorm(nn.Module):
    def __init__(self, dim: int, eps: float = 1e-5):
        super().__init__()
        self.gain_raw = nn.Parameter(torch.full((dim,), 0.5413248546, dtype=torch.float32))
        self.bias     = nn.Parameter(torch.zeros(dim, dtype=torch.cfloat))
        self.eps      = eps

    def forward(self, z: torch.Tensor) -> torch.Tensor:
        # z: (..., dim) complex
        rms = z.abs().square().mean(dim=-1, keepdim=True).sqrt().clamp(min=self.eps)
        z_normed = z / rms                              # magnitude-only, phase preserved
        gain = nn.functional.softplus(self.gain_raw).add(1e-4).to(z.real.dtype)
        return gain * z_normed + self.bias              # per-channel real gain + complex bias

12.4 Angular Attention

Angular attention normalizes \(Q\) and \(K\) to unit modulus per component, averages per-component phase agreements \(\text{Re}(q\bar{k})\), and applies a positive learned per-head temperature. When imaginary parts vanish, it reduces to signed-coordinate agreement, not standard dot-product attention or vector-level cosine similarity.

class AngularAttention(nn.Module):
    def __init__(self, dim: int, n_heads: int, causal: bool = True):
        super().__init__()
        assert dim % n_heads == 0
        self.n_heads = n_heads
        self.head_dim = dim // n_heads      # complex dims per head
        self.causal  = causal
        # Learnable per-head temperature gamma_h (§4.3), exp-parameterized > 0.
        self.log_tau = nn.Parameter(torch.zeros(n_heads))
        # Wq/Wk are ordinary weights, not unit phasors. Their quantizer uses
        # the generic weight threshold; normalization happens after matvec.
        self.Wq = ComplexQuantizedLinear(dim, dim)
        self.Wk = ComplexQuantizedLinear(dim, dim)
        self.Wv = ComplexQuantizedLinear(dim, dim)
        self.Wo = ComplexQuantizedLinear(dim, dim)

    @staticmethod
    def phasor_normalize(z: torch.Tensor, eps: float = 1e-4) -> torch.Tensor:
      return z * torch.rsqrt(z.abs().square() + eps ** 2)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: (B, T, dim) complex
        B, T, D = x.shape
        H, dh = self.n_heads, self.head_dim
        q = self.Wq(x).view(B, T, H, dh)
        k = self.Wk(x).view(B, T, H, dh)
        v = self.Wv(x).view(B, T, H, dh)

        q_valid = (q.abs() > 1e-4).float()
        k_valid = (k.abs() > 1e-4).float()
        q_hat = self.phasor_normalize(q) * q_valid
        k_hat = self.phasor_normalize(k) * k_valid
        inner  = torch.einsum("bthd,bshd->bhts", q_hat, k_hat.conj())
        valid_pairs = torch.einsum("bthd,bshd->bhts", q_valid, k_valid)
        scores = inner.real / valid_pairs.clamp(min=1.0)
        scores = scores.masked_fill(valid_pairs == 0, 0.0)
        scores = scores * self.log_tau.exp().view(1, H, 1, 1)   # learnable temperature
        if self.causal:
            # Autoregressive mask: position t can only attend to s <= t.
            mask = torch.ones(T, T, dtype=torch.bool, device=scores.device).triu(1)
            scores = scores.masked_fill(mask, float("-inf"))
        attn   = torch.softmax(scores, dim=-1)   # real-valued weights
        out    = torch.einsum("bhts,bshd->bthd", attn.to(v.dtype), v)
        return self.Wo(out.reshape(B, T, D))

12.5 One-Step QAT Training Loop

Quantization occurs inside ComplexQuantizedLinear.forward. The master parameter remains full precision, the forward value is ternary, and the clipped-tanh surrogate supplies its backward derivative. No .data mutation, no_grad region, or save-and-restore operation is used.

def train_step(model, opt, batch, targets):
    logits = model(batch)  # functional quantization occurs inside each linear layer
    loss   = nn.functional.cross_entropy(logits.reshape(-1, logits.size(-1)),
                                         targets.reshape(-1))
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

12.6 What is deliberately omitted

Three components from the full CRATER specification are not implemented in this reference because they add engineering surface without changing the mathematical story: (i) the CMoE router and expert dispatch (a standard top-k gating over the phasor-normalized token representation, plus the per-expert LoRA injection of §7.3); (ii) the IRW retrieval fusion (a gated cross-attention block whose gate follows the ReZero warmup schedule of §4.4, with the adaptive cadence policy); and (iii) the multi-frequency positional encoding with relative rotation, which is a straightforward complex generalization of RoPE. Each of these can be added in ~40 additional lines and is scheduled for the public release accompanying the §5 evaluation protocol.

Memory and Operation Calculator

Estimate footprint per token and per context length for a ternary-complex model.

Formulas (decimal GB): packed ternary weights = 0.4 byte × complex weights; KV cache = 2 × layers × d_real × context × bytes per real scalar. Each non-zero real or imaginary weight component contributes to two signed accumulations, so signed contributions/token = 4 × complex weights × (1 − component sparsity). The operation estimate treats all entered weights as active and therefore does not model top-k MoE routing; its default 7B-complex shape is illustrative and is not matched to the 7B-real comparison row. The footprint excludes scratch, activations, quaternary heads, scales, embeddings/head, router/LoRA, IRW, allocator overhead, and mmap residency.

Appendix A · Reproducibility Checklist

Settings needed to replicate a CRATER training run once the code is released.

# Pseudo-code for QAT training step
for batch in loader:
    x = embed(batch) + positional_encoding(batch)   # multi-frequency + relative rotation
    z = complex_lift(x)              # real → complex, once per token
    for layer in crater_layers:      # transitions stay in C^d_complex, d_complex=d_real/2
        z = z + angular_attention(cln(z), quantized_qkv(layer))  # learned Δ on weights;
                       # quaternary mask per head
        z = z + irw_retrieve(cln(z))      # adaptive cadence, ReZero-warmed gate
        z = z + c_moe(cln(z), quantized_experts(layer))          # + per-expert LoRA
    x = real_readout(z)              # Hermitian projection (Stage 7)
    logits = output_head(x)
    loss = cross_entropy(logits, targets) + phase_margin(z) + coherence_penalty(z)
    # Soft straight-through gradient through the quantizer rounding
    loss.backward()
    clip_grad_norm_(parameters, 1.0)
    optimizer.step()

Appendix B · Frequently Asked Questions

Why complex-valued representations instead of real ones?

Complex numbers encode magnitude and phase in the same object. In CRATER, phase is treated as relational structure: two vectors with similar phase produce high angular attention, which is a different geometry from ordinary dot-product similarity.

Is CRATER's inference fully multiplier-free?

No, and the paper never claims that. The multiplier-free property is scoped to the inner loops of the ternary linear layers, which dominate the FLOP count. The router, gates, CLN, softmax, embeddings, LoRA adapters and output head deliberately remain FP16 and use ordinary multiplications (§11.4, §11.6).

If 30–50% of components quantize to zero, how can phase be "preserved"?

It is not preserved for zero outputs. Generic non-zero weights lie on an eight-direction grid but do not receive the π/8 phasor bound; that bound is reserved for optional direct quantization of calibrated unit phasors. Post-matvec error is measured empirically.

Isn't the phase destroyed at the readout since logits are real?

The collapse to real probabilities is required by the language-modeling task, but the phase acts on the logits before that collapse: the Hermitian inner product between the hidden state and each complex vocabulary embedding depends on their phase alignment, and the phase-margin objective (§7.1) trains exactly that alignment to be discriminative. Phase is consumed by the output, not discarded.

Is CRATER more compact than BitNet b1.58?

No. The ≈9× compression compares CRATER's packed ternary weights to the FP16-complex representation of the same model. At matched real-parameter count, a real 1.58-bit checkpoint is comparable in size. CRATER's claimed advantage over real 1.58-bit models is representational (phase), not compression (§3).

How is the KV cache different from the weight compression problem?

The weights are fixed after training, so they can be packed once. The KV cache grows linearly with context length and stores every key/value vector generated during decoding; it eventually dominates memory even when weights are heavily compressed. Angular attention offers a specific remedy: since keys are phasor-normalized, the K side can be stored as 8-bit angles (§8).

Is ternary quantization lossless?

No. Quantization always introduces approximation error, and Theorem 1 bounds only the per-component angular error — the composed error of a full matvec is measured, not proven, for the default dither-free recipe (§4.7 scope note), though §4.7.1 proves a conditional composed bound (Theorem 2) if an explicit dither operator is added. The claim is that calibrated thresholds, the quaternary-head budget and QAT keep perplexity close to full precision while enabling multiplier-free ternary inner loops.

How does the model stay up to date between full retraining runs?

Incremental IRW index writes provide immediate non-parametric updates, while per-expert LoRA adapters provide limited router-targeted adaptation. Neither eliminates retraining: drift, backbone deficiencies, tokenizer changes, or broad distribution shifts can still require a full run and validated checkpoint swap.

When will benchmark numbers be available?

Not yet. CRATER is an architecture proposal. Benchmark results require the controlled ablations listed in the evaluation protocol and the reproducibility checklist.

Appendix C · Glossary

Core acronyms used throughout this document.

CLN

Complex LayerNorm — normalization with a positive real scaling core that stabilizes magnitude while preserving phase.

IRW

Imaginary Reflection Window — the integrated retrieval mechanism that queries an external index inside the forward pass and fuses candidates through a reflection gate (the "imaginary" is optical: a virtual image behind a semi-transparent mirror).

CMoE

Complex Mixture of Experts — sparse routing across complex-valued expert networks for capacity scaling with bounded compute.

LoRA

Low-Rank Adaptation — small trainable low-rank matrices attached to frozen weights, used per CMoE expert for continual specialization.

STE

Straight-Through Estimator — gradient trick that passes gradients through the non-differentiable quantization rounding during training (soft clipped-tanh variant used in CRATER).

QAT

Quantization-Aware Training — training with quantization simulated in the forward pass so weights adapt to the low-bit regime.

RAG

Retrieval-Augmented Generation — augmenting a language model with an external knowledge store at inference time.

RoPE

Rotary Position Embedding — relative position encoding via rotation of query/key vectors.

Δ_ℓ

Learnable per-layer quantization threshold in units of γ_w; only the absolute threshold Δ_ℓ·γ_w has geometric meaning (§4.5.1).

References

1. Vaswani, A. et al. (2017). Attention Is All You Need. BibTeX
2. Trabelsi, C. et al. (2018). Deep Complex Networks. BibTeX
3. Hendrycks, D. and Gimpel, K. (2016). Gaussian Error Linear Units. BibTeX
4. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. BibTeX
5. Borgeaud, S. et al. (2022). Improving Language Models by Retrieving from Trillions of Tokens. BibTeX
6. Katharopoulos, A. et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. BibTeX
7. Su, J. et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. BibTeX
8. Zhang, B. and Sennrich, R. (2019). Root Mean Square Layer Normalization. BibTeX
9. Kreutz-Delgado, K. (2009). The Complex Gradient Operator and the CR-Calculus. BibTeX
10. Hubara, I. et al. (2016). Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. BibTeX
11. Rastegari, M. et al. (2016). XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. BibTeX
12. Ma, S. et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. BibTeX
13. Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. BibTeX
14. Bachlechner, T. et al. (2021). ReZero is All You Need: Fast Convergence at Large Depth. BibTeX
15. Liu, Z. et al. (2022). Swin Transformer V2: Scaling Up Capacity and Resolution. BibTeX
16. Khandelwal, U. et al. (2020). Generalization through Memorization: Nearest Neighbor Language Models. BibTeX
17. Wu, Y. et al. (2022). Memorizing Transformers. BibTeX
18. Li, F., Zhang, B., and Liu, B. (2016). Ternary Weight Networks. BibTeX
19. Zhu, C., Han, S., Mao, H., and Dally, W. J. (2017). Trained Ternary Quantization. BibTeX
20. Eilers, F. and Jiang, X. (2023). Building Blocks for a Complex-Valued Transformer Architecture. BibTeX
21. Henry, A., Dachapally, P. R., Pawar, S., and Chen, Y. (2020). Query-Key Normalization for Transformers. BibTeX
22. Bengio, Y., Courville, A., and Vincent, P. (2013). Representation Learning: A Review and New Perspectives. BibTeX
23. Jégou, H., Douze, M., and Schmid, C. (2011). Product Quantization for Nearest Neighbor Search. BibTeX
24. Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. BibTeX
25. Fedus, W., Zoph, B., and Shazeer, N. (2021). Switch Transformers. BibTeX
26. Tancik, M. et al. (2020). Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. BibTeX
27. Schuchman, L. (1964). Dither Signals and Their Effect on Quantization Noise. BibTeX
28. Ashkboos, S. et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. BibTeX
29. Liu, Z. et al. (2024). SpinQuant: LLM Quantization with Learned Rotations. BibTeX
30. Arjovsky, M., Shah, A., and Bengio, Y. (2016). Unitary Evolution Recurrent Neural Networks. BibTeX
31. Deng, J., Guo, J., Yang, J., Xue, N., Kotsia, I., and Zafeiriou, S. (2019). ArcFace: Additive Angular Margin Loss for Deep Face Recognition. BibTeX
32. Wang, H. et al. (2018). CosFace: Large Margin Cosine Loss for Deep Face Recognition. BibTeX
33. Liu, W. et al. (2017). SphereFace: Deep Hypersphere Embedding for Face Recognition. BibTeX
34. Charikar, M. S. (2002). Similarity Estimation Techniques from Rounding Algorithms. BibTeX
35. Mardia, K. V., and Jupp, P. E. (2000). Directional Statistics. BibTeX