CRATER: Complex Reflective Angular Transformer with External Retrieval

Dr. Love & AI · Independent Research · Preprint v1.2 (architecture proposal, under review)

Dr.Love@dr.com

Abstract

We present CRATER, a transformer-family architecture built around complex-valued internal blocks, angular similarity for attention, and integrated retrieval through Imaginary Reflection Windows (IRW). The design combines five core directions: complex state construction, phase-preserving normalization, angular attention, multi-query retrieval fusion, and sparse Complex Mixture of Experts (CMoE). This page is a static publication-style conversion of the website version, intended for direct reading and archival in single-file HTML format.

Keywords: complex-valued neural networks, angular attention, retrieval-augmented generation, phase normalization, complex LayerNorm, complex mixture of experts.

Notation

SymbolDescription
\(z_t \in \mathbb{C}^{d/2}\)Complex hidden state at position \(t\)
\(\phi_{norm}\)Principal phase in interval \((-\pi, \pi]\)
\(\alpha^{(h)}_{ts}\)Angular attention score for head \(h\)
\(\mathrm{CLN}(z)\)Complex LayerNorm with phase-preserving core
\(\mathrm{cGELU}(z)\)Component-wise GELU on real and imaginary parts
\(q_t^{(m)}\)Retrieval query \(m\) generated from complex state
\(g_t\)Reflection gate controlling retrieval fusion
\(\mathcal{E}_e\)Expert network \(e\) in CMoE
\(\mathcal{K}\)External retrieval memory

1. Introduction

Standard transformers rely on real-valued dot-product attention. CRATER explores a different geometry: internal complex blocks where magnitude and phase are treated as first-class computational signals. The objective is to unify representation, attention, and retrieval under angular consistency rather than pure magnitude similarity.

The architecture is proposed as an engineering and research framework. Current performance statements are projected or preliminary and require broader multi-seed validation. The emphasis of this static paper is formulation clarity and implementation-level traceability.

  • Complex state representation with phase-aware normalization and activation.
  • Angular attention replacing standard dot-product logits.
  • IRW retrieval integrated into the forward path.
  • Sparse CMoE for capacity scaling with bounded compute growth.
  • Stability controls: clipping, phase regularization, magnitude regularization.

2. Related Work

2.1 Complex-Valued Neural Networks

Prior work in complex-valued deep learning demonstrated representational advantages in oscillatory and phase-sensitive domains. CRATER adapts these principles to transformer blocks rather than using complex arithmetic as a marginal add-on.

2.2 Attention Mechanisms

Beyond scaled dot-product attention, many efficient alternatives exist. CRATER focuses on angular relationships using normalized phasors and cosine-like phase agreement terms.

2.3 Retrieval-Augmented Generation

Existing RAG systems often keep retrieval loosely coupled. CRATER places retrieval in the core computation via multi-query complex retrieval signals and weighted fusion.

3. Architecture Overview

CRATER can be read as a seven-stage pipeline. Layers compute in complex space, while residual interfaces can be carried as stacked real-imaginary channels for interoperability.

Stage 1: Multi-frequency positional encoding with learnable bands.
Stage 2: Real-to-complex lift and rotational transform.
Stage 3: Angular attention with head-wise scaling.
Stage 4: IRW retrieval, candidate reranking, gated fusion.
Stage 5: Complex residual transport and phase-aware skip path.
Stage 6: Sparse complex experts (CMoE).
Stage 7: Projection back to output logits.

4. Mathematical Framework

4.1 Multi-Frequency Positional Encoding

\[ \mathrm{PE}(t,i)=\frac{1}{\sqrt{K}}\sum_{k=1}^{K}\sin\left(\omega_k^{(i)}t+\delta_k^{(i)}\right) \]

Multiple learnable frequency bands allow local and long-range periodic structure to coexist, while phase offsets prevent a degenerate zero-position signature.

4.2 Complex State Construction and CLN

\[ z_t = h_t^{re} + i\,h_t^{im},\quad \mathrm{CLN}(z_t)_j = \gamma_j\frac{z_{t,j}}{\mathrm{RMS}(z_t)}+\beta_j \]

CLN uses a positive real scaling core to stabilize magnitude while preserving phase in the normalization path. A complex bias remains optional and can shift phase after normalization.

4.3 Angular Attention

\[ s_{ts}^{(h)}=\frac{\gamma_h}{\sqrt{d_h}}\sum_j\mathrm{Re}\left(u(Q_{t,j}^{(h)})\overline{u(K_{s,j}^{(h)})}\right) \]

Angular similarity is computed through normalized phasors. This produces phase-agreement logits with optional kernel approximations for non-causal linear-time variants.

4.4 IRW Retrieval and Fusion

Each token state emits multiple retrieval queries. A fast external index proposes candidates, then CRATER reranks and fuses them with a reflection gate. Retrieval remains external but is integrated into the model update rule instead of only preprocessing context.

5. Predicted Evaluation Targets and Planned Protocol

The evaluation protocol is designed for language modeling, question answering, and retrieval-augmented generation settings, with explicit single-seed vs multi-seed reporting separation.

  • Benchmarks: WikiText-103, Natural Questions, KILT-style retrieval tasks.
  • Baselines: standard transformer and retrieval baselines with matched scale.
  • Metrics: perplexity, exact match/F1, retrieval precision at top-k, calibration quality.
  • Training stability: gradient spikes, phase variance evolution, norm distribution.

6. Discussion

CRATER offers a coherent geometric viewpoint in which phase acts as relational structure and magnitude as confidence/energy. The cost is increased architectural complexity and additional hyperparameter surfaces. The framework is promising but should be read as exploratory until larger-scale replication confirms gains.

7. Extensions of the CRATER Framework (Future Work)

Future modules include RoPE-style relative phase encoding, calibrated confidence gating, conjugate attention heads for contradiction sensitivity, and phase-coherence-guided decoding.

Longer-term directions include quaternion or hyperbolic variants, latent reflective deliberation loops, and memory compression strategies preserving both amplitude and phase structure.

8. Limitations and Mitigations

  • Compute overhead versus real-valued baselines.
  • Sensitivity to phase regularization weight schedules.
  • Dependency on external retrieval quality and ANN index maintenance.
  • Current evidence includes preliminary and projected results.

9. Conclusion

CRATER proposes a full-stack complex transformer formulation where angular attention and retrieval are first-class components. This static HTML version preserves the paper-like structure in a standalone format compatible with the Humanity root collection.

10. Evidence Framework and Confidence

This document blends established transformer literature with exploratory complex-valued architectural extensions. Confidence should be interpreted by claim type, not as a single global score.

  • High confidence: cited baseline methods and mathematical operators from peer-reviewed literature.
  • Medium confidence: controlled experimental trends and ablation-level behavior reports.
  • Medium-Low confidence: scaling projections and generalized deployment performance.

References

  1. Vaswani, A. et al. (2017). Attention Is All You Need.
  2. Trabelsi, C. et al. (2018). Deep Complex Networks.
  3. Hendrycks, D. and Gimpel, K. (2016). Gaussian Error Linear Units.
  4. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
  5. Borgeaud, S. et al. (2022). Improving Language Models by Retrieving from Trillions of Tokens.
  6. Katharopoulos, A. et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.
  7. Su, J. et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding.
  8. Zhang, B. and Sennrich, R. (2019). Root Mean Square Layer Normalization.
  9. Kreutz-Delgado, K. (2009). The Complex Gradient Operator and the CR-Calculus.
Back to Humanity