Authors: Tom Vatland

Affiliation: AI VISIONS AS

Date: March 22, 2025

Abstract

We propose RAREcast, a novel dual-model architecture designed to optimize large language model (LLM) efficiency using static vocabulary compression and intelligent token routing. Contemporary LLMs, while powerful, demand significant computational and energy resources (Radford et al., 2019). RAREcast addresses this by splitting tasks between a lightweight Model B and a full-capacity Model A. Before training, tokens are sorted by frequency and statically mapped, with the 70% least frequent tokens (IDs 30,001–100,000) represented by a single [RARE] token in Model B. Fully quantized (e.g., to 16-bit precision), Model B handles common interactions while offloading rare or complex sequences to Model A. Internally, Model B maintains connectivity to the full token space by computing the [RARE] token’s score as the maximum of its underlying token probabilities. Simulations suggest power consumption reductions of 40–55% and server capacity reductions of 30–40% compared to a baseline Transformer-based LLM with 70 billion parameters, aligning with trends in energy-efficient AI (Schwartz et al., 2020).

1. Introduction

Large language models (LLMs) built on Transformer-based architectures have enabled break-throughs in natural language processing (Vaswani et al., 2017; Radford et al., 2019). However, their high computational costs and energy consumption limit scalability and raise environmental concerns (Strubell et al., 2019). Reducing energy use also lowers the carbon footprint of AI deployments, enhancing sustainability. Prior research into model compression techniques, such as quantization (Jacob et al., 2018), knowledge distillation (Hinton et al., 2015), and Mixture of Experts (MoE) architectures (Shazeer et al., 2017; Du et al., 2022), has shown that selectively activating model components can yield efficiency gains.

Building on these principles, we present RAREcast, a dual-model architecture that leverages static token frequency distributions to minimize resource usage. RAREcast uses two models: a lightweight, fully quantized Model B optimized for high-frequency content, and a comprehensive Model A that handles rare or complex language phenomena. The key innovation is the use of a [RARE] token to compress infrequent vocabulary, inspired by subword tokenization techniques (Sennrich et al., 2016; Kudo and Richardson, 2018), and route control intelligently between models.

2. Methodology

2.1 Token Frequency Preprocessing

• Tokens are sorted by frequency prior to training based on the distribution observed in a representative training corpus, such as the Pile dataset (Gao et al., 2020). The top 30,000 tokens receive IDs 1–30,000 and are considered high-frequency, while the remaining 70,000 tokens (IDs 30,001–100,000) are designated as rare and grouped under a single [RARE] token in Model B. This static mapping, inspired by vocabulary optimization strategies (Takase et al., 2024), remains fixed after training and serves as the basis for compression and routing.

2.2 Model Architecture

• Model A: A full-capacity LLM with 70 billion parameters and a 100,000-token vocabulary, trained on a large, diverse corpus similar to those used in GPT models (Radford et al.,
  2019).
• Model B: A compressed version of Model A that collapses all rare tokens (IDs 30,001– 100,000) into a single [RARE] token, resulting in an effective vocabulary of 30,001 tokens.
  It is fully quantized from FP32 to FP16 (Jacob et al., 2018) to minimize memory and computational load, drawing on techniques from efficient inference literature (Gupta et al.,
  2015).

Internally, Model B preserves semantic connectivity to the full token space. Specifically, the probability of [RARE] during inference is computed as:

This approach, akin to dynamic scoring in MoE models (Shazeer et al., 2017), allows rare tokens to be considered in a compressed representation without explicit enumeration during most inference cycles.

2.3 Token Routing Mechanism

RAREcast uses dynamic control flow during inference, similar to adaptive computation strategies (Graves, 2016). Model B handles all inputs unless a [RARE] token is encountered:

• Input routing: If a token in the input stream maps to [RARE], control is passed to
  Model A for the remainder of the sequence.
• Output routing: During generation, if [RARE] is the top-ranked token or appears in top-k candidates, Model A resumes generation to ensure high-fidelity output.

Example: Input: “the quick brown fox hyperventilates.” Model B handles “the quick brown fox” using frequent tokens. “hyperventilates” is mapped to [RARE], so Model A takes over.
Output: Model B generates “the fox runs” and then predicts [RARE]; Model A completes with “the fox runs and hyperventilates loudly.” If all tokens are frequent (e.g., “the quick fox jumps”), Model B handles the entire sequence independently.

3. Results

3.1 Server Capacity Savings

3.2 Power Consumption Savings

4. Discussion

RAREcast introduces a practical efficiency mechanism for LLMs by routing low-frequency token processing to a full model only when necessary, building on dual-model approaches (Zhang et al., 2024). By statically defining rare tokens and offloading them to a compressed representation, Model B significantly reduces compute overhead while maintaining strong performance for common sequences, consistent with findings in vocabulary compression studies (Takase et al., 2024). However, routing errors could occur if the [RARE] token is mispredicted, a challenge also noted in MoE systems (Du et al., 2022). This could be addressed with uncertainty-based routing in future iterations, as explored in adaptive inference research (Graves, 2016).

A key technical innovation is the treatment of [RARE] token inference: rather than assigning it a fixed embedding, the score is dynamically calculated as the maximum probability of the original rare token set. This ensures semantic fidelity and contextual precision, aligning with techniques in efficient Transformer inference (Shen et al., 2020).

Future work may explore partial unfreezing or distillation of rare token pathways (Hinton et al., 2015) to enhance efficiency further, as well as hybrid routing strategies based on uncertainty thresholds (Graves, 2016).

5. Conclusion

RAREcast demonstrates that a dual-model architecture with static vocabulary compression and dynamic routing can yield substantial efficiency gains in LLM inference. By focusing computation on high-frequency patterns and outsourcing rare token processing, it achieves a favorable tradeoff between cost and capability, paving the way for scalable, sustainable AI deployments (Schwartz et al., 2020).

Acknowledgments
The author thanks AI VISIONS AS for providing computational resources and support during this research.

Competing Interests
The author declares no competing interests.

References

• • Du, N., Huang, Y., Dai, A. M., et al. (2022). GLAM: Efficient scaling of language models with mixture-of-experts. In Proceedings of the 39th International Conference on Machine Learning, PMLR, 162:1246–1266.
  • Gao, L., Biderman, S., Black, S., et al. (2020). The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
  • Graves, A. (2016). Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983.
  • Gupta, S., Agrawal, A., Gopalakrishnan, K., et al. (2015). Deep learning with limited numerical precision. In Proceedings of the 32nd International Conference on Machine Learning, PMLR, 37:1732–1741.
  • Hinton, G., Vinyals, O., Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  • Jacob, B., Kligys, S., Chen, B., et al. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2704–2713.
  • Kudo, T., Richardson, J. (2018). SentencePiece: A simple and language-independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226.
  • Radford, A., Wu, J., Child, R., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog.
  • Schwartz, R., Dodge, J., Smith, N. A., et al. (2020). Green AI. Communications of the ACM, 63(12):54–63. DOI:10.1145/3444946.
  • Sennrich, R., Haddow, B., Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL, 1:1715–1725.
  • Shazeer, N., Mirhoseini, A., Maziarz, K., et al. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
  • Strubell, E., Ganesh, A., McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243.
  • Takase, S., Kiyono, S., Kobayashi, K., et al. (2024). Large vocabulary size improves large language models. arXiv preprint arXiv:2406.16508.
  • Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. In Advances in Neural Information Processing Systems, 30:5998–6008.
  • Zhang, L., Li, H., Wang, Y., et al. (2024). Optimizing large language models for efficiency: A dual-model architecture with dynamic vocabulary adjustment. Research Square, DOI:10.21203/rs.3.rs-6247457/v1