AMD releases fully open source MoE model Intella-MoE 16B parameter scale challenges mainstream open source models

On July 25, AMD announced the launch of the fully open source hybrid expert model (MoE) Intella-MoE. The model has 16 billion total parameters, and each Token activates 2.8 billion parameters. It claims to have leading performance among open source language models of the same scale. AMD said that Intella-MoE is completely based on its own AMDInstinctMI300X and MI325X GPUs, as well as the ROCm software stack, and is trained from scratch, and uses architectural innovations such as Gated Multi-headLatentAttention (Gated MLA) and FarSkip-Collective to improve training and reasoning efficiency. Performance tests show that the Intella-MoE-16B-A3B basic model has an average score of 76.7 points, ranking ahead among all open source models, surpassing models such as SmolLM3 -3B and OLMo-3- 7B, and only activating 2.8 billion parameters., compete with larger models. In addition, the model supports 64KToken long context processing and completes complete training processes such as pre-training, mid-term training, long context extension, supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (RL). AMD simultaneously disclosed all Intella-MoE model weights, training configurations, data ratios, intermediate checkpoints and inference codes to promote open AI model research and reproduction. AMD said that Intella-MoE demonstrates the ability to conduct large-scale MoE model training based on AMD's hardware and open software ecosystem, and will continue to promote the development of larger-scale, stronger reasoning capabilities and more efficient open source language models in the future.