Mixture of Experts (MoE). Instead of running all 671B parameters on every query, DeepSeek activates only ~37B per token. Dense models are hitting diminishing returns. MoE is winning on efficiency.
nicornle