“`html
Recently, there has been discussion about the proportion of active parameters in large MoE (Multi-Output Encoder) models. A user observed that some very large models like 1T and 1.6T have significantly fewer active parameters than anticipated.
- This observation raises questions about whether a cap exists on how many active parameters can be effectively used without diminishing the quality of results.
- Some speculate if there might be an architectural shift or practical limits preventing further increases in active parameter counts beyond a certain point.
- The user wonders if it will ever be possible to see models with even larger numbers like 2T/A20B, and if so, whether they would maintain the same level of performance as smaller models do now.
“`
– The proportion between total and active parameters in very large MoE models is lower than expected.
– There are questions about whether there’s an architectural or practical limit to increasing active parameter counts.
– Users are curious if such a cap exists, particularly for very large models.
Source Read original →
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.
