Moonshot AI has released PerceptionBench, a new test showing that leading multimodal models struggle to interpret images. No model reached 60 percent accuracy. GPT-5.6 Sol came closest at 59.7 percent, yet it still failed more than four times out of ten.
The benchmark isolates visual ability from logic and external knowledge. It breaks sight down into ten specific skills. Questions require no reasoning or outside facts. The answer depends entirely on what is in the picture.
Categories built from real errors, not theory
The team behind Kimi, the Chinese AI assistant, noted that previous tests cover only narrow slices of visual failure. They reviewed 42 open-source benchmarks. Each showed different error profiles. No single test captured the whole picture.
Instead of defining categories beforehand, the authors built their list from actual model mistakes. They traced each failure back to the first wrong step. This produced ten skill domains: Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.
The dataset started with over 17,000 verified questions. Moonshot AI is publishing 3,000 tasks. Sixty percent came from attributed model errors. The remaining 40 percent used augmented images. The questions look simple. One asks where a symbol sits on a clock face. Another counts flowers inside a red box. A third decides which of two pencil cups shows a gray-pink combo versus a solid pink with a cartoon design.
No model breaks 60 percent
Sixteen frontier models took the test. GPT-5.6 Sol led with 59.7 percent. Kimi K3 followed at 58.5 percent. Claude Fable 5 scored 57.2 percent. Gemini 3.1 Pro reached 56.2 percent. GPT-5.5 came in at 55.8 percent. Open-source models trailed further behind. Qwen3.5-397B-A17B scored 47.5 percent. GLM-4.6V managed only 32.5 percent.
Category-level results show more difference than the overall ranking. Models with similar aggregate scores diverge sharply in individual areas. Hallucination remains the weakest skill across the board. GPT-5.6 Sol scored just 26.9 percent there. Gemini 3.5 Flash ranked among the best with 50.6 percent. This sub-test checks whether models invent objects that do not exist when the correct answer is simply “zero”.
Many “reasoning errors” are actually perception failures
The authors argue that many multimodal model failures usually blamed on reasoning actually happen at the perception level. When a model botches a multi-step task, the first step, correctly reading the image, has often already gone wrong.
PerceptionBench breaks those questions into perception-only sub-questions. This makes it possible to pinpoint which specific visual ability is failing. The dataset and evaluation code are available on GitHub at MoonshotAI/PerceptionBench.
A well-known problem with little progress
Moonshot AI builds the open-source Kimi K3. It has closed the gap to Claude Fable 5 and GPT-5.6 Sol to within a few points on general benchmarks. K3 still lags well behind in specialised areas like offensive cybersecurity and complex math. On visual perception, K3 now performs on par with its Western rivals.
The same research team already released WorldVQA, a benchmark that separates object recognition from reasoning. The best model there, Gemini 3 Pro, fell short of 50 percent at 47.4 percent. All models systematically overestimated their own confidence.
A separate study by Chinese institutions, with Moonshot AI’s involvement, used the BabyVision benchmark to show that frontier models fail at basic visual tasks tied to early childhood development. These include tracing lines or counting hidden blocks. Gemini 3 Pro scored 49.7 percent on those tasks. Humans hit 94.1 percent. The researchers attribute this gap to a verbalisation bottleneck where visual information gets translated into language and loses fidelity.
What it means
For people making things, the implication is clear. Tools that claim to understand images often do not. They struggle with basic facts like counting or spotting missing items. If a design tool or video editor relies on these models, the output will contain errors. The gap between human sight and machine sight remains wide. Fixing the foundation of how models see is required before logic can be trusted.
