Xiaomi’s MiLM Plus team has released PROVE, a new set of evaluation metrics for object removal models that align better with human perception than existing standards. The release includes RC-S for spatial coherence and RC-T for temporal consistency, paired with PROVE-Bench, a two-tier video benchmark. This work was accepted at ACM MM 2026.
Why current metrics fail
Object removal models have improved faster than the tools used to judge them. Modern diffusion erasers reconstruct shadows, reflections, and occluded structures convincingly. Yet standard scores like PSNR, SSIM, LPIPS, ReMOVE, and CFD frequently rank outputs incorrectly. The root cause is structural: erasure is an ill-posed, one-to-many task, meaning no single ground truth exists to compare against. PROVE addresses this by scoring the edited region locally using sliding-window Maximum Mean Discrepancy over DINOv2 features, without requiring a reference video.
Deployment requirements
PROVE arrives as an Apache 2.0 PyTorch repository with one CLI entry point, run_prove_metrics.py. It requires Python 3.10, PyTorch 2.6, Transformers 4.51, and DINOv2-giant weights. Masks are mandatory, with white pixels marking the removed object.
Any team with one GPU and a mask pipeline can run it. RC-S runs at 134.6 ms per frame on a single RTX 4090, making nightly CI gates practical for startups. Enterprises with large editing catalogs gain the most benefit since no paired ground truth is needed.
Key industries include smartphone gallery apps, short-video editing, e-commerce catalog cleanup, advertising and stock media, film and VFX post-production, real estate imagery, and privacy redaction in mapping archives.
Primary users are CV researchers and applied AI engineers on editing teams. MLOps engineers follow, wiring the score into CI pipelines. Product managers run vendor comparisons third.
The tool does not fit real-time on-device scoring. It also struggles with side effects such as large shadows or reflections extending beyond the cropped evaluation region.
Three documented failure modes
Object removal is ill-posed and one-to-many: many restorations are plausible for the same hole, so no unique ground truth exists.
- Full-reference bias: PSNR, SSIM and LPIPS assume point-to-point correspondence, rewarding copy-paste over genuine erasure. Residual shadows occupy few pixels and incur almost no penalty. Worse, cutting diffusion inference steps improves PSNR and SSIM while visual quality collapses — a regression to the mean.
- No-reference blind spots: On ROSE-Bench, the authors progressively blur the masked region. Neither ReMOVE nor CFD degrades; both eventually surpass their unblurred baselines. CFD’s SAM-based hallucination term also misfires: a legitimately restored bicycle seat is flagged as a hallucination, scoring worse than the unedited input.
- Temporal insensitivity: Temporal Consistency and Temporal Flickering operate on full-frame features. Since the edited area is a small fraction of the frame, injected Random Drop and Random Replace corruptions on DAVIS barely move them, and sometimes move them the wrong way.
How RC-S and RC-T work
Both metrics share one idea: local distribution matching in deep feature space instead of global aggregation.
RC-S (spatial): Connected-component analysis splits the mask into independent targets. Each bounding box is expanded by one third of its side length, the crop goes to DINOv2, and the mask is downsampled to feature resolution. A w×w window slides across the feature map, computing squared Maximum Mean Discrepancy with a Gaussian RBF kernel between masked and local background features. Scores are averaged per target, then across targets.
RC-T (temporal): Adjacent frames are cropped jointly under the union of their masks to avoid misalignment, then MMD is computed only inside the intersection — the region restored in both frames. An ablation shows that removing the crop makes RC-T blind to injected corruption.
Performance results
Against human rankings from 20 participants aggregated by Borda count, RC-S reaches 0.59 average Kendall’s τ and 0.66 Spearman’s ρ. This compares to 0.26/0.29 for ReMOVE and 0.16/0.18 for CFD. It ranks first on five of six benchmarks.
Background-only FR variants correlate negatively on most datasets. On RORD-Val, RC-S prefers the clean image over blurred and region-swapped variants in 100% of cases; ReMOVE manages 60.06% and CFD 49.27% under blur. RC-T responds monotonically to rising corruption where TC and TF do not.
Ablations isolate each contribution: DINOv2 (0.59 avg τ) beats DINOv3 (0.51) and SAM (0.44); dropping the sliding window costs 0.11; swapping MMD for cosine costs 0.07. RC-S is also the cheapest spatial metric tested, 13.7× faster than CFD.
PROVE-Bench details
PROVE-M holds 80 paired videos: tripod-captured input and target-free footage shot within two minutes, SAM3 masks refined frame by frame, three-stage quality control, then Ken Burns-style motion augmentation applied synchronously to the triplet. Each clip is 81 frames at 1080p.
PROVE-H adds 100 hard videos without ground truth — crowds, flowing water, flames, textured terrain, multi-puddle reflections, fast motion — using unrefined SAM3 masks on purpose.
On the public leaderboard, SVOR (1.3B) leads combined RC-S at 0.5197; EffectErase leads RC-T at 0.2525.
What it means
For developers building removal tools, this changes the workflow from guessing which metric to trust to using a system that correlates with human judgment. The open-source code allows immediate integration into CI pipelines, while the benchmark provides a standard for comparing models on real-world video data.
The release includes the Paper, GitHub Repo, Project Page, and Dataset. Researchers can follow the team on Twitter or join their Telegram channel.
