A new dataset from Anthropic, named claude-protein-binder-design, contains 1,440 AI-designed miniproteins tested against 16 different targets. The release includes both computational predictions and real-world wet-lab results from two independent labs. This allows researchers to evaluate how well structure predictors identify successful binders, whether combining predictions improves performance, and how much disagreement stems from the assays themselves.
How the data is loaded
The code begins by installing only the missing dependencies required for the runtime. It then enumerates the repository once to build a map from subset names to file paths. This approach avoids hard-coding locations, which is necessary because the naming is not uniform. For instance, the subset wetlab_summary lives at data/tables/wetlab/summary.parquet, and a guessed path would fail silently.
With the map in place, the script pulls design_summary, which contains one row per design. There are 1,440 rows, providing enough data to carry every join needed downstream.
Understanding the evaluable set
The analysis defines the evaluable set by filtering on actual vendor calls rather than the binder_final column. The binder_final column is a boolean that records the 120 never-measured designs as False rather than missing. The code computes hit rates by model, campaign, generator, and target, wrapping each in a Wilson interval. This is required because several subgroups sit in the small-n regime where the normal approximation misbehaves.
The resulting plot shows that antigen choice swamps every other factor being compared. Any model comparison that does not stratify by target is mostly measuring which targets that model was pointed at.
Discriminative power of individual predictors
The script extracts predictors starting with ipsae_min_ and calculates the Area Under the Curve (AUC) for each. It also computes average precision scores. Bootstrapping is used to generate confidence intervals for these metrics.
The results show that AUCs land well above chance but far below the ~0.9 required to trust a single filter. That gap is the entire practical reason this dataset exists.
Consensus scoring
The final step creates consensus scores by ranking designs based on multiple predictors. It calculates the mean rank across all ipsae_min and sc_dockq columns for each design. This produces a single consensus score that aggregates the performance of individual structure predictors.
What it means
Researchers can now use this dataset to train target-aware classifiers. The goal is to determine if these signals can reliably predict experimental success. By moving from in-silico predictions to wet-lab validation, the data provides a concrete benchmark for assessing the utility of AI protein design tools.
