The EdgeBench benchmark contains 51 tasks designed to test AI agents across different runtime environments and time budgets. ByteDance released the dataset on Hugging Face, allowing researchers to download specifications, judge logic, and raw scores for models including Claude Opus 4.8, GPT-5.5, and GLM-5.1.
Setting up the environment
The analysis begins by installing Python libraries such as pandas and scipy. The script pulls the dataset snapshot from the ByteDance-Seed repository. It then defines a list of time budgets ranging from two to twelve hours and standardises model names to ensure consistent labelling throughout the workflow.
Once cached locally, the code prepares to read the task specifications. This step involves parsing JSON files to extract details like internet requirements, submission paths, and the specific queries an agent must solve.
Examining the benchmark structure
Flattening the data reveals the composition of the 51 tasks. The analysis counts entries by category, base image runtime, and rescaling method. It also identifies how many tasks require internet access or operate in game mode.
Visualisations show the distribution of tasks across these categories. A single task entry is inspected to demonstrate the complexity of the evaluation, which includes an agent query and a specific parser for judging results.
Extracting leaderboard data
The script reads the repository README file to pull Markdown tables containing aggregate scores. It converts these tables into structured records, mapping raw scores against specific time budgets. The code handles missing values and normalises model names before creating a tidy dataset.
Comparisons are made between the task-level results and the aggregate 51-task summary provided in the README. This allows for a cross-check of the total performance metrics across the five models listed in the analysis.
Fitting scaling curves
The final stage involves calculating the mean score for each model at every time budget. These averages are then used to fit a log-sigmoid curve. This mathematical model helps visualise how performance improves as the interaction time budget increases from two to twelve hours.
What it means
This workflow demonstrates how to turn raw benchmark data into a usable format for performance analysis. By automating the extraction of scores and fitting scaling laws, researchers can objectively compare how different AI models perform under strict time constraints.
