A system built on GPT-4 helped Pakistani judges clear backlogs at a return of $38.50 for every dollar spent.
Researchers from ETH Zurich, Imperial College London, and the New Economic School have produced the strongest experimental evidence yet that AI can boost government productivity.
Pakistan has fewer than two judges per 100,000 residents, compared to 22 in the EU and 30 in England and Wales. At the end of 2024, 2.26 million cases were pending, with 82 percent located in trial courts. Judges operate with minimal technology and no support staff. Before the trial, only about 25 percent had ever used a large language model like ChatGPT.
The study involved 1,559 judges across 118 courts, representing roughly half of all Pakistani trial court judges.
The tool used was JudgeGPT. It uses retrieval augmented generation to search a database of 129,235 documents, including 128,292 court rulings and 943 Pakistani laws. When a judge enters a query, the system selects the ten most relevant passages and generates a cited answer.
Trained judges resolve 1,848 more cases per year per district
Researchers divided judges into three groups. One group received JudgeGPT access plus targeted training. This training consisted of six 90-minute lectures over three weeks, taught by ETH Professor Elliott Ash after court hours. Judges learned which tasks suited the tool, where it fell short, and how to check its output.
A second group received the same AI access but only a general seminar on technology and law. The control group attended that seminar with no JudgeGPT access.
AI access alone did little. Judges with targeted training used JudgeGPT four times as much as those in the general seminar group. After 40 weeks, trained judges averaged nearly 60 logins and over 200 prompts. The comparison group averaged about 20 logins and fewer than 50 prompts.
Districts with more trained judges resolved more cases. At moderate exposure levels, that meant about 1,848 extra cases per year per district, a 6.3 percent bump. Even districts in the bottom quartile still cleared about 616 more cases.
Judgment quality ticks up without added bias
Ruling quality held steady or improved. The appeal rate per 1,000 resolved cases fell slightly, and judges worked the same hours with no change in work-life balance. The researchers estimate savings of about $38.50 per dollar invested, based on what it would cost to hire enough extra judges to match the same output. Even conservative estimates put the return at “at least” $10 per dollar.
A review of roughly 4,000 court judgments found more AI-flagged text, as expected. But readability, length, and the number of legal arguments held steady. An LLM-based quality check, validated by two Pakistani lawyers, showed a slight improvement. Rulings from trained judges were rated better in 59 percent of pairwise comparisons, up from 42 percent in the control group. The study found no evidence that AI use increased gender or religious bias in judicial language.
Hands-on training shapes how judges use AI
The researchers reviewed anonymized chat logs from about 1,500 judges. Legal research, text editing, and text generation were the most common tasks. Around 60 percent of queries sought information about laws, procedures, or legal concepts.
Trained judges used JudgeGPT more for editing and summarizing text, tasks where language models are more reliable. They asked fewer broad legal questions, where hallucination risk is higher. The researchers say the training steered judges toward limited support tasks while leaving the final decisions in their hands.
Only about a fifth of requests involved what the researchers call “substantive AI delegation,” where judges asked JudgeGPT to evaluate decisions or draft reasoning on its own. Training made judges more likely to decide cases themselves and only use AI to write up their reasoning.
The researchers stress that their findings do not support replacing judges with AI. What the experiment shows is that AI can boost public sector productivity, but only when paired with training that steers users toward the right tasks. Without it, most of the gains disappear.
The productivity gains in this study are likely a floor, not a ceiling. JudgeGPT ran on GPT-4, a pre-reasoning model that was the best fit for the project at the time. Today’s reasoning models write better, hallucinate less, and handle complex tasks more reliably.
