Pakistani Judges Give Their Verdict on JudgeGPT

Judges around the world have made headlines for illicitly using generative AI in their work. But in Pakistan, a large-scale trial of a specially designed AI tool for judges found the technology—together with appropriate training–boosted the number of cases resolved by 6.3 percent with no obvious drop in the quality of judgments.

With a backlog of 2.26 million cases and fewer than two judges per 100,000 people—compared to 22 in the EU and eight in Brazil—Pakistan’s judiciary was in sore need of help. So, in consultation with the judiciary, economist Sultan Mehmood, of the New Economic School in Moscow, and collaborators tested whether AI could ease the burden.

They built a custom tool combining OpenAI’s GPT-4 large language model (LLM) with a knowledge base of nearly 130,000 Pakistani judicial opinions and statutes, to help judges with legal research and drafting judgments. They began offering the tool in 2024 to 1,559 trial judges—roughly half the country’s justices.

“We do find an increase in cases resolved, and we don’t find any corresponding decrease in decision quality,” Mehmood says.

First of its kind

“It’s pretty amazing that he’s able to pull this off,” says David Autor, an economics professor at MIT. “It’s not easy to do large-scale field experiments in civil service, but especially where the stakes are so high.” The 6.3 percent productivity boost is not overwhelming, he says, but it’s credible and likely to improve as the tool is more widely used.

AI tools for judges are already being rolled out in Brazil and India, and prominent U.S. law professor Eric Posner has compared LLM judgments to human judgments in a single case study. But until now, there has been no major independent assessment of ongoing judicial use of AI. The new study focused on Pakistan’s trial courts; Mehmood says judges there were enthusiastic from the start.

“They were more techno-optimist than we were,” he says. “The delays are so huge, this is something which they thought was worth trying anyway to reduce people’s suffering.”

Some judges were also already using AI chatbots, Mehmood says, but commercial offerings performed poorly on Pakistani legal queries, frequently hallucinating case law. So the team built a tool tailored to the Pakistani context, called JudgeGPT.

They used retrieval-augmented generation (RAG), which allowed the model to query a database of 128,292 Pakistani judicial opinions and 943 statutes. Responses included footnotes linking to cases and laws.

“It turns out that actually the way to fix [hallucinations] isn’t just more intelligent models,” says study coauthor Elliott Ash, an associate professor of law, economics, and data science at ETH Zurich in Switzerland. “It’s to attach the models to a tool that can do a search and verify the sources.” However, the researchers do not report hallucination rates.

The team also put 1,197 judges through six 90-minute Zoom training sessions where Ash covered how LLMs work, their limitations, the risk of bias and hallucinations, and the importance of verifying outputs. Another 180 judges only underwent general training on technology in legal research, while a final group got no training.

By the time 487 judges had been through the program, the median district saw a jump of 6.3 percent resolved cases, and the more trained judges in a district, the bigger the effect. Appeal rates also fell slightly, suggesting faster resolution wasn’t leading to sloppier decisions.

JudgeGPT can be used to surface relevant case law with a simple text query, and results provide links to the full text of the related judgments.Sultan Mehmood, Christoph Goessmann, and Elliott Ash

Addressing limitations

The team also assessed the quality of judgments. Having legal experts evaluate large numbers of judgments was infeasible, Mehmood says, so the team asked OpenAI’s GPT-5-mini to choose between pairs of judgments from the same judge before and after training. The LLM chose post-training judgments 59 percent of the time. Two experienced Pakistani lawyers also evaluated the model’s analysis of 90 judgment pairs. They agreed with GPT-5-mini 70.6 percent of the time, compared to 73 percent agreement with each other.

Training turned out to be vital. On average, JudgeGPT-trained judges logged in 56 times and sent 212 prompts over the study period, compared to 10 logins and 25 prompts after generic training. Those who had no training tended to use the tool for around a month and then drop off entirely, Mehmood says. “Just giving people the technology does not necessarily make them use it persistently,” he says.

A 6.3 percent increase sounds modest, but the researchers calculated that a trained judge was resolving 38.5 more cases a month than the baseline, translating to roughly US $38.50 saved in judicial costs for every dollar spent running the tool. Ash also notes that these figures come from a nine-month period at the start of the trial, and that they’ve since updated both the underlying AI model and the database.

For users, the tool has been a lifeline. One participating trial judge, who spoke on condition of anonymity, says the number of cases assigned to them hasn’t dropped below 1,000 in more than a decade. The tool saves significant time, in particular searching for case law and summarizing lengthy documents. “For research, it’s just one prompt away, whereas before I had to search for the precedents and laws for hours,” the judge says. “If I have to read 10 pages of a precedent, now I ask JudgeGPT to just summarize it for me and give me the crux, and it does that work in seconds.”

But efficiency isn’t the only thing you want out of a justice system, says John Zeleznikow, professor of law and technology at La Trobe University, in Australia. “What they’ve tried to do is be effective, [to] deal with more cases more quickly, and they’re able to do that,” he says. “What’s not that clear is whether what you call the quality of justice is better.”

Zeleznikow says AI can be useful, but only if judges are diligent about evaluating and verifying the output. However, the working paper’s authors found that roughly a fifth of participants’ prompts given to JudgeGPT involved what they call “substantial AI delegation”—asking the tool what the best decision is, to produce legal reasoning or write opinions with little input from the judge. On the bright side, training lowered the proportion of inappropriate delegation.

But given that judges are already using AI, Ash says better tools and training are crucial. “There are risks for using these AIs, for sure, even with all these safeguards. But at some point you have to just put the judges in as strong a position as you can,” he says. “Have technological safeguards, but then try to encourage the judges not to rely on it too much.”