From Manual Coding to AI Assistance: Using LLMs to Classify and Summarize Judicial Evaluation Comments

John M Charles, MS; Brian Robertson, PhD; Cooper Kelley; Xiaolei Pan

AAPOR 2026

At the 2026 AAPOR Annual Conference, MDR presented a proof-of-concept study exploring how large language models (LLMs) can help process and analyze thousands of open-ended judicial evaluation survey comments. Using a custom AI agent developed within Microsoft Copilot, the project evaluated comment attribution, classified constructive versus non-constructive feedback, and generated thematic summaries across key judicial performance domains. The research demonstrated the potential for AI-assisted qualitative analysis to accelerate reporting, improve scalability, and support efficient human review while maintaining methodological transparency. The study was conducted using nearly 9,400 comments collected through Colorado’s Judicial Performance Evaluation program.