MDR presented a research poster at the 2026 American Association for Public Opinion Research (AAPOR) Annual Conference exploring how large language models (LLMs) can support the analysis of large-scale qualitative survey data. The presentation, From Manual Coding to AI Assistance: Using LLMs to Classify and Summarize Judicial Evaluation Comments, highlighted a proof-of-concept workflow developed within Microsoft Copilot to help process open-ended comments from judicial performance evaluation surveys.

Using a dataset of 9,397 survey comments associated with 387 judges, the MDR team developed an AI-assisted pipeline that performs three key tasks: validating whether comments are attributed to the correct judge, identifying comments that contain meaningful and actionable feedback, and generating thematic summaries of judicial performance across areas such as communication, fairness, legal reasoning, demeanor, and case management.

The research demonstrates the potential for AI to reduce the time and effort required to review thousands of open-ended responses while maintaining transparency and supporting human oversight. Rather than replacing analysts, the approach is designed to help researchers focus their attention on flagged cases and review AI-generated outputs more efficiently.

The study also underscored the importance of prompt design, validation, and ongoing human review when applying AI tools to complex survey research tasks. Findings from the pilot provide valuable insights into both the opportunities and limitations of using LLMs for qualitative data analysis and point toward future enhancements that could improve accuracy and scalability.

The poster was authored by John M. Charles, Brian Robertson, Cooper Kelley, and Xiaolei Pan of MDR and reflects the firm’s continued exploration of innovative approaches to survey research, evaluation, and data analysis.

This work aligns with MDR’s commitment to combining methodological rigor with emerging technologies to help clients gain timely, actionable insights from increasingly complex data sources.