The Scientific Puzzle: Four Major Challenges：

Before looking at the solution, let’s consider why the task is so hard:

1. Complexity — Policy style is multi-dimensional, covering decision-making, implementation, and communication.
2. Subjectivity — Experts often disagree on how to classify the same text, leading to inconsistent annotations.
3. Data scarcity — High-quality expert annotations are expensive and difficult to obtain.
4. High cost — Even large language models (LLMs) struggle to score reliably and consume huge computational resources.

Together, these factors make policy style measurement a formidable challenge.

KOALA: Turning AI into a “Learner”

To tackle these challenges, researchers designed a novel framework with a memorable name: KOALA (KnOwledge distillation framework based on large lAnguage modeL collAboration). Its logic can be captured through a simple analogy:

• LLMs as learners: Instead of asking large language models to assign direct scores—where they often stumble—KOALA reframes the task as pairwise comparisons: Which policy text sounds more forceful? LLMs perform much better in this format.
• Expert guidance: Only a handful of annotated samples (in this study, just four!) are sufficient to align the model with expert judgment.
• Collaborative roles: The framework casts LLMs in three roles—Prompter, Ranker, and Analyst—mimicking an expert panel discussion that iteratively refines answers.
• Knowledge distillation: The insights produced through these interactions are distilled into smaller, more efficient models that can carry out large-scale measurements reliably and at low cost.

In short, KOALA works like a classroom: experts provide demonstrations, large models practice and reason through examples, and compact models internalize the knowledge for practical use.

Putting It to the Test: 70 Years of Government Reports

To validate the approach, the team analyzed 4,572 Chinese government work reports (1954–2019) across central, provincial, and municipal levels. They focused on the dimension of imposition—how top-down, authoritative, and non-negotiable a policy appears.

The results were striking:

• GPT-4 achieved only 66% accuracy in ranking task performance.
• KOALA, even when based on GPT-3.5, reached 85% accuracy

This demonstrates that carefully designed frameworks can outperform brute force reliance on the most powerful models.

Why Does It Matter?

1. Scientific significance: Policy style, long discussed qualitatively, can now be measured quantitatively, making comparative political and public administration research more precise.
2. Governance value: Quantifying policy style across regions and over time provides new tools for evaluating governance patterns and policy effectiveness.
3. Cross-disciplinary innovation: KOALA shows how artificial intelligence can bring rigor and scalability to social science research, bridging “soft” political concepts with computational methods.

Conclusion

The significance of KOALA lies not just in boosting accuracy but in offering social science a new paradigm: combining AI’s computational power with human expertise to quantify concepts once thought unmeasurable.

Just as microscopes allowed biologists to observe cells, KOALA enables social scientists to observe the “texture of governance” embedded in policy texts. In the future, more frameworks like KOALA may emerge, making political, social, and economic processes increasingly observable, comparable, and predictable.

It turns the elusive notion of “policy style” from an abstract theory into something as measurable as temperature.

This article is based on the following paper:  Zhang, Y., Huang, B., Yuan, W., Jiang, Z., Peng, L., Chen, S., & Tan-Soo, J. S. (2025). Expert-level policy style measurement via knowledge distillation with large language model collaboration. Information Processing & Management, 62(4), 104090.