For years, sentiment analysis dominated automated text evaluation—classifying text as positive, negative, or neutral. But what if you needed to measure something more nuanced? What if you wanted to quantify dehumanization in a text, or psychological construct strength on a scale of 1–100, or consumer purchase intent across five Likert points? The answer lies is Semantic Evaluation, enter LLM-as-a-judge evaluation frameworks, a paradigm shift that lets language models assess text against custom rubrics, producing numerical scores instead of binary classifications. Two Python libraries are emerging as frontrunners for this exact use case: judges (a lightweight grader framework) and semantic-similarity-rating (a probabilistic approach grounded in psychometric theory). Here’s why they matter for your research pipeline.
This article is intended to extend our article From Spam to Impact: A Practical Workflow for Writing Psychology Papers Based on Secondary Data.
What is LLM-as-a-Judge Evaluation?
LLM-as-a-judge (LaaJ) is a Semantic Evaluation method where one language model—the judge—evaluates text (e.g., interview transcripts, research notes, generated summaries) against researcher-defined criteria. Unlike traditional sentiment analysis or bag-of-words metrics, LaaJ judges can:
- Evaluate context and nuance rather than just word frequency
- Apply domain-specific rubrics (e.g., “rate the coherence of this argument 1–5”)
- Return numerical or ordinal scores suitable for statistical analysis
- Scale to thousands of texts at a fraction of human coding cost
- Provide reasoning alongside scores, enabling qualitative validation
This is exactly what researchers have long needed: a machine-readable way to measure psychological constructs, thematic strength, or argument quality directly from text. The catch? The LLM’s output must be carefully constrained to return machine-parsable scores, and the rubric anchors must be detailed enough to be reliable.
Tool 1: judges – Fast, Flexible, LLM-Agnostic Grading
The judges library (available on PyPI) is the simplest entry point to LLM-as-a-judge. It provides two core judge types: classifiers (boolean yes/no judgments) and graders (numerical or Likert-scale scores). Graders are what you want for semantic analytic scales.
Why judges?
- Works with any LLM API (OpenAI, Anthropic, local Ollama, etc.)
- Minimal setup—just define the rubric in a prompt string
- Returns parsed numerical scores ready for analysis
- Built-in voting/jury support for multi-judge consensus
Example: Rating Perceived Familiarity in Text
Imagine you’ve collected interview transcripts about social belonging and want to rate “perceived familiar size” (how many people the respondent feels truly aligned with) on a 0–100 scale. Here’s how to use judges for Semantic Evaluation in this situation:
from openai import OpenAI
from judges.graders import CustomGrader
# Initialize your LLM client
client = OpenAI(api_key="your_key_here")
# Define your grader with a custom prompt
grader = CustomGrader(
model="gpt-4o-mini",
criteria="""
Rate the perceived familiar size expressed in this text on a scale of 0-100.
Anchors:
0 = respondent explicitly states they feel completely isolated, no one understands them
25 = respondent mentions a few acquaintances but feels misunderstood by most
50 = respondent feels aligned with some people but notes gaps in deep connection
75 = respondent describes a moderate circle of "true people" who understand them
100 = respondent expresses feeling deeply connected to a large, coherent group
Return ONLY a single integer between 0 and 100. Do not include explanation.
""",
extract_score=lambda x: int(x.strip()) # Parse the integer from response
)
# Example interview excerpt
interview_text = """
I'd say there are maybe 5 or 6 people in my life who really 'get' me—
people whose values and approach to the world align with mine.
It's not a huge circle, but it's solid. Beyond that, I know lots of people,
but the connection is more surface-level.
"""
# Score the text
result = grader.judge(input=interview_text)
print(f"Score: {result.score}")
print(f"Reasoning: {result.reasoning}")This returns a single integer (e.g., 58) and the model’s reasoning, which you can then aggregate across hundreds of transcripts and feed into statistical analysis or machine learning models.
Caveat: The judges library on PyPI shows GPT-4o in its README, suggesting the package has not been updated recently. It still works, but you may want to verify it supports your preferred model before production use.
Tool 2: semantic-similarity-rating – Psychometrically Grounded Likert Mapping
If you want a more sophisticated Semantic Evaluation library—one that preserves distributional uncertainty and is grounded in measurement theory—the semantic-similarity-rating (SSR) package from PyMC Labs is a game-changer. Instead of forcing the LLM to pick a single number, SSR asks the model to generate free-text opinions, then maps those embeddings to a probability distribution over Likert points using semantic similarity to carefully chosen reference anchor statements.
Why SSR?
- Preserves uncertainty—you get a full probability mass function (PMF) over 1–5 (or custom scale), not just a point estimate
- Eliminates prompt bias from forcing direct numerical answers (LLMs tend to cluster at the middle of Likert scales)
- Achieves ~90% human test–retest reliability in published validation studies
- Suitable for downstream IRT (Item Response Theory) or Bayesian analysis
- Theoretically grounded in embedding similarity and psychometric best practices
Example: Mapping Perceived Social Alignment to Likert Scale
You want to convert open-ended responses about social alignment into a 5-point Likert distribution. SSR lets you define anchor statements for each scale point, then score new responses against them:
import polars as pl
from semantic_similarity_rating import ResponseRater
# Step 1: Define your anchor statements for each Likert point
# These should be carefully worded to represent the construct at each scale level
anchor_statements = [
"I feel completely isolated and misunderstood by everyone", # 1 = Strongly Disagree
"I have a few people I connect with, but feel mostly alone", # 2 = Disagree
"I have some people who understand me, with some gaps", # 3 = Neutral
"I have a good circle of people who align with my values", # 4 = Agree
"I feel deeply connected to a large group that truly gets me", # 5 = Strongly Agree
]
# Step 2: Create reference dataframe
reference_df = pl.DataFrame({
"id": ["alignment"] * 5,
"int_response": [1, 2, 3, 4, 5],
"sentence": anchor_statements,
})
# Step 3: Initialize the rater
rater = ResponseRater(reference_df)
# Step 4: Score your interview responses
llm_responses = [
"There are maybe 5-6 people in my life who really understand me.",
"I feel pretty isolated—most people don't get where I'm coming from.",
"I'm somewhere in the middle—some alignment, but lots of gaps.",
]
# Step 5: Get probability mass functions (PMFs) for each response
pmfs = rater.get_response_pmfs(
reference_set_id="alignment",
llm_responses=llm_responses,
temperature=1.0, # Controls softness of the PMF
epsilon=0.0, # Regularization to prevent division by zero
)
# pmfs is a DataFrame where each row is a response and each column is a Likert point
print(pmfs)
# Output: probability distribution across [1, 2, 3, 4, 5] for each responseNow instead of forced single scores, you have full distributions. You can compute expected values (1*p1 + 2*p2 + ... + 5*p5), confidence intervals, or feed the PMFs directly into Bayesian hierarchical models. This is particularly powerful if you’re analysing heterogeneous populations or want to model disagreement explicitly.
Which Tool Should You Use?
Choose judges if you need:
- Speed and simplicity (a few lines of code)
- A single numerical score per text (0–100, 1–10, etc.)
- Flexibility to swap LLM providers
- Direct integration with downstream ML pipelines
Choose semantic-similarity-rating if you need:
- Probabilistic outputs that capture uncertainty
- Validated Likert-scale behaviour matching human distributions
- Theoretical grounding in measurement theory (for publication)
- Input to IRT, Bayesian, or psychometric analysis
- Resilience to prompt-wording artifacts
Application: Automating the Coding Loop in Scientific Papers
Here’s where this becomes invaluable for scientific writing. Traditional qualitative research requires hand-coding hundreds of interview transcripts or survey responses against a codebook—a process that is time-consuming, subjective, and prone to coder drift. Semantic Evaluation LLM-as-a-judge libraries shortcut this bottleneck and allow you to apply tools like ydata-profiling and scikit-learn to analyse trends and correlations:
- Rapid hypothesis testing: Score 500 interview excerpts against a theory-driven rubric in seconds, then test whether the distribution of scores predicts your outcome variable.
- Triangulation: Run the same texts through multiple independent judges (or jury voting) to estimate inter-rater reliability programmatically.
- Transparent methodology: Your rubric is a documented prompt; your code is auditable. This satisfies peer reviewers’ demands for reproducibility.
- Iterative refinement: If your scale scores don’t predict outcomes, tweak your rubric anchors and re-score in minutes—not weeks.
- Data pipeline integration: Chain scoring directly into polars/pandas DataFrames, then feed into statistical models using scikit-learn or statsmodels, all in one script.
This is especially powerful when combined with exploratory data profiling (e.g., using ydata-profiling to check distributions) and IRT analysis to validate that your LLM-scored items behave like classical test items. The result: a fully reproducible, audit-trail-capable coding pipeline that would have taken months by hand.
Getting Started
Install both libraries:
pip install judges
pip install git+https://github.com/pymc-labs/semantic-similarity-rating.gitStart small: score a few texts manually, inspect the results and reasoning, then scale up. Pay close attention to your rubric anchors—they are the linchpin of reliability. Vague anchors (e.g., “rate how much the person felt connected”) will produce unreliable scores; specific anchors (e.g., “1 = respondent explicitly states isolation; 5 = respondent describes a cohesive group of 10+ people who share their values”) will anchor the model’s scoring behaviour.
Once you’ve validated the scale on a small sample and checked inter-rater agreement, you can confidently scale to your full dataset and integrate the scored data into your statistical analysis—turning months of manual coding into an auditable, reproducible pipeline that your reviewers will appreciate.