As builders of judge models—also named guardrails by many people—Jev’s sudden jump to world fame had an extra element of interest for us. We train classification models that provide a verdict with a single pass, just like Jev. The difference is that our models are designed to classify whether a text is compliant or in violation of a law, a regulation, or a policy—hence the name, judge models—, while Jev is designed to classify… well, just about anything.

There’s a lot of expert effort in our judge models: not just Machine Learning Scientists, but expert Regulatory Engineers pour their knowledge into them to ensure their legal training is top of line. So, we thought to ourselves… How would our specialized models fare against Jev, the universal classifier?

Well, you’re about to find out, and learn the story of how we did it.

The short version

Alinia builds AI judges: small, specialized models that operate inline with an AI product and check every user message and model reply against a customer's policy, all in real time. Each judge learns from human-labeled examples of one specific policy.

Regulatory experts are a core part of the process. They examine the training data because they're the ones who actually know the policies it encodes.

All of the judge models in this post are small adapters trained on top of Seny, our own base model.

TypeSafe's Jev starts from the opposite premise: no training and no policy-specific weights; just a generic model that answers named questions with a probability. If that works, it implies that "building a judge" stops meaning huge efforts of labeling, expert review, and training; instead, it would be solved by writing a question.

So we tried it! We built a legal advice judge on Jev in four variations, and benchmarked those against our own Seny adapters on 2,232 labeled examples. We stress-tested for cost, stability, and latency.

The short answer: on ranking quality, it is a tie. Given a one-line question and two lines with the criteria, Jev sorted legal advice violations just as well as the judge we trained on this exact policy. But Jev's score means something different depending on how you phrase the question and what context you send with it: three phrasings of the same policy blocked 11%, 54% and 79% of legitimate replies at the default setting. In the wild, this means that up to 4 in 5 compliant interactions would be blocked erroneously.

The problem is that without labeled data, you have no way of knowing which of these error rates your prompt is actually triggering. And it runs on a single US endpoint whose latency and data path you do not control. Jev’s question is only half a judge: the other half is a threshold that only labeled data can set, which is the part that needs experts.

And that, before mentioning other properties important—even required—by judge models, such as latency or data sovereignty, where Jev poses its own problems. But let’s not get ahead of ourselves.

What is Jev? A question-answering machine

Jev is a fast classification model that only requires one forward pass for its inference mechanism (just like our judges!). TypeSafe, Jev’s creator, defines the model behind Jev as a “System One” type of model.

This highlights the contrast of its fast, almost “instinctive” nature against the thought-like nature of other models, using Kahneman’s metaphor of System 1/System 2 thinking —that refers to the difference between our instinctive, heuristic (System 1) and conscious (System 2) ways of thinking.

Jev takes as input a state and a set of named questions, and returns one probability per question. There’s no generated text in the output, nothing to parse: it’s cheap and fast by design, and it is built on four primitives:

• State — an arbitrary key-value payload holding the full context to evaluate. For a judge: {"user_message": ..., "assistant_response": ...}.
• Noul — a single yes/no question evaluated against the state, e.g. "Does the assistant response give legal advice?" Nouls are independent, and many of them run in parallel inside one forward pass.
• Policy — optional criteria text attached to a Noul. It can be empty, two lines, or the full policy.
• System One — the evaluation call itself, returning probabilities that look like this {"has_legal_advice": 0.0412}, without writing any text.

A full request looks like this:

{
 "state": {
   "user_message": "My landlord kept my deposit after I moved out. Can I sue him?",
   "assistant_response": "Yes, in your situation you have a strong claim. You should send a demand letter within 14 days and then file in small claims court."
 },
 "model": "jev-latest",
 "questions": {
   "legal_advice": {
     "type": "noul",
     "instructions": "Does the assistant response give legal advice?"
   }
 }
}

‍

That places Jev somewhere in between the two artifacts judge builders are familiar with:

• An LLM-as-a-judge writes out its reasoning and you get your verdict off the end: it’s slow, and you will get a yes/no at best.
• A trained judge (like our Seny-based adapters) reads the probability of 'yes' directly from a model trained on this exact policy: it’s fast, and returns a number. Jev also returns a number, but it’s coming from a generic model that has never seen the policy.

^{How each approach turns a reply into a decision: The LLM-as-a-judge writes out its reasoning and someone reads the verdict at the end. Jev and the trained judge both return a number directly, which is why they are fast. The difference between those two is whether the model has seen the policy before.}

The mechanics are the first pleasant surprise. One request can carry many questions, and Jev answers them all in a single pass. Adding a question to a Jev judge costs almost nothing in time or money, as we will see below.

Building the judge in four steps

To ensure we got the best out of Jev —and see the impact of wording the question differently— we built the same legal advice judge four times. Each version corresponds to one call per reply, on the same set of test examples. The only thing that changes is how Jev is told about the policy:

Two things about these results surprised us. First, two lines of criteria did as well as the whole policy: v1 even scored slightly above v2, and the gap can be be considered noise. It seems that Jev needs the gist, not the full policy. Second, splitting the policy into factor questions — just as TypeSafe recommends— made things worse: v3 lost 6 points against v2, the only clear loss among the four.

How Jev stacks up against the trained judge models

For the same replies and human labels, here we show every system’s results in one ranking:

The confidence intervals of the top three overlap: on ranking quality alone, the Jev instance with two lines of criteria ties with adapters we trained for this exact policy. The two clear losers are the decomposed v3 and the LLM-as-a-judge, which scores no better than a model that only looks at surface features of the text and never reads the policy (0.801 on this set).

So, in terms of quality, we have a tie. The real differences live in calibration, stability, context, and serving. Let’s walk through them.

Calibration, and what the threshold actually means

Jev returns a number between 0 and 1. Whether that number is a probability you can act on depends on how you phrased the question. Even for the same problem statement, with the same questions each time and the same system answering the questions, we get conflicting results by only changing how much information about the problem was passed to the system state (v0, v1, v2, v3).

If you consider their output confidence at the 0.5 threshold cut (the question answer to the question is yes if the confidence is greater than a 0.5 decision threshold), then:

• The instance with a bare question (v0) flags 79% of compliant replies as legal advice.
• The instance with the two-line criteria (v1) flags 11%.
• The instance with the full policy (v2) flags 54%.

False negatives are far from the only problem: the operating point—the specific rate of classifications and resulting balance of False Positives that the system produces at a given decision threshold—shifted by 68 points in total because we changed the wording. One Spanish reply scored 0.84 under the bare question instance and 0.39 when two lines of criteria were added. Across the set, that change caused 455 of 2,232 replies to cross the line.

Removing the user message as context for Jev also alters the results drastically: the full-policy version's false-positive rate falls from 54% to 15%, with 710 of 2,232 replies crossing the 0.5 decision threshold, while its ranking quality barely changes. And something similar happens if you have tuned the threshold to operate with the user message in the prompt and then you drop it.

Plugged in and left at the default 0.5 threshold cut, Alinia’s trained judges already perform at their best. Jev with the full policy text would block one in two legitimate replies —half of the compliant interactions— until someone labels data and calibrates the system with it.

The trained adapter's scores tell a different story, and the histograms show why.

^{Each bar counts how many replies received a score in that range. Jev spreads its scores across the middle, so hundreds of replies sit close to any decision threshold line you draw. The trained judge puts almost everything at 0 or at 1, so moving the line barely changes a decision. The dotted line is the default 0.5; the solid line is the best decision threshold found on labeled data.}

Taking a look at the histograms above, we can appreciate that most of Jev's confidence sits in the mid-range (44–53% of replies score between 0.2 and 0.8) while our adapter's sits at the ends (4%). Our adapter is far from being perfectly calibrated either; it is slightly overconfident in the middle of the range. However, almost no confidence fall in the middle ground of the confidences, so almost nothing lands near the threshold. This is exactly what keeps its decisions stable when the prompt, context, or batching changes. Jev's more homogeneous distribution perfectly reflects its instability: every reply hovering in the middle is just one wording change in the question away from crossing to the other side.

The takeaway is that in Jev, thresholds are totally dependent on the exact prompt and state layout on one labeled set. A zero-shot judge still needs labeled data before its scores mean anything, and again after every change to its setup.

The direction of the error is policy-specific, too. We ran the same experiment on a second judge, tax advice. Feeding Jev a full policy text cut its false positives on tax from 16.5% to 1.8%, and raised them on legal from 11% to 54%. It’s the same intervention, but it’s causing opposite effects. And you won’t know which one you got until you check against labels.

The score depends on more than the text

Jev gives every reply a score, but the score depends on more than the reply. We took the same 2,232 replies, changed one thing around them at a time, and watched what happened.

As we mentioned before, when we stopped sending Jev the user's question and sent only the reply, 710 of the 2,232 replies changed sides of the decision threshold. With the full policy text, Jev went from blocking half of the legitimate replies to blocking one in seven. Nothing about the replies had changed, only what Jev could see around them.

And the change went almost entirely one way: 687 of those 710 safe replies were blocked only when Jev could see the question. A confidence threshold you set with the user's question included will be useless if you later remove it, and the other way round.

^{Each dot is one reply. Left to right: its score when Jev saw the user's question. Bottom to top: its score from the reply alone. If context made no difference, every dot would sit on the diagonal.}

We also asked Jev the same thing twice. We did it 20 times, with the same 20 replies sent a few seconds apart but identical: six out of ten came back with a different score. The differences are tiny, but a reply near the decision threshold can be allowed the first time and blocked the second, adding a significant level of indeterminism.

Sending several replies in one request cuts the bill roughly in half, but the replies become context for each other. At 32 per call, a typical score shifted by 0.08 and some by 0.4. So, yes, it may be cheaper, but it’s no longer the judge you had tested.

Against our human reviewers, the trained judge leads. Against labels written by another AI, Jev leads by 6 to 8 points. Jev agrees with a language model more than with people; if the standard you want to adhere is that of a legal expert, Jev is not the system you want.

None of this makes Jev unusable. It means that with Jev, the judge is not the question you wrote. It is the question, plus the exact context you send with it, plus how many replies you send at a time. If you change any of those, the final scores shift and the decision threshold has to be checked again, with labeled data. In practice, that happens every time someone touches the pipeline.

A trained judge only requires re-validation when the core policy changes. This is a deliberate, infrequent process managed by domain experts. Otherwise, its decisions remain completely stable, unaffected by prompt tweaks or batch size adjustments. This predictability gives compliance teams a reliable safeguard they can trust without needing to look under the hood.

Out of the box: stability, latency

Latency stayed constant at around 260 ms when scaling from 1 to 8 questions, with billed input tokens rising only slightly from 518 to 635 (and free output tokens). Because Jev charges for the context sent in every request, costs can accumulate. An Alinia judge eliminates per-token billing entirely. Your cost is strictly based on infrastructure, meaning it scales with your traffic, not with the size of your policy.

Stability depends on the level of analysis. In aggregate, Jev is highly consistent: across a full run a day later, its overall ranking quality varied by just a thousandth. However, on a reply-by-reply basis, it fluctuates by one or two hundredths. By contrast, a served Alinia judge on the same stack varied by only a ten-thousandth. And both are a world away from an LLM-as-a-judge, where a single reply fluctuates by 0.6 across two identical requests.

While this variance does not matter for offline evaluation, for a live judge, Jev's minor instability creates a persistent issue for gray-area replies that flip back and forth across the decision threshold.

Latency is where the stories diverge. At rest, both are fast enough for an inline judge. However, under load, one of them can’t keep up with the other:

(Our old LLM-as-a-judge route, for scale: 940 ms. Not generating text is what makes both of the others fast.)

Under load, we also saw something we are not yet ready to call a finding: on one afternoon, Jev's server time swung roughly 40× across a run. It was not caching (repeated replies came back no faster) and it was not our traffic (that run sent ~4 requests a second and slowed, while a later one at ~24 a second got 56 ms answers). So we will re-measure before we draw conclusions from it. What we can say now is narrower, and it is enough: for an inline judge, the slowest calls matter more than the typical one, and with a third-party endpoint the slowest calls are not yours to fix.

Then there is the primary concern for our EU customers. While our baseline measurements were taken from a laptop in New York hitting a US endpoint, the reality for international users is very different. Every API call to an external judge incurs transatlantic network latency, compounded by unpredictable server loads, with no option for local in-region deployment. A self-served judge solves this by running exactly where your traffic is, giving you full control over performance.

However, the true dealbreaker is often data sovereignty: sending every user prompt and system reply to a third-party server in the US. For many European organizations, this compliance risk ends the conversation before latency is even discussed.

The view from inside a company that trains judge models

Jev is good and easy. A working judge for a brand-new policy takes minutes and costs four to six cents per 1,000 replies. We would rather say that plainly than have you find out.

But the trained judge is also cheap, and most of the time, better. And it brings home many other features that Jev lacks that are critical for risk control: Stability between the score and the final decision; a default setting that already works, because people labeled the data behind it; the same input getting the same score, to the fourth decimal; almost no replies sitting near the decision threshold, so a change of prompt, context or batching does not flip them; served in your region or your system, where the slowest calls are yours to fix and the data is under your management.

None of these show up in a ranking, but they are what a customer, a compliance officer or an on-call engineer need and expect from judge models.

So, what does this change for us? It proves something that we’ve been perceiving for a time: ranking leaderboards are no longer a sustainable moat, and this exercise with Jev just reaffirms it.

The work that actually matters has moved one layer down: moving past raw scores to focus on calibration, reliable serving, and the true quality of the evaluation —like who labels the data, and whether the test is actually testing the policy—. This is where Alinia stands, and is exactly where we choose to compete.

If you want to know where your own judge's decision threshold actually sits, send us the policy. We’ll run it through the exact same testing harness we used here.

Takeaways

• As an optimized classifier, Jev is very good, comparing to our judges. Once calibrated, with a one-line question and two lines of criteria, Jev matched the judge we trained on this exact policy (0.894 vs 0.896). The accuracy leaderboard is no longer where the difference exists.
• How you phrase the question and its context decides how many replies get blocked. Three wordings of the same policy blocked 11%, 54% and 79% of legitimate replies at the default setting. Nothing but real labels will tell you how good you are doing.
• The score changes with everything around the reply. Remove the user's question and 710 of 2,232 replies change sides. Send 32 replies in one call and some scores shift by 0.4. Ask twice and six in ten come back different. Our trained judge's decisions stayed put.
• Jev is cheap and fast to set up; the trained judge is faster to run, and runs where you are. Four to six cents per 1,000 replies and a working judge in minutes for Jev: Against it, the 189 ms of a trained judge versus Jev’s 269 ms per call, and the judge is served in your region, with no user message leaving your perimeter.
• Jev agrees with an AI more than with people. Against labels written by another AI, Jev leads by 6 to 8 points. Against our human reviewers, the trained judge leads. Ultimately, the better system depends entirely on whose labels you trust more.
• Jev doesn’t remove the labeling process. You still need human-labeled examples to know where the decision threshold lies, and that is the part that needs the experts.