chevron_left
Back to all engineering posts
Decision Model Bakeoff: Jev vs Clef vs OpenAI Decisions
We ran TypeSafe's Jev, OpenAI's Decisions API and Cloudflare's Clef on 9,000 real Traba decisions, then on forged documents, tampered IDs, face matching and video. Jev wins on language at the lowest cost; none of them can do math.
Most AI models write, one word at a time. But much of what software asks AI is multiple choice: which team should handle this text, is this message an attack, was this shift cancelled late?
This fall, a new kind of model arrived for exactly that. TypeSafe's Jev, OpenAI's Decisions API and Cloudflare's Clef (with a smaller, faster Clef-flash) read the input once and return a probability for every allowed answer. No writing means they're fast and cheap: a fraction of a second, and a few cents per thousand decisions.
Traba manages labor and intelligence for the industrial supply chain. Core to our business is the staffing marketplace: businesses post warehouse and light-industrial shifts, and workers pick them up in our app. Running it takes thousands of small judgments a day, each a fixed-choice question where speed and cost matter. We already run Jev on some of these.
We put all 3 to the test through the lens of real world use cases at Traba. Gemini 3.5 Flash ran alongside as a baseline.
How an LLM answers: one pass per token
How a decision model answers: one pass, a probability per answer
A real example
Question: What is the customer asking for? Pick one of 17.
Jev's answer
Methodology
| Test | Traba use case | Data | Models |
|---|---|---|---|
| Text decisions | Routing worker texts, support tickets and customer emails; screening messages to Neo | 31 internal evals, 300 cases each (simple, medium, hard) | All 4 + Gemini |
| Forged handwriting | Checking uploaded timesheets, contracts and certificates | 500 public notebook scans | OpenAI, Clef, Clef-flash, Gemini |
| Tampered ID cards | Checking worker IDs at onboarding | FantasyID: 1,385 cards | Same 4 |
| Face matching | Matching a worker to the ID on file | 350 card pairs | Same 4 |
| Exercise form | Proving lifting or forklift skills from video | 2 bench-press videos | Clef, OpenAI, Gemini |
Every test stands in for a Traba business use case. Same input for every model; answer keys checked blind.
Results
Text
Why we tried this: Jev already makes these decisions in production. We wanted to know if a cheaper, faster or more accurate model should replace it.
We start with the three things that decide whether a model is usable at all: how often it's right, how fast it answers and what it costs.
| # | Model | Accuracy | Per second | Per dollar |
|---|---|---|---|---|
| 1 | Gemini 3.5 Flash (baseline) | 95.7% | 1.4 | 1,548 |
| 2 | Jev 1.13 | 92.5% | 4.1 | 42,589 |
| 3 | OpenAI Decisions | 92.1% | 8.0 | 31,586 |
| 4 | Clef | 91.3% | 2.6 | 10,575 |
| 5 | Clef-flash | 86.1% | 2.9 | 28,199 |
Jev and OpenAI tie on average, and Gemini leads at a much higher price. But a model that handles easy questions can still fail on real ones, so every task has 100 cases at each of three levels:
The message states the answer outright. Most production traffic looks like this.
The person gives one answer, then takes it back. The model has to use the final one.
Several facts have to be combined, and one of them points the wrong way.
| # | Model | Accuracy | Per second | Per dollar |
|---|---|---|---|---|
| 1 | Gemini 3.5 Flash (baseline) | 98.0% | 1.4 | 1,548 |
| 2 | Jev 1.13 | 96.2% | 4.1 | 42,589 |
| 3 | OpenAI Decisions | 95.8% | 8.0 | 31,586 |
| 4 | Clef | 94.9% | 2.6 | 10,575 |
| 5 | Clef-flash | 90.8% | 2.9 | 28,199 |
| # | Model | Accuracy | Per second | Per dollar |
|---|---|---|---|---|
| 1 | Gemini 3.5 Flash (baseline) | 95.4% | 1.4 | 1,548 |
| 2 | OpenAI Decisions | 91.0% | 8.0 | 31,586 |
| 3 | Clef | 90.5% | 2.6 | 10,575 |
| 4 | Jev 1.13 | 90.1% | 4.1 | 42,589 |
| 5 | Clef-flash | 83.6% | 2.9 | 28,199 |
| # | Model | Accuracy | Per second | Per dollar |
|---|---|---|---|---|
| 1 | Gemini 3.5 Flash (baseline) | 93.7% | 1.4 | 1,548 |
| 2 | Jev 1.13 | 91.3% | 4.1 | 42,589 |
| 3 | OpenAI Decisions | 89.5% | 8.0 | 31,586 |
| 4 | Clef | 88.3% | 2.6 | 10,575 |
| 5 | Clef-flash | 83.8% | 2.9 | 28,199 |
Every model drops when people change their mind or facts pile up, yet the order barely moves. To see whether that holds for each kind of decision, here is every task.
Accuracy by task (%), sorted by Jev; bold = best on that task
No single model wins every task, and Gemini's lead costs 28× more. That suggested a cheaper idea: let Jev answer, and call Gemini only when Jev is unsure. Jev first, Gemini when Jev is under 80% confident beats every single model.
| Setup | Accuracy | Cost per 1,000 |
|---|---|---|
| Jev alone | 92.5% | $0.023 |
| Jev, then Gemini | 95.9% | $0.139 |
| Gemini alone | 95.7% | $0.646 |
Plotting accuracy against cost shows how good a deal that is. Every cut-off we tried sits above every single model.
Accuracy against cost: the hybrid beats every single model
Knowing which model to use is half the answer; knowing where they all struggle is the other half. First, by area of the business:
Accuracy by business area
Each dot is one model; the grey line spans the lowest to the highest. Rows sorted by Jev.
Shipments and claims are hardest everywhere. Cutting the same cases by situation shows the common cause: arithmetic.
Accuracy by situation
Difficulty levels are in the tabs above, so only the other situations are shown here.
Language, slang and speech-to-text errors are handled well; numbers are not.
Every model makes mistakes, so the next question is whether it knows when it might be wrong. If its confidence is honest, software can act on the sure cases and escalate the rest. Confidence is usable: at 90%+, Jev is right 98.8% of the time on 71% of cases. Gemini's self-rated confidence isn't.
Accuracy and confidence settle which model to trust. Speed decides where it can run, especially anywhere a person is waiting on the answer.
Speed, all 9,026 cases
Numbers only go so far. These six real cases show the three ways models go wrong: over-alarming, missed corrections and arithmetic.
Six hard scenarios: probability each model gave the correct answer (above the dashed line = right)
Keep Jev, and put Gemini behind it for the cases Jev is unsure about.
Deep dive: Neo prompt-injection detection
Why we tried this: this check is live. A wrong answer either lets an attack through or flags a real customer.
Jev screens every message sent to Neo, and flags anything at 0.7 or above. Every model catches real attacks. Nearly all errors are attacks the user took back, where the rule itself is unclear.
Is the user trying to override or extract the assistant's instructions?
Correct: Yes
Every model flags it.
Is the user trying to override or extract the assistant's instructions?
Correct: Yes
Jev misses it; the other four catch it.
Is the user trying to override or extract the assistant's instructions?
Correct: No (per our rule)
The models split, because the rule doesn't say whether this counts.
Is the user trying to override or extract the assistant's instructions?
Correct: No
Only Clef-flash crosses the flag line.
Fix the rule, not the model. Decide whether pasted and taken-back attacks count, then reword the question.
Forged handwriting
Why we tried this: workers and customers upload documents such as timesheets, contracts and certificates. Can a model spot an edit?
Alone, a page gives nothing to compare; beside the original, the edit stands out.
One example is suggestive, not proof. Across all 500 pages, we measured how well each model separates edited pages from clean ones, alone and beside the original.
Only useful when we hold an original to compare against.
Tampered ID cards
Why we tried this: onboarding relies on ID documents. Could a model flag a tampered one on its own?
The example shows Clef's scores rise on tampered cards but also run high on genuine ones. Across all 1,385 cards, we checked which kinds of tampering each model actually catches.
Clef is a cheap first filter for edited text; nothing catches face swaps.
Face matching
Why we tried this: checking that the person on an ID matches the one we already have on file.
Judging one card alone failed, so we tried comparing two. Across 350 pairs, each dot shows how sure a model was that both cards show the same person.
Face matching: each dot is one ID card to face pair
Works on ID-to-face pairs, with a caveat: the face comes from the same card scan, either photographed on a second phone or swapped in, so the model only has to spot the swap. A real check compares an ID with a selfie taken under different lighting and angles, which is harder. That test, and legal sign-off, come next.
Exercise form from video
Why we tried this: proving a worker's skill, such as safe lifting or driving a forklift, from a short video.
Video A: phone, gym floor
Video B: filmed, powerlifter
Is it a bench press?
Someone spotting?
Bar reaches chest?
Hips stay on bench?
Feet planted?
Elbows tucked?
Wrists straight?
Bar under control?
Models confirm what someone is doing, not how well.
Where else this fits
There are many more use cases than the ones we tested: anything that repeatedly picks from a fixed set of options. The examples below are illustrative.
State machines
Run a conversation in turns. This one collects a worker’s address: after every reply, Jev picks the next state from a fixed list (gave an address, confirmed, corrected, unclear), and low confidence hands the thread to a person.
Quick context compaction
On every turn of a long agent conversation, decide line by line what to keep and what to drop before sending the context to the model. Cheap enough to run every turn.
Sentiment on every message
Score each worker message for frustration as it arrives and count how many land in each band, from calm to angry. When the red end fills up, page a person before the worker gives up.
An agent's next step
“Clock Maria out at 3:30, she left early.”
Model routing
Send each request to the cheapest model that can handle it. Jev reads the request and picks large, medium or small; most traffic can go to the small one.
Conclusion: where Jev comes out ahead
Decision models are good at quick, fixed-choice judgments about language, and at comparing two things. They're not good at math, or at judging a single image on its own.
| Use them for | Don't use them for |
|---|---|
| Sorting worker texts, support tickets and customer emails | Calculating times, shift codes or pay (do that in code) |
| Screening messages for prompt attacks | Deciding whether a single document or ID is fake |
| Checking a document against an original we hold | Grading how well someone does a task on video |
| Checking whether two ID photos show the same person |
What we learned: these models are a good fit for the small, repeated judgments that run Traba every day. Give one a message and a fixed list of answers and it picks the right one about as often as a much bigger model, in a fraction of a second, for a few cents per thousand. Jev comes out ahead where the volume is high and the question is about language: it ties OpenAI on accuracy at the lowest cost, and with Gemini behind it for the cases it is unsure about, it beats every single model. The natural places to use it are routing worker texts and support tickets, screening messages to agents, choosing an agent’s next step, and comparing a document or ID photo against one we already hold. Where the answer depends on arithmetic, such as whether a cancellation came 24 hours ahead, write code instead. Judging a single image on its own, like whether an ID is fake, is beyond all of them today.
