Jev Rips 975 I/O Tag Expressions in 0.22 seconds

Benchmarking Jev

There’s a new AI in town and his name is Jev. He’s sleek, fast, and ridiculously cheap to use. There’s a catch though. Jev doesn’t have chat or agents like other AI models do. Jev doesn’t create images or proof read your writing. I can see you furrowing your brow and wondering why anyone would care. The answer is simple: Jev excels at making structured decisions quickly and accurately, which is exactly what many industrial and enterprise applications need.

What is Jev

Jev, built by Typesafe AI, is a System One Model. Which is a new class of frontier models built to make fast, structured decisions that software can use directly. The term is inspired by Daniel Kahneman’s “System 1” thinking: quick, intuitive judgments rather than slow, deliberate reasoning.

How it Works

Instead of prompting a model and parsing generated text, your app sends:

State/context - such as a support message, form data, logs, or JSON.
Typed questions - such as a fixed set of categories, a score scale, or a yes/no claim.

It returns machine-consumable decisions, typically with a probability or confidence value. TypeSafe AI’s first publicly described System One model, Jev, supports:

Choice: select one of your predefined options
Score: assign a value on an ordered scale
Noul: return the probability that a statement is true

If it isn’t making sense yet, think of it this way: instead of generating text, Jev gives you structured answers that your software can act on immediately. Imagine you have incoming support tickets and you want to automatically categorize and prioritize them. You might provide something like this:

const state = {
  message: "I was billed twice after upgrading my plan.",
  customerPlan: "Pro"
};

const questions = {
  team: {
    type: "choice",
    instructions: "Which team should handle this ticket?",
    criteria: {
      "billing": "Charges, invoices, refunds",
      "technical-support": "Bugs, outages, errors",
      "account": "Login, profile, permissions"
    }
  },
  duplicateCharge: {
    type: "noul",
    instructions: "The customer reports a duplicate charge."
  },
  urgency: {
    type: "score",
    instructions: "How urgent is this ticket?",
    criteria: [
      "Low: no time pressure",
      "Normal: wants it fixed soon",
      "High: blocked or losing money right now"
    ]
  }
};

Rather than generating a paragraph, the model would respond with structured answers like this:

{
  "team": {
    "choice": "billing",
    "confidence": 1,
    "probabilities": { "billing": 1, "technical-support": 0, "account": 0 }
  },
  "duplicateCharge": {
    "noul": 0.97
  },
  "urgency": {
    "score": 1.46,
    "confidence": 0.29,
    "probabilities": { "0": 0, "1": 0.53, "2": 0.47 }
  }
}

The urgency answer is the interesting one. A score of 1.46 with 0.29 confidence says the model is split between “normal” and “high”, and it tells you so instead of picking one and sounding sure.

You would use this response to route the ticket directly, while using the probabilities and confidence to escalate ambiguous cases to an LLM or support team.

Real-world test

I’ve seen some crazy examples of how people are using it, but I wanted to test Jev against something real in my world. A few years ago, I built an expression engine in C# that allows me to determine process states dynamically based on combinations of multiple tag values.

For example, if I wanted to determine when a CIP goes into Caustic Circulation and I don’t have a single tag that tells me directly, I would need to evaluate a combination of multiple tag values. Is the supply valve open? Is the return valve open? Is the return temperature above a certain temperature? All of these conditions together indicate Caustic Circulation. But I don’t want to hard code that in my app, I want a flexible way to evaluate these expressions dynamically based on live tag values. I want to store these expressions and process them any time the tag values change.

Here is a basic example of how the expression engine works:

// Create an instance of the ExpressionParser
IExpressionParser parser = new ExpressionParser()

// Create a dictionary of "tags" and values that can be evaluated.
// For this example, we just have one tag/value pair.
var tags = new Dictionary<string, object> { ["BoolTag"] = true };

// Define an expression
const string expression = "BoolTag == true";

// Create an expression definition based on the expression and tags
IExpressionDefinition definition = BuildExpressionDefinition(expression, tags);

// Tell the parser this is for a boolean and process the definition
IExpressionResult result = parser.ProcessExpressionDefinition<bool>(conditionDefinition);

// To get the value of the expression, you would access the `Result` property of the `IExpressionResult` object:
bool value = ((IBooleanExpressionResult)result).ExpressionValue;

Console.WriteLine($"Expression value: {value}");

With that, I can store pre-defined expressions in a database and evaluate them based on live tag values, such as:

(@Caustic_Supply_Valve == true && @Caustic_Return_Valve == true && @Return_Temp > 150)

When any tag in an expression changes, the engine swaps in the live values and evaluates the expression to true or false.

Ask Jev

Taking my example above, I ran it through TypeSafe’s API, and without writing a single line of code to evaluate the expression myself, Jev did it for me…and did it really fast.

TypeSafe ships a small SDK, @typesafe-ai/sdk. You pass one state and a map of named questions to client.systemOne. Each question is a noul, a choice, or a score.

TypeSafe
Vercel

import { noul, TypeSafeClient } from '@typesafe-ai/sdk';

const client = new TypeSafeClient(); // reads TYPESAFE_API_KEY
const { answers } = await client.systemOne({
  state: { Caustic_Supply_Valve: true, Caustic_Return_Valve: true, Return_Temp: 180 },
  questions: {
    running: noul('Evaluate: Caustic_Supply_Valve == true && Caustic_Return_Valve == true && Return_Temp > 150'),
  },
});

console.log(answers.running); // { type: 'noul', noul: 0.98 }

import { experimental_evaluate as evaluate } from 'ai';

const result = await evaluate({
  model: 'typesafe-ai/jev',
  state: '{"PUMP_1": true, "TT_RECIRC_TEMP": 180, "VALVE_1": false}',
  questions: {
    running: {
      type: 'boolean',
      instructions: 'Evaluate: Caustic_Supply_Valve == true && Caustic_Return_Valve == true && Return_Temp > 150',
    },
  },
  providerOptions: {
    gateway: { zeroDataRetention: true },
  },
});

console.log(result.answers.running);

NodeJS 24 runs the .mts file directly, so there is no build step, and the only dependency in the repo is @typesafe-ai/sdk. The whole harness, the dataset and every number in this post are in jakeashcraft/jev-expression-benchmark on GitHub.

Building a Fair Test

To test this out, I created a generic list of 975 expressions across 309 dummy tags. The grammar is small and includes tags, True/False, numbers, comparisons, &&, ||, ! and parentheses. This runs against local files, so there is no PLC to read. Three things had to be right before any timing meant anything.

Simulated tag values that make a real test. Random values are a trap. A three-way && of random Booleans is true one time in eight, and Step == 72 with a random step is basically never true. Nearly everything comes out false, and a model that always answers “false” would look 90% accurate. So the script gives Boolean tags true or false, gives numeric tags the constants they are compared against plus the neighbors on each side for Temp > 150 that is 149, 150 or 151. The tag values are set in tags.json.

Ground truth for free. The syntax is a subset of JavaScript so each expression compiles with new Function, behind a token whitelist, with every identifier turned into a function parameter so nothing outside the snapshot is reachable.

const TOKENS = /\s+|&&|\|\||[=!]=|[<>]=?|[()!]|-?\d+(?:\.\d+)?|[A-Za-z_]\w*/g;
const BOOLEAN = /\b(?:true|false)\b/gi;
const tagsOf = (expr: string) => [...new Set(expr.replace(BOOLEAN, '').match(/[A-Za-z_]\w*/g))];
const compile = (expr: string, tags: string[]) =>
  Function(...tags, `return !!(${expr.replace(BOOLEAN, literal => literal.toLowerCase())})`) as Row['truth'];

A clean stopwatch. The timer covers the network pass and nothing else. Payloads are built before the clock starts. I had wondered whether to pre-build them into a JSON file, so I measured it and building all 975 takes about 1 ms, so there was nothing to gain. Before each run, one throwaway request per worker opens a keep-alive socket, which keeps TLS handshakes out of the numbers. That warm-up uses a payload the timed pass never sends, so no cache anywhere can influence the result.

graph LR
    CSV["975 expressions"] --> Bench["bench.mts"]
    Tags["tags.json (simulated PLC)"] --> Bench
    Bench -->|"requests"| Jev["TypeSafe API (Jev)"]
    Jev -->|"975 probabilities"| Bench
    Bench --> Truth["Compare with local JS evaluation"]

Attempt One: One Expression per Request

This mirrors how my engine works: one trigger, one evaluation. Any time a tag value changes, any expression using that tag must be re-evaluated. Each request carries one question and only the tags that expression needs. A single round trip takes about 105 ms from my dev machine, so one at a time would take about 100 seconds.

The obvious fix is a worker pool:

async function sendAll(payloads: Payload[], concurrency: number): Promise<Outcome[]> {
  const outcomes: Outcome[] = new Array(payloads.length);
  let next = 0;
  await Promise.all(Array.from({ length: Math.min(concurrency, payloads.length) }, async () => {
    while (next < payloads.length) {
      const index = next++;
      outcomes[index] = await send(payloads[index]);
    }
  }));
  return outcomes;
}

Requests in flight	Wall time	Latency (p50)
16	7.6 s	112 ms
32	4.4 s	126 ms
64	2.3 s	131 ms
128	1.3 s	138 ms
256	0.90 s	168 ms
512	0.79 s	224 ms
975	0.80 s	279 ms

That sweep ran with retries off, so a refusal would have shown up as a failure. There were none, even with all 975 requests in the air at once. Doubling the workers halves the wall time up to about 128 in flight. Past that, each request gets slower as fast as the pool gets wider, and throughput flattens out around 1,200 expressions per second. Brute-force concurrency tops out around 0.8 seconds for a complete run.

The Lever: More Questions per Request

Look at the API again. questions is a map. The model docs say Jev ingests the state once and evaluates every question against it in parallel. So I packed more expressions into each request and watched the per-request latency, with 4 requests in flight so they don’t crowd each other:

Questions in one request	Latency (p50)
1	105 ms
10	108 ms
25	126 ms
50	132 ms
100	189 ms
250	237 ms

Fifty questions cost about 27 ms more than one. That is the whole trick.

The script chunks the CSV in order, which keeps related equipment together. Each request’s state is only the tags its chunk uses, so 50 expressions share about 18 tags:

function buildRequests(batch: number): Request[] {
  const requests: Request[] = [];
  for (let start = 0; start < selected.length; start += batch) {
    const chunk = selected.slice(start, start + batch);
    requests.push({
      rows: chunk,
      state: Object.fromEntries(chunk.flatMap(row => row.tags).map(tag => [tag, snapshot[tag]])),
      questions: Object.fromEntries(chunk.map(row => [row.id, question(row.expr)])),
    });
  }
  return requests;
}

The response mirrors the request: one probability per question ID. At 50 per request, all 975 expressions fit in 20 requests, and all 20 go out at once.

The Result

Here is what the two approaches look like side by side. Each line is one answer as it landed: the row ID, P(yes), a check against the local evaluator, the expression, and the tag values it was judged against. The first run sends one expression per request with 128 in flight; the second sends 50 per request, all 20 requests at once.

Five back-to-back passes with 50 expressions per request:

Run	Wall time	Expressions/s	Accuracy
1	0.229 s	4,267	99.28%
2	0.195 s	5,004	99.28%
3	0.177 s	5,518	99.38%
4	0.183 s	5,339	99.49%
5	0.241 s	4,041	99.28%

I ran that set of five three times over an hour and a half, the last one right after pushing 8,000 single-expression requests through the API. Across all fifteen passes the median is 0.22 s, the range is 0.16 to 0.29 s, and not one request needed a retry. I quote the median because the best run only happens when everything lines up, and a number you can’t reproduce isn’t worth publishing.

Approach	All 975 expressions
One at a time	About 100 s (estimated)
One per request, 16 in flight	7.6 s
One per request, all 975 in flight	0.80 s
50 per request, all in flight	0.22 s median
My deterministic engine, for scale	0.3 ms

That is about 3.6 times faster than the best brute-force run, and it is also the cheap way to do it. A batched pass is about 99,000 input tokens. At $0.042 per million, that is $0.004. The one-per-request pass is 371,000 tokens and $0.016, because each request carries about 300 tokens of fixed overhead and each question adds only about 100.

Where the Time Goes

TypeSafe’s API sits behind an Envoy proxy, and every response carries an x-envoy-upstream-service-time header, which indicates how many milliseconds the proxy waited on the service behind it. It isn’t in TypeSafe’s docs, so don’t build on it, but it splits each request in two. A single question spends about 80 ms inside TypeSafe, and a 50-question request is about 105 ms. The other 25 to 30 ms is my connection and the proxy.

With every request in flight at once, the wall time is simply the slowest single request. That is why 25, 50, 100 and 250 per request all land between 0.19 and 0.27 s, and why balancing request sizes wasn’t worth building (I measured it: token count barely correlates with latency at 50 questions or fewer). From my dev machine, the floor is about 0.16 s. There isn’t much geography left to win, either. Even a client sitting next to the API would only save that last 25 ms.

What Pushed Back

A token ceiling. One request with all 975 questions fails with 400 max_tokens_exceeded. The documented budget is 64K tokens per request. 500 questions, about 47,000 tokens, fits, but it is a bad trade and each request takes 0.49 s and accuracy drops to 98.9%. Nothing gets better past about 100 questions per request.

Rate limits I never found. The documented limits are 1,200 requests per minute and 250,000 tokens per second, answered with a 429, plus a 529 Overloaded when the service sheds load. I sent more than 8,000 requests in a few minutes, 975 of them at the same instant, and saw neither. TypeSafe says those limits move without notice while they add capacity, so keep that in mind.

Retries, but not the SDK’s. The TypeSafe SDK retries by default, but its backoff starts at 500 ms and the result never tells you how many retries happened. One refusal would turn a 0.2 s run into 0.7 s with no explanation. So the script builds the client with retries off and retries on its own. It retries the same errors the SDK would, using the status list the client exposes, and it lets a 429’s retry-after override the delay:

const client = new TypeSafeClient({ timeout: 60_000, retry: { maxRetries: 0 } });

for (let retries = 0; ; retries++) {
  try {
    const result = await client.systemOne({ model, state, questions });
    return { ms: performance.now() - started, retries, answers: result.answers };
  } catch (error) {
    const { message, retryAfterMs } = error as { message: string; retryAfterMs?: number };
    const isRetryable = error instanceof APIConnectionError
      || (error instanceof APIError && client.retry.httpStatuses.has(error.status));
    if (isRetryable && retries < maxRetries) {
      await sleep(retryAfterMs ?? (retries + 1) * (50 + Math.random() * 100));
      continue;
    }
    return { ms: performance.now() - started, retries, error: message };
  }
}

The retries column read zero in every run in this post. It stays in the harness anyway, because a benchmark that hides its retries can’t tell a slow service from a busy one. That distinction is the whole story of the next section.

Will This Replace My Expression Engine?

No and that wasn’t really the point. I just wanted to test against something that made sense to me and the result is, Jev is blazing fast and remarkably accurate. Having AI serve as an expression engine without having to write an expression engine, that’s pretty amazing.

What the benchmark shows is a shape of dozens of typed decisions against one shared state, in about 130 ms, for a fraction of a cent, each with a probability you can gate on. That shape fits the plant-floor decisions you can’t write as an expression. A few I want to try next:

Downtime reason coding. Operators type “filler jammed again, cleared cap chute” or pick “Other”. A choice question over pre-defined reason codes, with the stop’s context as state, it could code the confident ones and queue the rest for review. “Other” is often the biggest bar on an OEE Pareto chart.
Shift notes into structured flags. One note as state, many questions at once could determine: safety concern? quality hold? needs a work order? which asset? how urgent? That is the same many-questions-one-state batching that made this benchmark fast.
Alarm flood triage. The last 50 alarms as state: is this one root cause cascading? which alarm came first? does it need action in the next ten minutes?
Review by exception. A score question grading each completed CIP cycle against a plain-language rubric, so QA reads the ten odd washes instead of all two hundred.
Checking the configuration itself. Does this expression actually describe the activity it is named after? Maybe someone fat-fingered an expression where an acid step requires a caustic valve to be open. No expression engine can catch that, because the expression is valid.

Looking Ahead

The same harness works for any of the ideas above. Swap the CSV for historical downtime events, make the question a choice over reason codes, and use the codes you already use.