跳到正文
原文
OpenRouter:Blog(Announcements)·· 2026-09-19精选AI 评分69

Jev 决策模型与 LLM 对比实测,何时该用判断模型而非生成文本

Jev vs LLM: When to Use a Decision Model Instead of Generating Text

文章要点
  • 1OpenRouter 实测 TypeSafe 决策模型 Jev 1.13 与 GPT Luna、Claude Opus 在客服分诊上的成本与准确率取舍
  • 260 张工单分诊 Jev 准确率 59/60、中位延迟 194ms、千张成本 $0.0248,Claude Opus 准确率 60/60 但千张 $2.88
  • 3Jev 返回带概率的类型化答案、无需解析文本,但只读文本且算术、计数、日期比较不可靠,也不会写回复
  • 4文章给出 Jev 先路由再加 LLM 写回复、LLM 起草后 Jev 按政策核验两种模式,完整流程单条约 $0.00015、约 1.5 秒
决策要点

选项固定的判断类任务出现了成本仅为通用 LLM 四分之一到百分之一以下的专用模型路径,写作环节仍需 LLM 承担。

  • 影响:客服分诊、升级判定和注入筛查环节的模型成本与延迟有数量级压缩空间;低置信度和高风险请求仍需回流人工,人力编排要同步设计;决策模型只读文本且对规则描述敏感,输入需先做清理。
  • 参考:可考虑先盘点代码里以 JSON 解析加 switch 收尾的 LLM 调用,作为第一批替换对象;用一两百条已标注工单跑出准确率随阈值变化的曲线后再定 cutoff;写回复的分支保留 LLM,发送前由决策模型按政策文本核验草稿。
  • 依据:成本与准确率数据来自 OpenRouter 单日 60 和 40 个样例的实测,作者自述样本小、只展示取舍而非排行榜;价格为 OpenRouter 模型页口径,暂无独立评测。
正文 · AI 翻译

假设你有一个产品,每月会收到 40,000 张支持工单,每一张都需要分类,并给出三样东西:它是关于什么、是否应该升级、以及回复内容是什么。

一个带 JSON schema 的前沿 LLM 可以胜任这项工作。我们的基准测试显示,它完成这项工作的成本为每 1,000 张工单 $2.88,中位延迟两秒,约合每月 $115。一个小型 LLM 的成本是每 1,000 张 $0.09,耗时约一秒。

另一种思考方式是问问决策模型能做什么。Jev 是 TypeSafe 推出的一款决策模型,在同一轮测试中,它完成分类的成本为每 1,000 张工单 2.5 美分,中位延迟 194 毫秒,约合每月一美元。此外它返回的是概率,因此无需解析任何内容。

不过它无法撰写回复。为此,你仍然需要一个 LLM。因此,我们通过 OpenRouter 在 100 个支持案例上对 Jev、GPT Luna 和 Claude Opus 进行了基准测试,并构建了两种同时利用两者的模式。

Jev 是什么,以及每个模型实际返回什么

让语言模型做一个代码层面的决策,它返回的是文本。你可以要求 JSON,但这类模型天生是为写作而构建的。你的代码必须解析输出,并确保模型给出的是回答而不是解释。

Jev 返回的则是类型化答案。你向它发送状态和由三种基元构成的问题:choice 从你定义的选项集中选出一个选项,noul 返回一个“是”的概率,score 将内容放置在你描述的层级上。选择和评分答案可以包含你自己键上的概率,而 noul 是一个单独的数字。下面是针对一张计费工单的两个问题的响应。

{
  "answers": {
    "intent": {
      "type": "choice",
      "choice": "billing_dispute",
      "confidence": 1,
      "probabilities": { "billing_dispute": 1, "order_status": 0, "other": 0 }
    },
    "escalate": { "type": "noul", "noul": 0.04 }
  },
  "model": "typesafe/jev-1.13-20260917",
  "provider": "TypeSafe",
  "usage": { "cost": 0.000016002, "inputTokens": 381, "outputTokens": 62 }
}

意图是你自己的键之一,而升级则是一个数字,你可以拿它与你设定的阈值进行比较。

TypeSafe 将 Jev 称为系统一模型。这意味着它做出的是快速、直觉的判断,而不是缓慢、审慎的分析。它只接受文本,因此不支持图像、音频或 PDF 作为输入。它按字面意思解读评判标准。此外,当它检查的状态中包含无关细节时,往往会损失准确性,而且它在算术、计数和日期比较方面也不可靠。

并排对比 Jev 与生成式 LLM

Jev 1.13传统 LLM(GPT Luna、Claude Opus)
输出类型化选择、是/否概率或评分,并附带每个选项的概率自由文本(可选择约束为 JSON)
最擅长分类、路由、验证、排名、护栏检查、有界抽取写作、解释、总结、改写、代码,以及任何输出空间事先未知的任务
输入仅文本(字符串、JSON、数组)。每次请求 64k token,状态加上最长的问题共 32k文本,对许多模型来说还有图像、音频、PDF
价格(观测于 2026-09-19)每百万输入 token $0.042,输出免费GPT Luna 每百万输入 $0.20 / 输出 $1.20。Claude Opus 每百万输入 $5 / 输出 $25
60 张工单分类的中位延迟194 ms1,106 ms (Luna), 1,957 ms (Opus)
每 1,000 张已分类工单的成本$0.0248$0.0921 (Luna), $2.88 (Opus)
OpenRouter 端点POST /api/alpha/decisionsPOST /api/v1/chat/completions

价格来自运行当天 OpenRouter 的模型元数据。在制定预算之前,请先查看模型页面。

我们测量了什么

测试的模型是 Jev 1.13、GPT Luna(解析为 GPT 5.6 Luna)和 Claude Opus(解析为 Claude Opus 5)。这些 LLM 在系统提示中获得了与 Jev 相同的定义和升级规则,temperature 设为 0,并要求返回一个裸 JSON 对象。每个示例发送一次请求。这些都是小规模数据集:单日内进行 60 和 40 个示例的测试,且只使用了一组提示。这展示的是这种权衡的形态,而非一份排行榜。

任务 A:把 60 个支持工单分类到五种意图,外加一个升级标记

六十个支持工单,涵盖五种意图:订单状态、退货或退款、账单争议、产品问题、账户访问,每种都配有一句话的定义。升级情形涵盖法律威胁、拒付、疑似欺诈、安全隐患,以及威胁将事情公开曝光。

模型意图准确率升级准确率p50 延迟p95 延迟总成本(60 个)每 1,000 个
Jev 1.1359/60 (98.3%)60/60 (100%)0.194s0.633s$0.001489$0.0248
GPT Luna59/60 (98.3%)60/60 (100%)1.106s2.395s$0.005527$0.0921
Claude Opus60/60 (100%)59/60 (98.3%)1.957s2.594s$0.17281$2.8802

准确率不相上下。Jev 发送的输入 token 更多,因为每个请求都携带完整的判定标准,但它的成本仍只有 GPT Luna 的四分之一左右、不到 Claude Opus 的百分之一。意图误判只有一例,是一个关于拆分退货商品退款的问题。Jev 以 0.56 的置信度把它归入了退货或退款。它是唯一一个低于 0.8 的工单,所以 0.8 的阈值反正也会把它转给人工处理。

任务 B:筛查 40 条消息中的提示注入

二十二条普通支持消息和十八条注入尝试(“无视指令”类、伪造管理员标记、角色扮演话术、引用指令)。我们给 Jev 一个是非题:这是否是一次改变助手行为的尝试;各个 LLM 拿到相同的定义,并返回一个布尔值。

模型准确率p50 延迟p95 延迟总成本(40 条)每 1,000 条
Jev 1.1340/40 (100%)0.194s0.688s$0.000646$0.0161
GPT Luna39/40 (97.5%)0.805s1.491s$0.001828$0.0457
Claude Opus40/40 (100%)2.099s4.377s$0.063555$1.5889

Jev 干净利落地分开了两组。注入消息得分在 0.86 到 0.99 之间,普通消息在 0.01 到 0.20 之间。Luna 唯一的漏判是一条例行角色扮演话术。

当答案是 N 选一时,用 Jev

任何最终归结为一条 switch 语句的问题都是 Jev 问题。意图、优先级、语言、情感、进哪个队列、是否违反政策、这一主张是否成立。在每种情况下,答案都事先已知。你只需要每个可能答案的概率。下面是基准测试中使用 OpenRouter TypeScript SDK 的分类调用。

// triage.ts
import { OpenRouter } from '@openrouter/sdk';

const openrouter = new OpenRouter({
  apiKey: process.env.OPENROUTER_API_KEY, // server-side only
});

const INTENTS = {
  order_status:
    'The customer asks where an order is, when it will ship or arrive, wants to change or cancel an order before delivery, or reports a package missing or partially delivered.',
  return_refund:
    'The customer wants to return, exchange, or replace an item they received, or asks about the status or rules of a return they already started.',
  billing_dispute:
    'The customer says a charge, invoice, tax, discount, or refund amount is wrong, duplicated, unexpected, or unauthorized.',
  product_question:
    "The customer asks about a product's features, compatibility, sizing, materials, stock, warranty, or safety before or after buying, without asking to return it.",
  account_access:
    'The customer cannot log in, needs to change login or account details, or asks to merge, delete, secure, or share an account.',
} as const;

export type Intent = keyof typeof INTENTS;

const ESCALATE =
  'Does the ticket describe any of the following: a threat of legal action, a regulator complaint, or a chargeback; suspected fraud or an account takeover; a safety hazard such as fire, smoke, or injury; or a customer who says this is a repeated failure and threatens to publicize it?';

export type Triage = {
  intent: Intent;
  confidence: number;
  probabilities: Record<string, number>;
  escalateProbability: number;
  costUsd: number;
};

function isIntent(value: string): value is Intent {
  return Object.hasOwn(INTENTS, value);
}

export async function triage(ticket: string): Promise<Triage> {
  const result = await openrouter.alpha.decisions.create({
    decisionsRequest: {
      model: 'typesafe/jev-1.13',
      state: { ticket },
      questions: {
        intent: {
          type: 'choice',
          instructions: 'What is the primary intent of the ticket?',
          criteria: INTENTS,
        },
        escalate: { type: 'noul', instructions: ESCALATE },
      },
    },
  });

  const intent = result.answers.intent;
  const escalate = result.answers.escalate;
  if (intent?.type !== 'choice' || escalate?.type !== 'noul') {
    throw new Error('Unexpected answer types');
  }
  if (!isIntent(intent.choice)) {
    throw new Error(`Unknown intent ${intent.choice}`);
  }
  return {
    intent: intent.choice,
    confidence: intent.confidence ?? 0,
    probabilities: intent.probabilities ?? {},
    escalateProbability: escalate.noul,
    costUsd: requireCost(result.usage.cost),
  };
}

function requireCost(cost: number | undefined): number {
  if (cost === undefined) {
    throw new Error('Response did not include usage.cost');
  }
  return cost;
}

那段分类代码里有三点值得注意:

每个问题只要求一个小的判断,然后在代码中组合这些答案。这里意图和升级是同一次请求中的两个独立问题。

像写规格说明那样写判定标准。由于 Jev 是按字面理解的,当某个案例被归错类别时,先修改判定标准的文字。

只发送每个问题所需要的状态。无关细节会降低答案的准确率。

当答案是成段文字时,用 LLM

现在,当我们需要真正生成一条回复时,就用 LLM。

// draft-reply.ts
import { OpenRouter } from '@openrouter/sdk';

const openrouter = new OpenRouter({ apiKey: process.env.OPENROUTER_API_KEY });

const REPLY_MODEL = '~openai/gpt-luna-latest';

const REPLY_SYSTEM =
  'You write short, warm replies for the Northwind support team. Use only the facts in the user message. Do not promise refunds, credits, or dates that are not in the facts. Three sentences maximum.';

type Draft = { text: string; model: string; costUsd: number };

export async function draftReply(ticket: string, facts: string): Promise<Draft> {
  const result = await openrouter.chat.send({
    chatRequest: {
      model: REPLY_MODEL,
      messages: [
        { role: 'system', content: REPLY_SYSTEM },
        { role: 'user', content: `Ticket: ${ticket}\n\nFacts: ${facts}` },
      ],
    },
  });
  // chat.send can return a stream when streaming is requested. It isn't here, so reject that case.
  if (result instanceof ReadableStream) {
    throw new Error('Expected a non-streaming response');
  }
  const text = result.choices[0]?.message.content;
  if (typeof text !== 'string') {
    throw new Error('Expected text content');
  }
  const cost = result.usage?.cost;
  if (typeof cost !== 'number') {
    throw new Error('Response did not include usage.cost');
  }
  return { text, model: result.model, costUsd: cost };
}

按写作任务挑选模型。当你需要快速又便宜的简短回复时,GPT Luna 就能胜任,本次运行中每条回复约 $0.0001。当写作需要真正的推理、长上下文或代码时,再升级到前沿模型。遇到 Jev 的一些局限时,LLM 同样是答案,比如输入包含图片、音频或 PDF,或者输出是文档、diff 或计划。

用 Jev 路由,用代码计算,用 LLM 写作

拉远了看,第一种模式很简单。Jev 负责分类,代码检查它的置信度有多高。只有当交付物是成段文字时,代码才调用 LLM。

处理程序会把升级值达到 0.5 或更高的任何内容转给人工,也会把意图置信度低于 0.8 的任何内容转给人工。除此之外,订单状态直接从订单系统获取答案,不经过模型;只有需要成段文字的意图,才会让 LLM 起草。

// handle.ts
import { triage, type Intent, type Triage } from './triage';
import { draftReply } from './draft-reply';

// These thresholds are application policy. Tune them on your own labeled tickets.
const ROUTE_CONFIDENCE = 0.8;
const ESCALATE_THRESHOLD = 0.5;

type ReplyIntent = Exclude<Intent, 'order_status'>;

type Route =
  | { kind: 'human'; reason: string }
  | { kind: 'deterministic'; intent: 'order_status' }
  | { kind: 'reply'; intent: ReplyIntent };

function chooseRoute(t: Triage): Route {
  if (t.escalateProbability >= ESCALATE_THRESHOLD) {
    return { kind: 'human', reason: `escalation probability ${t.escalateProbability}` };
  }
  if (t.confidence < ROUTE_CONFIDENCE) {
    return { kind: 'human', reason: `intent confidence ${t.confidence}` };
  }
  if (t.intent === 'order_status') {
    return { kind: 'deterministic', intent: t.intent };
  }
  return { kind: 'reply', intent: t.intent };
}

// Stand-in for your order system. Exact data never goes through a model.
function lookupOrder(ticket: string): string {
  const id = ticket.match(/\b\d{5}\b/)?.[0];
  if (id === undefined) {
    return 'Please reply with your five-digit order number and we will check the shipment.';
  }
  return `Order ${id} shipped 2026-09-17 via UPS, tracking 1Z999AA10123456784, estimated delivery 2026-09-21.`;
}

export const FACTS: Record<ReplyIntent, string> = {
  return_refund:
    'Returns are accepted within 30 days of delivery for regular items. Exchanges for another size follow the same 30-day window. Final sale items cannot be returned or exchanged. Prepaid labels are emailed within one business day.',
  billing_dispute: 'A billing specialist will review the charge within one business day.',
  product_question: 'Product specifications are listed on each product page.',
  account_access: 'Password resets are available from the sign-in page.',
};

export type Handled = {
  triage: Triage;
  route: Route;
  text: string;
  costUsd: number; // Jev call plus the LLM call, if one happened
};

// Routes a ticket. A 'reply' route carries an unverified LLM draft; send.ts checks it before anything goes out.
export async function handle(ticket: string): Promise<Handled> {
  const t = await triage(ticket);
  const route = chooseRoute(t);
  switch (route.kind) {
    case 'human':
      return { triage: t, route, text: `queued for a person: ${route.reason}`, costUsd: t.costUsd };
    case 'deterministic':
      return { triage: t, route, text: lookupOrder(ticket), costUsd: t.costUsd };
    case 'reply': {
      const reply = await draftReply(ticket, FACTS[route.intent]);
      return { triage: t, route, text: reply.text, costUsd: t.costUsd + reply.costUsd };
    }
    default:
      return route satisfies never;
  }
}

一个简短的运行器会在每个结果旁边打印 Jev 的信号。

// run.ts
import { handle } from './handle';

const TICKETS = [
  "Hi, I ordered a pair of running shoes on the 3rd and the tracking page hasn't updated in five days. Where is my package? Order 84721.",
  'The jacket I got is too small. Can I swap it for a large? I got it last Tuesday.',
  "There's an unauthorized $450 charge from your company on my card. I've already called my bank to dispute it and I'm reporting this as fraud.",
  "I paid with a gift card and a credit card, but the refund only went to the credit card. Where's the gift card balance?",
];

for (const ticket of TICKETS) {
  const r = await handle(ticket);
  const { intent, confidence, escalateProbability } = r.triage;
  console.log(`"${ticket}"`);
  console.log(`  intent=${intent} confidence=${confidence} escalate=${escalateProbability} route=${r.route.kind} cost=$${r.costUsd.toFixed(6)}`);
  console.log(`  ${r.text}\n`);
}

下面是四个示例工单的输出。

"Hi, I ordered a pair of running shoes on the 3rd and the tracking page hasn't updated in five days. Where is my package? Order 84721."
  intent=order_status confidence=1 escalate=0.02 route=deterministic cost=$0.000026
  Order 84721 shipped 2026-09-17 via UPS, tracking 1Z999AA10123456784, estimated delivery 2026-09-21.

"The jacket I got is too small. Can I swap it for a large? I got it last Tuesday."
  intent=return_refund confidence=1 escalate=0.01 route=reply cost=$0.000118
  Yes, you can exchange the jacket for a large if it’s a regular item and within 30 days of delivery. If it’s a final sale item, it can’t be exchanged; for eligible exchanges, a prepaid return label is emailed within one business day.

"There's an unauthorized $450 charge from your company on my card. I've already called my bank to dispute it and I'm reporting this as fraud."
  intent=billing_dispute confidence=1 escalate=0.96 route=human cost=$0.000025
  queued for a person: escalation probability 0.96

"I paid with a gift card and a credit card, but the refund only went to the credit card. Where's the gift card balance?"
  intent=return_refund confidence=0.54 escalate=0.02 route=human cost=$0.000025
  queued for a person: intent confidence 0.54

四条中有一条接触了 LLM。订单状态查询是精确且免费的。欺诈报告从未到达过一个可能作出任何承诺的模型。由于拆分退款工单——也就是 Jev 在任务 A 中归错类的那张——的置信度低于 0.8,它被转给了人工。最后,关于夹克的回复在这里仍未经过检查。检查它就是第二种模式。

在发送之前用 Jev 验证 LLM 输出

第二种模式正好反过来。LLM 起草回复。然后在发送之前,Jev 根据政策检查草稿。这可以捕捉到友好但承诺了违反政策的 AI 回复。

// verify-draft.ts
import { OpenRouter } from '@openrouter/sdk';

const openrouter = new OpenRouter({ apiKey: process.env.OPENROUTER_API_KEY });

export const POLICY =
  'Returns are accepted within 30 days of delivery for regular items. Final sale items cannot be returned. Refunds go back to the original payment method. Prepaid return labels are emailed within one business day of approval.';

type Label = 'supported' | 'unsupported' | 'declined';

export type Verdict = {
  label: Label;
  confidence: number;
  probabilities: Record<string, number>;
  costUsd: number;
};

function isLabel(value: string): value is Label {
  return value === 'supported' || value === 'unsupported' || value === 'declined';
}

export async function verifyDraft(policy: string, question: string, draft: string): Promise<Verdict> {
  const result = await openrouter.alpha.decisions.create({
    decisionsRequest: {
      model: 'typesafe/jev-1.13',
      state: { policy, customer_question: question, draft_reply: draft },
      questions: {
        support: {
          type: 'choice',
          instructions: 'How does draft_reply relate to policy and customer_question?',
          criteria: {
            supported:
              'The draft answers customer_question, and every fact, number, timeframe, and promise in it appears in policy.',
            unsupported:
              'The draft states a fact, number, timeframe, or promise that policy does not contain or contradicts, or it answers a different question.',
            declined:
              'The draft says policy does not cover the question and adds no facts of its own beyond what policy states.',
          },
        },
      },
    },
  });

  const support = result.answers.support;
  if (support?.type !== 'choice' || !isLabel(support.choice)) {
    throw new Error('Unexpected answer');
  }
  const cost = result.usage.cost;
  if (cost === undefined) {
    throw new Error('Response did not include usage.cost');
  }
  return {
    label: support.choice,
    confidence: support.confidence ?? 0,
    probabilities: support.probabilities ?? {},
    costUsd: cost,
  };
}

Jev 会拿到政策、客户问题和草稿,然后回答一个选择题:该草稿是被支持、不被支持还是被拒答。我们用四个草稿跑了它。第一个来自 GPT Luna,另外三个是手写的,每个都是为命中特定分支而写的。

Q: "How long until my refund shows up?"
Draft (GPT Luna): "Refunds are sent back to the original payment method, but we don't have a specific timeline for when they will appear. If your return is approved, a prepaid return label will be emailed within one business day."
  supported  confidence=0.09  { supported: 0.39, unsupported: 0.32, declined: 0.29 }

Q: "How long until my refund shows up?"
Draft (fabricated): "Refunds are processed within 5 to 7 business days after we receive the item."
  unsupported  confidence=1.00  { unsupported: 1, supported: 0, declined: 0 }

Q: "Where does my refund go?"
Draft (grounded): "Refunds go back to the original payment method, so it will return to however you paid for the order."
  supported  confidence=1.00  { supported: 1, unsupported: 0, declined: 0 }

Q: "How long until my refund shows up?"
Draft (grounded, wrong question): "Refunds go back to the original payment method, so it will return to however you paid for the order."
  unsupported  confidence=0.48  { unsupported: 0.65, supported: 0.14, declined: 0.21 }

编造的时间线返回 unsupported(不支持),置信度为 1.00,如果这条回复发出去了,客户会在第八天回信问退款在哪里。

Luna 的草稿才是有意思的那个,因为它陈述的每个事实都符合政策。但它一半是拒答,一半是回答问题,还添加了一个没人要求的细节。Jev 把概率一分为三,0.8 的阈值把它送给客服人员快速看一眼。

最后两行使用同一个有据可依的草稿,但问了两个不同的问题。注意,当草稿回答了客户所问的问题时,它以 1.00 被判定为支持;当它回答了另一个问题时,则被判定为不支持。这正是你希望验证器做到的事情,也清楚地说明了为什么政策文本应该使用草稿会用到的相同词汇。

我们最初尝试了两个是非题,结果一片混乱,因为一个正确指出政策不覆盖此情况的草稿既是有据可依的,同时又不是一个回答。互斥的结果促使我们改问一个选择题。

把它们全部连接起来

还有一个文件把路由优先的流水线连接到验证器,这样 LLM 写的任何内容都不会在没有得到“支持”判定的情况下发出。

// send.ts
import { FACTS, handle, type Handled } from './handle';
import { verifyDraft, type Verdict } from './verify-draft';

const SEND_CONFIDENCE = 0.8;

type Outcome = Handled & { disposition: 'sent' | 'human_review'; verdict?: Verdict };

export async function process(ticket: string): Promise<Outcome> {
  const handled = await handle(ticket);
  switch (handled.route.kind) {
    case 'human':
      return { ...handled, disposition: 'human_review' };
    case 'deterministic':
      return { ...handled, disposition: 'sent' };
    case 'reply': {
      // Verify against the same facts the draft was written from.
      const verdict = await verifyDraft(FACTS[handled.route.intent], ticket, handled.text);
      const ok = verdict.label === 'supported' && verdict.confidence >= SEND_CONFIDENCE;
      return { ...handled, verdict, costUsd: handled.costUsd + verdict.costUsd, disposition: ok ? 'sent' : 'human_review' };
    }
    default:
      return handled.route satisfies never;
  }
}

如果验证器退回了一条单独看起来没什么问题的回复,就把回复对照事实读一读。事实经常遗漏客户问到的某些内容。去看看完整的请求路径吧。

ticket text
  |
  v
[Jev]  intent (choice) + escalate (noul) .......... 1 request, ~200 ms, ~$0.000025
  |
  v
[code] escalate >= 0.5 or confidence < 0.8 ?  -->  human queue
  |
  v
[code] intent == order_status ?  -->  database lookup, exact reply, no model
  |
  v
[LLM]  draft reply from ticket + facts ............ ~1 s, ~$0.0001 (GPT Luna)
  |
  v
[Jev]  supported / unsupported / declined ......... 1 request, ~200 ms, ~$0.000025
  |
  v
[code] supported and confidence >= 0.8 ?  -->  send
       otherwise                          -->  human review

完整走完全程的工单需要两次 Jev 调用和一次 LLM 调用。总成本约 $0.00015,总耗时 1.5 秒。两次 Jev 调用使得可以用便宜的模型来写作,同时又无需让一个人去阅读模型的每一条回复。

从你自己的数据中选取阈值

无论阈值出现在你代码中的哪个位置,请记住这个数字并非凭空而来。它来自你。这是你的政策选择。Jev 的概率是校准过的,这基本上意味着取大量预测,然后看它们正确的频率。所以如果我们看到 0.8,那么这些预测中大约有 80% 最终是正确的。但并非每一个都正确。那是一个平均值。所以请把这篇文章中看到的 0.8 和 0.5 当作我们的数字,而不是你的。

如果你想找到自己的阈值,几百条带标注的案例就够了。在所有案例上运行模型。针对每个临界值,统计它答对了多少。把准确率与临界值的关系绘制成图表。然后把临界值设在自动路径的输出与你的人工团队本会给出的结果相一致的那一点上。在你修改评判标准之后,要重新核对这一点。别忘记。改写其中一个选项的措辞,可能会以出乎意料的方式改变其他选项的概率。

从一个 switch 语句开始

如果你的代码库中有一个以 JSON.parse 结尾、随后跟着 switch 的 LLM 调用,那就是你的第一个 Jev 问题。把它替换进去,在需要成文输出的分支上保留 LLM,然后对两者都进行测量。

常见问题

Jev 是什么?它与 LLM 有何不同?

Jev 是 TypeSafe 的 System One 决策模型。它接收状态加上一个类型化的问题——或是在固定集合中做选择,或是一个是非命题,或是按你描述的等级打分——并返回概率而非文本。大语言模型生成开放式文本,你随后还得解析并信任这些文本。

在分类任务上,Jev 比小型 LLM 更便宜吗?

在 2026 年 9 月 19 日通过 OpenRouter 运行的 60 张支持工单上,Jev 的成本为每 1,000 张工单 $0.0248,GPT Luna 为每 1,000 张工单 $0.0921,Claude Opus 为每 1,000 张工单 $2.88,而意图识别准确率分别为:Jev 59/60,GPT Luna 59/60,Claude Opus 60/60。Jev 1.13 的价格为每百万输入 token $0.042,输出免费。定价可能随时变化,因此在你做预算之前,请务必查看模型页面。

如何将 Jev 与 LLM 结合使用?

有两种模式。第一种是先路由:Jev 对请求进行分类并返回一个置信度。你的代码将该置信度与你选定的阈值进行比较,只有需要成文输出的请求才会发给 LLM。第二种是事后验证:LLM 先起草,Jev 检查草稿中的每一项论断是否都有你的政策作为依据,之后回复才会发出。

Jev 使用哪个 OpenRouter 端点?

Jev 使用 Decisions API 端点,即 POST https://openrouter.ai/api/alpha/decisions,配合模型 ID typesafe/jev-1.13,或在 TypeScript SDK 中使用 openrouter.alpha.decisions.create()。传统 LLM 则使用 POST https://openrouter.ai/api/v1/chat/completions,或 openrouter.chat.send()。

在 Jev 中我应该使用多大的置信度阈值?

没有一个适用于所有情况的通用数字。Jev 的概率是在大量预测上经过校准的,因此从整体上看是可靠的,但任何单次预测仍可能是错的。所以请根据你自己的标注数据和出错的代价来选取临界值。举例来说,这是我们的路由规则:意图置信度低于 0.8,或预测出的升级概率达到或超过 0.5 的请求,就转交人工处理。

来源:OpenRouter:Blog(Announcements) · openrouter.ai