Use decision models with Amazon Bedrock AgentCore Evaluations
AWS Machine Learning Blog · 人工智能
Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust. Amazon Bedrock AgentCore Evaluations , a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call your choice of external system, model, or API, including a decision model. Decision models suit questions with a fixed set of answers that an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. After you validate a question on your own traffic, a decision model can be sufficient for that decision or score, and you can extend the same approach to specific harmfulness or stereotyping categories in your policy. Because both backends can answer several questions about the same state in a single request, you can use them as a screening layer rather than a replacement, running a pass or fail signal across your metrics to select the turns that warrant deeper assessment by the built-in evaluators, whose explanations help you investigate what went wrong. This post shows