[基于AgentEvals的自动化评估-01]面向LangChain的轨迹评估

OpenEvals是由LangChain团队推出的开源轻量级LLM/Agent评估框架旨在帮助开发者在将AI应用推向生产环境时能够系统化、标准化地测试和验证模型的输出质量告别单纯凭感觉调整提示词的落后方式。在此基础上LangChain团队又推出了一个名为AgentEvals评估框架。AgentEvals完全建立在OpenEvals之上目前只提供了两种基于Agent执行轨迹的评估一种是面向OpenAI消息风格的轨迹评估正好可以应用到LangChain和DeepAgents构建的Agent上另一种则是专门针对LangGraph执行轨迹的评估。我的系列29.基于OpenEvals的自动化评估对OpenEvals进行系统深入的介绍这个系列主要关注AgentEvals。1. 无LLM参与的轨迹匹配评估器AgentEvals定义了如下两个用来创建基于轨迹匹配评估器的create_trajectory_match_evaluator和create_async_trajectory_match_evaluator函数分别返回同步和异步执行的SimpleEvaluator和SimpleAsyncEvaluator对象。这两个方法不仅签名与OpenEvals下的同名函数完全一致底层调用的还是同一个方法。基于OpenEvals的自动化评估-12:Agent执行轨迹评估(无LLM参与)已经对这两个函数进行了详细介绍这里就不再赘言了。defcreate_trajectory_match_evaluator(*,trajectory_match_mode:TrajectoryMatchModestrict,tool_args_match_mode:ToolArgsMatchModeexact,tool_args_match_overrides:Optional[ToolArgsMatchOverrides]None,)-SimpleEvaluatordefcreate_async_trajectory_match_evaluator(*,trajectory_match_mode:TrajectoryMatchModestrict,tool_args_match_mode:ToolArgsMatchModeexact,tool_args_match_overrides:Optional[ToolArgsMatchOverrides]None,)-SimpleAsyncEvaluator正因为这两个方法是照搬OpenEvals的所以对于基于OpenEvals的自动化评估-12:Agent执行轨迹评估(无LLM参与)提供的演示程序如果我们将create_async_trajectory_match_evaluator函数导入的路径从原来的openevals改成如下所示的agentevals.trajectory评估程序一样会正常运行。importjson,asynciofromtypingimportcastfromlangchain.agentsimportcreate_agentfromlangchain.toolsimporttoolfromlangchain_openaiimportChatOpenAIfromopenevals.typesimportSimpleAsyncEvaluatorfromlangchain_core.messagesimportHumanMessage,AIMessage,ToolMessage,AnyMessagefromdotenvimportload_dotenv load_dotenv()asyncdefeval(*,evaluator:SimpleAsyncEvaluator,outputs:list[AnyMessage],reference_outputs:list[AnyMessage]|NoneNone,**kwargs):resultawaitevaluator(outputsoutputs,reference_outputsreference_outputs,**kwargs)print(json.dumps(result,ensure_asciiFalse,indent2))tooldeflook_up_location_code(city:str)-str:提取指定城市的位置代码 Args: city: 城市名称 Returns: 指定城市对应的位置代码 returnlocation-123tooldefget_weather(location_code:str)-str:提取指定位置代码所在地的天气 Args: location_code: 位置代码 Returns: 天气信息 return晴气温25度agentcreate_agent(modelChatOpenAI(modelgpt-5.4-mini),tools[look_up_location_code,get_weather])referenced_messages[HumanMessage(...),AIMessage(content,tool_calls[{name:look_up_location_code,args:{city:苏州},id:call-001}]),ToolMessage(contentlocation-123,tool_call_idcall-001),AIMessage(content,tool_calls[{name:get_weather,args:{location_code:location-123},id:call-002}]),ToolMessage(content...,tool_call_idcall-002),AIMessage(content...)]fromagentevals.trajectoryimportcreate_async_trajectory_match_evaluatorasyncdefmain():resultawaitagent.ainvoke(input{messages:[{role:user,content:今天苏州是晴天吗}]})messagescast(list[AnyMessage],result.get(messages))evaluatorcreate_async_trajectory_match_evaluator()awaiteval(evaluatorevaluator,outputsmessages,reference_outputsreferenced_messages)asyncio.run(main())输出{key:trajectory_strict_match,score:true,comment:null,metadata:null}2. 基于LLM-as-a-Judge的轨迹评估器除了上述两个用来创建无LLM参与的轨迹评估器的工厂函数AgentEvals还将如下两个名为create_trajectory_llm_as_judge和create_async_trajectory_llm_as_judge的工厂函数搬了进来名称、签名和实现都一样对此又兴趣的可以查阅我之前的文章基于OpenEvals的自动化评估-13:Agent执行轨迹评估(LLM-as-a-Judge)在这里我们也不算重复介绍它们。defcreate_trajectory_llm_as_judge(*,prompt:str|Runnable|Callable[...,list[ChatCompletionMessage]]TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,model:Optional[str]None,feedback_key:strtrajectory_accuracy,judge:Optional[Union[ModelClient,BaseChatModel,]]None,continuous:boolFalse,choices:Optional[list[float]]None,use_reasoning:boolTrue,few_shot_examples:Optional[list[FewShotExample]]None,)-SimpleEvaluatordefcreate_async_trajectory_llm_as_judge(*,prompt:str|Runnable|Callable[...,list[ChatCompletionMessage]]TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,model:Optional[str]None,feedback_key:strtrajectory_accuracy,judge:Optional[Union[ModelClient,BaseChatModel,]]None,continuous:boolFalse,choices:Optional[list[float]]None,use_reasoning:boolTrue,few_shot_examples:Optional[list[FewShotExample]]None,)-SimpleAsyncEvaluator2.1 基于参考轨迹的评估在前面的演示实例中我们利用create_async_trajectory_match_evaluator函数创建无LLM参数的评估器如果需要使用基于LLM-as-a-Judge的评估器可以按照如下的方式切换到针对create_async_trajectory_llm_as_judge函数的调用即可。fromagentevals.trajectoryimportcreate_async_trajectory_llm_as_judgeasyncdefmain():evaluatorcreate_async_trajectory_llm_as_judge(judgeChatOpenAI(modelgpt-5.4-mini))resultawaitagent.ainvoke(input{messages:[{role:user,content:今天苏州是晴天吗}]})messagescast(list[AnyMessage],result.get(messages))awaiteval(evaluatorevaluator,outputsmessages,reference_outputsreferenced_messages)resultawaitagent.ainvoke(input{messages:[{role:user,content:根据位置代码location-123提取天气信息}]})messagescast(list[AnyMessage],result.get(messages))awaiteval(evaluatorevaluator,outputsmessages,reference_outputsreferenced_messages)输出{key:trajectory_accuracy,score:true,comment:The actual trajectory follows the reference trajectory exactly in structure and semantics: it first looks up Suzhous location code, then queries the weather using that code, and finally responds with the weather result. The steps are logically ordered, efficient, and equivalent to the reference, with only trivial differences in tool call IDs and the final natural-language phrasing. Thus, the score should be: true.,metadata:null}{key:trajectory_accuracy,score:false,comment:The actual trajectory is logically consistent and efficiently accomplishes the user’s request by directly using the provided location code to call get_weather, then reporting the result. However, it is not semantically equivalent to the reference trajectory because the reference includes an earlier step that resolves the city 苏州 to location-123 via look_up_location_code before calling get_weather, whereas the actual trajectory skips that lookup and assumes the code is already known. Thus, the score should be: false.,metadata:null}2.2 无参考轨迹的评估除了上面演示的基于参考轨迹需要由reference_outputs参数提供参考轨迹我们还可以按照如下的方式利用自定义提示词实现无参考的轨迹评估。fromagentevals.trajectoryimportcreate_async_trajectory_llm_as_judgeasyncdefmain():eval_prompt You are an expert data labeler. Your task is to grade the accuracy of an AI agents internal trajectory. Rubric An accurate trajectory: - Makes logical sense between steps - Shows clear progression - Is relatively efficient, though it does not need to be perfectly efficient /Rubric First, try to understand the goal of the trajectory by looking at the input (if the input is not present try to infer it from the content of the first message), as well as the output of the final message. Once you understand the goal, grade the trajectory as it relates to achieving that goal. Grade the following trajectory: trajectory {outputs} /trajectory evaluatorcreate_async_trajectory_llm_as_judge(prompteval_prompt,judgeChatOpenAI(modelgpt-5.4-mini))resultawaitagent.ainvoke(input{messages:[{role:user,content:今天苏州是晴天吗}]})messagescast(list[AnyMessage],result.get(messages))awaiteval(evaluatorevaluator,outputsmessages)resultawaitagent.ainvoke(input{messages:[{role:user,content:根据位置代码location-123提取天气信息}]})messagescast(list[AnyMessage],result.get(messages))awaiteval(evaluatorevaluator,outputsmessages)输出{key:trajectory_accuracy,score:true,comment:The trajectory follows a logical and efficient sequence: it identifies the location code for Suzhou, queries the weather using that code, and then answers the users question directly based on the tool result. The final response is consistent with the weather tool output (“晴气温25度”). Thus, the score should be: true.,metadata:null}{key:trajectory_accuracy,score:true,comment:The trajectory is coherent and directly addresses the users request. The assistant correctly identifies the task, calls the weather tool with the provided location code, receives a plausible result, and then reports the weather information back clearly. The steps show a logical progression with no unnecessary detours, and the interaction is efficient. Thus, the score should be: true.,metadata:null}2.3 聚焦工具调用的如果轨迹评估只需要考虑工具调用可以直接按照如下方式直接使用Openevals利用常量TOOL_SELECTION_PROMPT定义的提示词。fromagentevals.trajectoryimportcreate_async_trajectory_llm_as_judgefromopenevals.prompts.trajectoryimportTOOL_SELECTION_PROMPTasyncdefmain():evaluatorcreate_async_trajectory_llm_as_judge(promptTOOL_SELECTION_PROMPT,judgeChatOpenAI(modelgpt-5.4-mini))resultawaitagent.ainvoke(input{messages:[{role:user,content:今天苏州是晴天吗}]})messagescast(list[AnyMessage],result.get(messages))awaiteval(evaluatorevaluator,outputsmessages)resultawaitagent.ainvoke(input{messages:[{role:user,content:根据位置代码location-123提取天气信息}]})messagescast(list[AnyMessage],result.get(messages))awaiteval(evaluatorevaluator,outputsmessages)输出{key:trajectory_accuracy,score:true,comment:The agent used a sensible and efficient tool sequence for answering whether Suzhou is sunny today. It first resolved the city name to a location code with look_up_location_code, which is a necessary dependency for the weather query, and then called get_weather using that location code. The order was logical, no redundant tools were used, and the final answer matches the weather result. Thus, the score should be: true.,metadata:null}{key:trajectory_accuracy,score:true,comment:The agent used a single, directly relevant tool call to retrieve weather information for the provided location code, which is the most appropriate action. The tool was called once with the correct parameter, there were no unnecessary or redundant calls, and the result was returned clearly to the user. Thus, the score should be: true.,metadata:null}由于三个实例演示已经在基于OpenEvals的自动化评估-13:Agent执行轨迹评估(LLM-as-a-Judge)中有过详细介绍这里仅仅是修改了导入create_async_trajectory_llm_as_judge函数的位置罢了。如果不明白的地方可以阅读原文。