This section covers the AI Assertor tool, an LLM-powered message analyzer that uses your LLM provider to determine whether a message satisfies one or more assertions or requirements as written by you using natural language. It is most commonly connected to a tool that sends or receives messages in order to verify the content of those messages.
This tool requires a validate license. Additionally, you must configure your LLM provider account information in the LLM Provider preferences. OpenAI, Azure OpenAI, and other LLM providers with similar functionality are supported. See LLM Provider preferences to determine whether your LLM provider can be used. Be aware that LLMs can provide inaccurate information, which can affect this feature. Due diligence is recommended.
Table of Contents
Understanding the AI Assertor
...
The AI Assertor is designed to support multiple validation assertions to create complex verifications. It can be added via the Add Output wizard or as a stand-alone Standard test.
Configuring AI Assertor
The AI Assertor's tool settings consist of the following tabs:
- Configuration: This tab is used to create and configure AI assertions for the LLM.
- Expected Document: Specifies the expected document, creating a template from which you can select elements. If AI Assertor receives a valid document (for example, from traffic or from the tool it is attached to), this panel will be populated automatically. Alternatively, you can copy a sample message into tab. Note that the expected document does not get saved by default; if you want to save it, enable Save expected document.
- Input: Specifies either text or a file on which the tool should operate. If the AI Assertor is chained to another tool, this tab does not appear, and the AI Assertor uses the output of that tool as its input.
...
If you want to change your assertion, you can do so and save your changes. Clicking click Evaluate again, which will query the LLM with the updated assertion against the previous input. In this way, you can fine-tune your assertions until they are functioning the way you want. If you find the AI struggling to fully comprehend your assertion, providing it with an example or two of a passing or failing condition can help.