Agentic Validator
Live
Agentic Validator is OTOT's approach to validating AI systems that take action
Federal AI is moving from systems that answer to systems that act: agents that read a case file, query a system of record, draft a determination, and file it. The oversight question changes with it. Validating an agent is not only asking whether the response was accurate. It is asking whether every step along the way was authorized, recoverable, and provable after the fact.
The answer can be right and the run can still be wrong.
An agent can reach a defensible final output after calling the wrong tool, reading a record it was never scoped to see, or writing to a production system on a step no one reviewed. If you only judge the output, this would be considered a pass.
Consider an agent that has a service account and resolves tickets end to end without a human in the loop. It completes the task correctly ninety-eight times. On the ninety-ninth, a document it retrieves contains instructions. The agent treats that retrieved text as direction rather than data, and acts on it. Nothing in the final response looks unusual. The failure is in the trajectory, and a test that only grades outputs will never see it.
What a validation run involves
Four things need to happen for an agent assessment to mean anything.
Each step answers a question posed by the previous step.
Scope
Write the authority down before you test it, not after.
Most agentic deployments arrive with an implicit permission model: whatever the service account can reach, the agent can reach. Scope makes that boundary explicit before any testing starts. We inventory the tools the agent can call, the data it can retrieve, the systems it can write to, the decisions it can make unattended, and the points where a human is supposed to be in the loop. That inventory becomes the specification everything downstream is tested against. Building it takes no execution at all: it reads configuration, identity policy, and tool manifests.
Exercise
Grade the path, not just the destination.
The agent runs against scenario suites built from your mission workflows. We score whether the plan was sound, whether the agent selected the right tool, whether parameters were correct, whether retrieved context supported the action taken, and whether the run stayed inside the authority model Scope established. Because the same agent can respond differently each time it runs a scenario, even with identical inputs, scenarios run repeatedly, producing a distribution of behavior rather than a single pass or fail.
Stress
Assume the environment is hostile and the tools are unreliable, because eventually both are true.
Agents fail in ways single-turn models do not. Stress targets those failure modes directly: instructions embedded in retrieved documents and tool responses, tools that time out or return malformed data, handoffs between agents that lose or distort context, loops that consume budget without converging, and actions that cannot be reversed once taken. The question is not only whether the agent succeeds, but whether it degrades safely and stops when it should.
Attest
Evidence an auditor can follow, not a dashboard that disappears.
Oversight of agentic systems is moving toward demonstration rather than documentation. Attest packages the run record into evidence: what was tested, what the agent did, where it stayed inside its authority and where it did not, and how behavior compares to prior baselines. Re-running the same suite after a model update, a prompt change, or a new tool integration turns assurance into a monitored property rather than a one-time review.
What gets measured
Agent behavior is scored across domains that a single-turn model evaluation does not reach.
Results map to the frameworks agencies are already using, including the NIST AI Risk Management Framework.
- Task completion. Whether the agent achieved the mission outcome, not merely produced a plausible response.
- Trajectory quality. Whether each intermediate step was justified by what the agent knew at that point.
- Tool selection and parameter accuracy. Whether the right tool was called correctly, at the right point in the run.
- Authority adherence. Whether the run stayed inside the documented authority boundary.
- Injection resistance. Whether instructions in retrieved content or tool output were treated as data rather than direction.
- Recovery and graceful failure. Whether the agent recovers cleanly when a dependency fails, returns malformed data, or times out.
- Reversibility. Whether consequential actions can be undone, and whether the agent knows which ones cannot.
- Checkpoint effectiveness. Whether human-in-the-loop gates actually intercept the decisions they were placed to intercept.
- Loop and cost control. Whether the agent converges, and what it consumes when it does not.
- Behavioral variance. Whether the same scenario produces consistent behavior across repeated runs, and how far it diverges when it does not.
Anvil is OTOT’s innovation lab. OTOT Technologies LLC is an SBA-certified 8(a) Small Disadvantaged Business focused on federal AI assurance, test and evaluation, and digital transformation, headquartered in Arlington, Virginia.
Browse additional accelerators to see what OTOT is forging next.
LLM Validator
Your AI passed the vendor's benchmark. That doesn't mean it works in your environment.
OTOT applies AI test and evaluation to large language models (LLMs), repeatable testing to validate whether your models provide consistent results, and whether they maintain compliance with applicable governance standards. We provide LLM validation in concert with our AI/ML governance work as a means to, first, confirm that the government receives the expected value from its AI investment, and second, to preserve and optimize that value over time.
Automated Software Quality Suite
The bottleneck in test automation isn't running the tests. It's the suite underneath them.
ASQ is a test suite comprised of four modules to automate the most time-intensive components of the T&E lifecycle. OTOT automates test case generation, automated test workflow reuse, automation test execution, and automated feedback. We put these capabilities to work on enterprise test programs depending on customer need, in full or in part.