An assistant can give a convincing answer without being able to finish an errand. The gap often appears after the answer: a website needs an action, a process stops halfway, or the task reaches a payment that nobody approved.
We would evaluate an assistant against nine parts before giving it work that matters. This is a proposed checklist, not a claim that every assistant has them or that checking the boxes guarantees success.
1. Workflows must resume after a crash without repeating actions
A recurring task should not depend on someone remembering to open a chat. The workflow needs a starting condition, a clear sequence and a way to recover when a step fails.
For example, a weekly invoice check can collect records on a schedule and flag missing payments. The model can judge an unclear description. The workflow should handle the ordinary steps and keep the job open until the required result exists.
Ask the provider what happens after a crash. Can the job continue, or does it start again and risk repeating an action? Anthropic separates predefined workflows from model-directed agents, while Temporal documents workflows that resume after interruption. Those are useful building blocks, not proof that a particular product finishes your task.
2. A task board must show what is waiting and what is finished
The board should tell you what is pending, running, waiting or finished. Each task needs an owner and evidence of its result.
A draft is not a sent message. A reservation request is not a confirmed table. An upload is not a live page. Look for those distinctions in the product's reporting.
Give the assistant a task that must wait for a reply. Then check whether it remembers the open work without claiming it is done. A system that only tracks activity will leave you tracking the outcome.
3. An audit trail must explain what changed and what failed
An owner needs to see what changed and what did not. Technical traces are useful for developers. A plain report is useful for the person who handed off the job.
OpenAI's Agents SDK documents tracing for model runs and tool calls. An assistant still needs to turn that record into a readable account: requested result, last checked step, failure reason and next decision.
The log should avoid unnecessary private content. It should explain the action without copying every password, message or customer detail into another store.
4. Browser work must end with a checked result
A browser lets the assistant use portals that have no suitable integration. It also introduces a different kind of failure. An input may not save. A button may work but lead to the wrong result. The page may ask for a check the assistant cannot complete.
Ask for a final-state check, not a click count. If it books a table, it should confirm the restaurant, party size, date and time from the resulting reservation.
The original WebArena research found web tasks difficult for its baseline agents. It is older benchmark evidence, not today's ranking of browser products. The practical lesson remains worth testing: completing a sequence of actions and completing the user's task are different things.
5. API retries must check whether the first attempt succeeded
Where an API fits the job, it can offer structured inputs and clear results. The assistant needs to handle expired access, uncertain responses and retries without silently duplicating work.
Ask what happens when a write succeeds but its response is lost. The safe next step is to check the stored result before repeating the write. Permissions should cover the required operation, not every operation the service supports.
Credentials should stay in a secure store rather than appear in generated reports. The assistant should also distinguish an access failure from a task outcome: "could not check" is different from "nothing found".
6. Payments must match the item, total and terms you approved
Payment access should come with limits and a record of authorization. The assistant needs the selected item, the total and the conditions the owner approved.
Google's Agent Payments Protocol describes mandates for carrying evidence of intent and authorization. That is one approach to a real problem: an instruction to research a purchase is not the same as permission to spend.
Check how the assistant handles a changed price, a cancellation fee or an uncertain payment result. It should not use yesterday's approval for a different purchase or retry a payment before checking whether the first attempt succeeded.
7. Memory must honor corrected and deleted preferences
Useful preferences and past decisions should carry into later tasks. They also need a way to change. OpenAI's memory announcement describes remembering details and controls to manage or remove them.
Test a corrected preference in a new task, then delete it. An assistant should not keep acting on a stale fact simply because it was once useful.
8. Human approval must pause the job, not erase its progress
A job can reach a point where the owner needs to decide. The assistant should pause, explain the missing choice and keep the work ready to resume.
LangGraph's interrupts documentation describes saving state while waiting for external input. Ask the provider how its product uses that mechanism, or another one, without restarting the job or asking the same question again.
9. Account access must stay within the permission you gave
The assistant needs to identify itself where required and use the owner's correct account. These are different checks. Cloudflare's signed-agents work lets sites verify agent identity; it is not blanket permission to act for an account owner.
Check what happens with two accounts, an expired login and a permission the assistant does not have. Secrets should remain in secure storage, not generated reports. The assistant should state the missing access rather than invent a result.
Test crashes, missing facts, delayed replies and changed prices
Choose a low-risk job you can inspect. Interrupt it halfway. Make one required fact unavailable. Ask it to wait for a reply. Change a price before payment.
The answer you want is not "the assistant never failed". You want it to notice the failure, keep a truthful task state and return the missing decision without making you investigate what happened.
The nine parts belong together. Workflow keeps work moving. Task state keeps it accounted for. Audit makes it explainable. Browser and API access let it act. Payment limits stop access from turning into unlimited authority.