Vision, Planning, Safety, and Evaluation
Integrate visual perception, typed planning, controlled execution, and a fixed evaluation set.
Previous · Course overview · Next
Assignment
Add visual perception and controlled execution to the complete agent. Continue in your project repository and:
- Choose an image containing information required for the project’s task.
- Send the user goal, current state, and image to the model.
- Ask the model for a typed plan or one typed next action.
- Validate the proposed action in application code.
- Execute only approved actions in a mock or sandbox, then return the new state to the agent so it can replan or finish.
- Add termination, confirmation, step, and cost limits.
The image may be a screenshot, document page, chart, diagram, photograph, or another domain-relevant visual input.
Create a fixed evaluation set with at least three normal cases and two failure or adversarial cases. Cover ambiguous or missing information, tool failure, prompt injection, and an unauthorized action. Reproduce one failure, add a deterministic control, and rerun the same case to measure the difference.
The visual input must affect the agent’s decisions. A decorative image, image caption, or unvalidated one-shot action does not satisfy the assignment.
Example
Choose visual inputs and actions that fit your project; they do not have to imitate browser automation.
Deliverables
- A meaningful multimodal path integrated with the existing agent.
- Typed planning, deterministic validation, and a safe executor.
- An architecture and trust-boundary diagram.
- A fixed set of normal, failure, and adversarial evaluation cases.
- Effectiveness, safety, latency, model-usage, and cost results.
- A documented failure, implemented control, and release recommendation.
- Final tests, traces, technical report, and presentation material.