Computer-Use Agent Pixel Audit Lab: Spot Screen Tricks
This is a hands-on lab for deciding what to check before a screen-reading AI agent clicks. You see a fictional mock screen, decide what the audit stage should flag and what the harness should block, and learn which tricks pixels can reveal and which they cannot. You make 12 scored calls and see which source backs each answer.
The screens are drawings and every name is invented. The rules come from vendor computer-use documentation, OWASP and two paper abstracts, listed below. For text-only injection through tools and documents, try the Agent Prompt Injection Harness Lab.
You are Priya Raman, security lead at Northwind Logistics. Your GUI agent Atlas sees screenshots and clicks, types and scrolls like a person at a browser. You are adding a pixel-audit stage: a check that looks at each screenshot, and at what the page reports about itself, before Atlas acts. Over 12 scored decisions, each shown with a fictional mock screen, you decide what the stage flags, what the harness blocks and when a human is asked. Every screen and name is invented. Each call scores 0 to 10 with the source behind the feedback.
Make it yours
Replace the sample names so the scenario and the debrief you export are about your organisation.
TL;DRWhat this is
5 points
- A scenario lab for people who run or review computer-use (GUI) agents: AI agents that look at screenshots and click and type. You act as security lead at a fictional company and design a pixel-audit stage, a check made before the agent acts.
- You make 12 scored decisions, each beside a fictional mock screen: nearly invisible text, a panel styled like a system dialog, a button whose label and action differ, a spoofed set-of-mark number, transparent and off-screen elements, a look-alike address, an unrequested pop-up, an irreversible confirm box, a shared browser profile, an allow-list of sites, a clean page with a persuasive notice, and the run log.
- It draws on Anthropic's and OpenAI's computer-use documentation, the abstracts of the Set-of-Mark paper (arXiv 2310.11441) and the pop-ups paper (arXiv 2411.02391), OWASP LLM01 and OWASP's agentic guide. The arXiv papers were read at abstract level only.
- Honest limit built into the lab: a pixel audit catches display deception, not plain visible content that persuades a vision model. Isolation, allow-lists and human approval carry the rest.
- Free to play, no account needed. The screens are drawings, not screenshots of any product. No exploit text is shown.
How this differs from our other agent labs
Most of our injection labs are about text: a poisoned tool description, a document in a retrieval store, a message from another agent. This lab is about pictures. The agent sees pixels, so the attack surface is how things look: contrast, overlays, what a button says versus what it does, what an address bar appears to read. The decisions sit beside mock screens for that reason.
Neighbouring labs cover other layers. The Agent Egress Containment Lab is about what an agent may reach on the network, the Agent Step-Up Approval Lab about which actions need independent approval, the Agent Credential Broker Lab about where credentials live, and the AI Prompt Injection Incident Lab about responding after one lands. Here the question is what to check on the screen itself, and what that check cannot see.
Frequently asked questions
What is a pixel audit for a computer-use agent?
It is a check, run before the agent acts, on what the agent is about to see and click. It looks at the screenshot (contrast, overlays, look-alike marks) and compares it with what the browser or operating system reports about the same screen (the real address, the real target of a button, hidden or off-screen elements). It is our name for the stage, not a standard. The lab teaches what to put in it.
What can a pixel audit catch?
Display tricks: text too faint or small for a person to read, a panel drawn to look like a system dialog, a label that does not match the control's target, a mark drawn by a page that imitates the harness's own numbering, and invisible or off-screen elements (when pixels are compared with page structure). It also gives you a log line and a reason to ask a person.
What can a pixel audit not catch?
Plain, visible content that persuades the model. Anthropic's documentation says Claude can follow commands found in content, including instructions on webpages or in images, even when they conflict with yours. A normal-looking notice passes every pixel test. That is why the lab also scores isolation, allow-lists, confirmation of irreversible actions and not giving the agent sensitive data.
Can a vision model be steered by what is visible to humans too?
Yes. Anthropic says instructions on webpages or contained in images might override your instructions, and OpenAI says to treat screen content as untrusted and that text on a page cannot grant permission for an action. The Pop-ups paper's abstract reports that adversarial pop-ups that humans would typically ignore were clicked by the tested agents on average 86% of the time in two test environments, and that simple defences such as telling the agent to ignore pop-ups did not stop the attack. Those are the authors' figures, from the abstract; we did not read the full paper.
Why not just tell the agent to ignore pop-ups and hidden text?
The pop-ups paper's abstract reports that instructing the agent to ignore pop-ups, or adding an advertisement notice, did not stop the attack. A rule in the harness that the model cannot argue with (only the step's named control may be clicked, only allow-listed sites may load) is a stronger design. That is PlayCISO's reading, not the paper's.
What is Set-of-Mark prompting, and why is it in a security lab?
Set-of-Mark is a visual prompting method: a segmentation model divides an image into regions, the regions are labelled with marks such as numbers, and the marked image is given to a model like GPT-4V so it can point to specific parts. The paper presents it as a way to improve visual grounding. Agent harnesses that draw numbered boxes use the same idea, so a page that draws look-alike numbers can confuse "click 2". That risk is our reasoning, not the paper's.
Which controls do the vendors recommend for computer use?
Anthropic lists a dedicated virtual machine or container with minimal privileges, no access to sensitive data such as login information, an allowlist of domains, and human confirmation of decisions with real-world consequences. OpenAI advises an isolated browser or VM, an allow list of sites and actions, a fresh browser profile, keeping users in control of hard-to-reverse actions, and bounding runs by step, time or cost. Both vendors sell the tools they describe.
Are the screens real screenshots?
No. They are drawings made for this lab with invented names and no logos, and the instruction-like lines are shown as grey bars with no wording. They do not show any real product, site or attack string.
How is the lab scored?
Each of the 12 decisions scores 0 to 10 and the result is your share of the maximum, with a short band saying what it means. The debrief shows the source behind each rationale. Lines labelled PlayCISO practice are our reasoning, not a statement by a source.
Sources
All read on 4 November 2026. Lines in the feedback marked “PlayCISO practice” are our own reasoning, not a statement by any source. The scenarios are fictional. We cite primary sources, not vendor advisories, and say where a page could not be fully read.
- Anthropic documentation: Computer use tool: security considerations (undated documentation page) Read as far as the security section; the rest of the page was not read. Anthropic is also a vendor of the tool described.
- OpenAI developer documentation: Computer use guide (Run safely) (undated documentation page) OpenAI is a vendor of the tool described. The older platform.openai.com address redirects here.
- Yang, Zhang, Li, Zou, Li and Gao: Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V (arXiv 2310.11441) (submitted 17 October 2023, revised 6 November 2023) Abstract only: the paper itself was not read. It describes the marking method, not an attack.
- Zhang, Yu and Yang: Attacking Vision-Language Computer Agents via Pop-ups (arXiv 2411.02391) (submitted 4 November 2024, revised 24 May 2025) Abstract only: the paper itself was not read. Figures quoted on the page are the authors' own, from the abstract.
- OWASP GenAI Security Project: LLM01:2025 Prompt Injection (2025 edition) Says it is unclear whether fool-proof prevention exists; lists mitigations including output validation, least privilege, human approval for high-risk actions and segregating external content.
- OWASP GenAI Security Project (Agentic Security Initiative): Agentic AI: Threats and Mitigations (17 February 2025 (as dated on the page)) We read the introduction and cite the guide as a threat-model reference; we quote no individual threat entry.
More security tools
The tools people use most on PlayCISO.
Catch injected instructions hiding in content your AI agent should only read.
Score your readiness against the first international standard for AI security threats.
Critical CVEs across 21 major vendors, month by month — and what the 2026 surge means.
Build an ordered ruleset, replay labelled packets and catch shadowed and any-to-any rules.
Self-assess all six Functions, from Govern to Recover, and see your weakest one.
Search 172 threat actors across Microsoft, CrowdStrike, Mandiant and community names.