Add fullscreen screenshot mode for safe area and edge verification#33
Add fullscreen screenshot mode for safe area and edge verification#33jfversluis wants to merge 2 commits intomainfrom
Conversation
Wire up the existing `fullscreen` HTTP API parameter through the full stack: - AgentClient.ScreenshotAsync: new `bool fullscreen` parameter - MCP tool (maui_screenshot): new `fullscreen` parameter with descriptive metadata guiding AI agents to use it for edge/safe area verification - CLI (MAUI screenshot): new `--fullscreen` option - JSON output includes fullscreen mode indicator Update skill file (SKILL.md) with safe area guidance: - New 'Safe Area & Edge Verification' section listing trigger keywords (safe area, insets, notch, Dynamic Island, clipping, edge layout, etc.) - Updated command reference table documenting default vs fullscreen behavior - Token usage tips mentioning --fullscreen for edge verification - Added fullscreen example to inspection flow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR wires the existing /api/screenshot?fullscreen=true agent capability through the DevFlow Driver (AgentClient), MCP tool (maui_screenshot), and CLI (MAUI screenshot) so AI agents and users can capture full-device screenshots for safe-area/edge verification.
Changes:
- Added
fullscreenparameter threading from Driver → CLI/MCP → agent query string. - Added CLI
--fullscreenoption plus JSON/text output labeling. - Updated MCP tool metadata and skill guidance to encourage fullscreen usage for edge/safe-area checks.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| src/DevFlow/Microsoft.Maui.DevFlow.Driver/AgentClient.cs | Adds fullscreen parameter and sends fullscreen=true query param to the agent screenshot endpoint. |
| src/DevFlow/Microsoft.Maui.DevFlow.CLI/Program.cs | Adds --fullscreen option, threads it into screenshot capture, and includes it in output. |
| src/DevFlow/Microsoft.Maui.DevFlow.CLI/Mcp/Tools/ScreenshotTool.cs | Adds fullscreen tool parameter and returns a mode label in the MCP response. |
| .claude/skills/maui-ai-debugging/SKILL.md | Documents fullscreen usage and adds a “Safe Area & Edge Verification” section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
src/DevFlow/Microsoft.Maui.DevFlow.CLI/Mcp/Tools/ScreenshotTool.cs
Outdated
Show resolved
Hide resolved
…ms, fix mode label whitespace - MCP tool: reject fullscreen + elementId/selector combination with clear error - MCP tool: use IsNullOrWhiteSpace for mode label to match agent endpoint behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Hmm I think right now this is sort of done as a fallback... So iOS I believe will try and use simctl first, and then start falling back to the agent's implementation if it doesn't work for some reason (or if it's a device). Maybe android we should just do the same as that? Use ADB first, if there's no specific element id specified and fallback ? |
|
Or I guess maybe we also need to clean up where that series of steps happens.... MCP should in theory do the same thing as the CLI... |
|
@Redth not sure what the right approach is here. My scenario was: I was building an app on iOS and the statusbar/safe area was not the color I wanted. So I told Copilot: hey I want the color underneath to also continue upwards in the safe area. And it would take a screenshot and be like: the color is already all the way in the top, so nothing to do here! Probably because it took a partial screenshot, which is good for efficiency, but not if you're looking at edges or safe areas or things :D So did an attempt to "fix" that |
Problem
When an AI agent uses DevFlow to verify layout near screen edges, it takes a default screenshot that captures only the Page content area — excluding the status bar, home indicator, notch/Dynamic Island, and safe area regions. The agent then concludes the layout "looks fine" because it literally cannot see the device chrome the content may be overlapping with.
The in-app HTTP API already has a
fullscreen=truequery parameter on/api/screenshotthat triggers composited full-device capture (all UIWindows on iOS, CGWindowListCreateImage on macOS), but this parameter was never wired through to:AgentClient.ScreenshotAsync()maui_screenshotMCP toolMAUI screenshotcommandThe CLI had a separate workaround for iOS (
simctl io screenshotfallback), but the MCP tool path — which is what AI agents use — had no way to request a full-device capture.Changes
Code (parameter threading)
bool fullscreen = falseparameter onScreenshotAsync(), sendsfullscreen=truequery paramfullscreenparameter with rich description guiding agents to use it for edge/safe area checks--fullscreenoption onMAUI screenshotcommandfullscreenfield; text output shows(fullscreen)label; MCP returns mode labelfullscreendefaults tofalseto preserve token efficiency — full-device screenshots are larger and most verification doesn't need them.Skill file (AI agent guidance)
UseSafeArea--fullscreenalongside element-level screenshotsTesting
The
fullscreen=trueHTTP API parameter already worked end-to-end (it's been inDevFlowAgentService.HandleScreenshotsince inception). This PR only wires it through the public API layers. CI will validate compilation.