Skip to content

Add fullscreen screenshot mode for safe area and edge verification#33

Open
jfversluis wants to merge 2 commits intomainfrom
feature/fullscreen-screenshot-safe-area
Open

Add fullscreen screenshot mode for safe area and edge verification#33
jfversluis wants to merge 2 commits intomainfrom
feature/fullscreen-screenshot-safe-area

Conversation

@jfversluis
Copy link
Copy Markdown
Member

Problem

When an AI agent uses DevFlow to verify layout near screen edges, it takes a default screenshot that captures only the Page content area — excluding the status bar, home indicator, notch/Dynamic Island, and safe area regions. The agent then concludes the layout "looks fine" because it literally cannot see the device chrome the content may be overlapping with.

The in-app HTTP API already has a fullscreen=true query parameter on /api/screenshot that triggers composited full-device capture (all UIWindows on iOS, CGWindowListCreateImage on macOS), but this parameter was never wired through to:

  • AgentClient.ScreenshotAsync()
  • The maui_screenshot MCP tool
  • The CLI MAUI screenshot command

The CLI had a separate workaround for iOS (simctl io screenshot fallback), but the MCP tool path — which is what AI agents use — had no way to request a full-device capture.

Changes

Code (parameter threading)

Layer Change
AgentClient New bool fullscreen = false parameter on ScreenshotAsync(), sends fullscreen=true query param
MCP Tool New fullscreen parameter with rich description guiding agents to use it for edge/safe area checks
CLI New --fullscreen option on MAUI screenshot command
Output JSON output includes fullscreen field; text output shows (fullscreen) label; MCP returns mode label

fullscreen defaults to false to preserve token efficiency — full-device screenshots are larger and most verification doesn't need them.

Skill file (AI agent guidance)

  • New "Safe Area & Edge Verification" section — explains the default vs fullscreen difference and lists trigger keywords for when agents should automatically use fullscreen:
    • Safe areas, insets, UseSafeArea
    • Content clipping/cutoff at screen edges
    • Status bar overlap, notch clearance, Dynamic Island
    • Home indicator, bottom bar, navigation bar overlap
    • Rounded corner clipping
    • Toolbar/tab bar positioning relative to device chrome
    • Keyboard avoiding behavior
  • Updated command reference table — documents both modes
  • Token usage tips — mentions --fullscreen alongside element-level screenshots
  • Inspection flow — added fullscreen example

Testing

The fullscreen=true HTTP API parameter already worked end-to-end (it's been in DevFlowAgentService.HandleScreenshot since inception). This PR only wires it through the public API layers. CI will validate compilation.

Wire up the existing `fullscreen` HTTP API parameter through the full stack:
- AgentClient.ScreenshotAsync: new `bool fullscreen` parameter
- MCP tool (maui_screenshot): new `fullscreen` parameter with descriptive
  metadata guiding AI agents to use it for edge/safe area verification
- CLI (MAUI screenshot): new `--fullscreen` option
- JSON output includes fullscreen mode indicator

Update skill file (SKILL.md) with safe area guidance:
- New 'Safe Area & Edge Verification' section listing trigger keywords
  (safe area, insets, notch, Dynamic Island, clipping, edge layout, etc.)
- Updated command reference table documenting default vs fullscreen behavior
- Token usage tips mentioning --fullscreen for edge verification
- Added fullscreen example to inspection flow

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 29, 2026 17:25
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR wires the existing /api/screenshot?fullscreen=true agent capability through the DevFlow Driver (AgentClient), MCP tool (maui_screenshot), and CLI (MAUI screenshot) so AI agents and users can capture full-device screenshots for safe-area/edge verification.

Changes:

  • Added fullscreen parameter threading from Driver → CLI/MCP → agent query string.
  • Added CLI --fullscreen option plus JSON/text output labeling.
  • Updated MCP tool metadata and skill guidance to encourage fullscreen usage for edge/safe-area checks.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
src/DevFlow/Microsoft.Maui.DevFlow.Driver/AgentClient.cs Adds fullscreen parameter and sends fullscreen=true query param to the agent screenshot endpoint.
src/DevFlow/Microsoft.Maui.DevFlow.CLI/Program.cs Adds --fullscreen option, threads it into screenshot capture, and includes it in output.
src/DevFlow/Microsoft.Maui.DevFlow.CLI/Mcp/Tools/ScreenshotTool.cs Adds fullscreen tool parameter and returns a mode label in the MCP response.
.claude/skills/maui-ai-debugging/SKILL.md Documents fullscreen usage and adds a “Safe Area & Edge Verification” section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ms, fix mode label whitespace

- MCP tool: reject fullscreen + elementId/selector combination with clear error
- MCP tool: use IsNullOrWhiteSpace for mode label to match agent endpoint behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jfversluis jfversluis added the breaking-change This PR contains breaking changes label Mar 29, 2026
@Redth
Copy link
Copy Markdown
Member

Redth commented Mar 31, 2026

Hmm I think right now this is sort of done as a fallback... So iOS I believe will try and use simctl first, and then start falling back to the agent's implementation if it doesn't work for some reason (or if it's a device).

Maybe android we should just do the same as that? Use ADB first, if there's no specific element id specified and fallback ?

@Redth
Copy link
Copy Markdown
Member

Redth commented Mar 31, 2026

Or I guess maybe we also need to clean up where that series of steps happens.... MCP should in theory do the same thing as the CLI...

@jfversluis
Copy link
Copy Markdown
Member Author

@Redth not sure what the right approach is here. My scenario was: I was building an app on iOS and the statusbar/safe area was not the color I wanted. So I told Copilot: hey I want the color underneath to also continue upwards in the safe area. And it would take a screenshot and be like: the color is already all the way in the top, so nothing to do here!

Probably because it took a partial screenshot, which is good for efficiency, but not if you're looking at edges or safe areas or things :D

So did an attempt to "fix" that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change This PR contains breaking changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants