Conversation
) Extract shared `getVisionEnrichedError` utility and add `VisionAnalyzer` interface for testability, then wire vision fallback into the error paths of SwipeOn, PinchOn, and DragAndDrop. Refactor TapOnElement to use the shared utility as well. Add vision fallback tests for all four actions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MCP BenchmarksOverall Status: ✅ PASSED Context Thresholds
Overall Status: ✅ PASSED Generated at 2026-02-24T04:36:13.885Z Tool Call ThroughputSample Size: 50 iterations per tool Fast Operations (<100ms)
Medium Operations (100ms-1s)
Slow Operations (1s+)
Summary: 7/7 tools passed Generated at 2026-02-24T04:35:12.649Z Startup PerformanceMCP Server (stdio)
Device discovery: skipped (not run) Daemon
Overall Status: ✅ PASSED Generated at 2026-02-24T04:37:14Z NPM Unpacked Size
Package: @kaeawc/auto-mobile@0.0.13 Overall Status: ✅ PASSED Generated at 2026-02-24T04:35:22.907Z |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd6290d176
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const baseErrorMessage = error instanceof Error ? error.message : String(error); | ||
| let errorMessage = `Failed to perform swipeOn: ${baseErrorMessage}`; | ||
|
|
||
| if (this.visionConfig.enabled && (normalizedOptions.lookFor || normalizedOptions.container)) { |
There was a problem hiding this comment.
This catch block now invokes vision fallback for any exception whenever lookFor or container is set, but several thrown paths are infrastructure/runtime failures rather than selector misses (for example ScrollUntilVisible.execute throws Failed to get initial observation for scrolling until visible. in src/features/action/swipeon/ScrollUntilVisible.ts). In those cases we now take an unnecessary screenshot + AI call (cost/latency) and may replace the real root-cause message with generic "Element not found" guidance, which makes production debugging harder.
Useful? React with 👍 / 👎.
| const isSourceError = baseErrorMessage.toLowerCase().includes("source"); | ||
| const failedTarget = isSourceError ? options.source : options.target; |
There was a problem hiding this comment.
Inferring the failed side with baseErrorMessage.toLowerCase().includes("source") is ambiguous and can misclassify target failures whenever the target selector text/ID itself contains the word "source" (e.g. dragAndDrop target not found with text 'Source account'). That sends vision analysis to the wrong element and returns misleading remediation steps.
Useful? React with 👍 / 👎.
kaeawc
deleted the
work/1361-feat--expand-vision-fallback-to-support-
branch
1 participant
Closes #1361
Summary
getVisionEnrichedErrorutility (src/vision/applyVisionFallback.ts) to consolidate the vision fallback logic that was previously inlined only inTapOnElementVisionAnalyzerinterface toVisionTypes.tsto allow dependency injection and keep tests fast and non-flakySwipeOn,PinchOn, andDragAndDrop, in addition to the existingTapOnElementsupportTapOnElementto use the shared utility instead of the previous inline implementationFakeVisionAnalyzertest fake and vision fallback tests for all four action classesTest plan
bun run buildpasses with no errorsbun test test/vision/ test/features/action/passes (333 pass, 0 fail)applyVisionFallback.test.tscovers the shared utility directly🤖 Generated with Claude Code