UI Navigation Fine-tuning

[0bi0n3]

Is there a recommended script or workflow for applying the SoM preprocessing (e.g.get_som_labeled_img) to datasets such as Mind2Web or AITW?

Could you clarify the expected format for the annotation file (e.g., JSON's "conversations" list) when the target output is an SoM-based action (like a Mark ID) for the UI navigation tasks?

Are there specific modifications needed for the fine-tuning script (e.g., finetune_magma_820k.sh) or config files when fine-tuning for SoM-based UI action prediction compared to instruction following in Magma-820K?

Any clarification you could offer would be greatly appreciated and would significantly help in reproducing your UI navigation results.

[junho328]

Same question.
How to construct dataset for AITW downstream task?

[jwyang]

@0bi0n3 @junho328 , thanks for your interests. Will release the finetuning data for AITW and Mind2web shorty this week.

[junho328]

I have one more question: could you also explain how the SeeClick and Vision2UI datasets are structured? As always, thank you for your amazing research.

[jayabrata97]

@0bi0n3 @junho328 , thanks for your interests. Will release the finetuning data for AITW and Mind2web shorty this week.

Hi, thank you for the wonderful work. Is the dataset out?

[jwyang]

The finetuning dataset is out for Mind2Web and AITW on huggingface! Check it out here:

Mind2Web
AITW

Will upload the evaluation scripts for both very soon!

[jwyang]

@0bi0n3 @jayabrata97 @junho328 , let me know if you have any questions about this data!

[jayabrata97]

@jwyang, thank you very much for sharing the links for the dataset. I have one query: Did you use tools/som_tom/demo.py script to generate the SOM?

[jwyang]

Good question, it was a separate file, I will upload the sample code today.

…

On Tue, Apr 29, 2025 at 1:48 AM Jayabrata (Joy) ***@***.***> wrote: @jwyang <https://github.com/jwyang>, thank you very much for sharing the links for the dataset. I have one query: Did you use tools/som_tom/demo.py scripts to generate the SOM? — Reply to this email directly, view it on GitHub <#64 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5WXZ5HTBCJKNKPVNLGGUT2344FRBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOJIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVE4DSMRYGM3DGNZVQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSHE4TGNRZG42TAOFHORZGSZ3HMVZKMY3SMVQXIZI> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[jwyang]

@jayabrata97, I just recalled that actually you can refer to this line to generate SoM for UI Screenshot:

Magma/agents/ui_agent/app.py

Line 170 in 3fcfa8e

dino_labled_img, label_coordinates, parsed_content_list = get_som_labeled_img(image_input, yolo_model, BOX_TRESHOLD = box_threshold, output_coord_in_ratio=False, ocr_bbox=ocr_bbox,draw_bbox_config=draw_bbox_config, caption_model_processor=caption_model_processor, ocr_text=text,iou_threshold=iou_threshold, imgsz=imgsz,)