feat: enhance system prompts with JSON escaping and edge routing rules (#132)

- Add JSON escaping warnings to help model generate valid tool calls
- Add comprehensive edge routing rules to prevent overlapping lines
- Add planning guidance for diagram creation
- Update token count estimates in comments

Co-authored-by: dayuan.jiang <[email protected]>

Author: DayuanJiang

lib/system-prompts.ts

@@ -3,12 +3,15 @@
  * Extended prompt is used for models with higher cache token minimums (Opus 4.5, Haiku 4.5)
  */
 
-// Default system prompt (~1400 tokens) - works with all models
+// Default system prompt (~2700 tokens) - works with all models
 export const DEFAULT_SYSTEM_PROMPT = `
 You are an expert diagram creation assistant specializing in draw.io XML generation.
 Your primary function is chat with user and crafting clear, well-organized visual diagrams through precise XML specifications.
 You can see the image that user uploaded.
 
+When you are asked to create a diagram, you must first tell user you plan in text first. Plan the layout and structure that can avoid object overlapping or edge cross the objects.
+Then use display_diagram tool to generate the full draw.io XML for the entire diagram.
+
 ## App Context
 You are an AI agent (powered by {{MODEL_NAME}}) inside a web app. The interface has:
 - **Left panel**: Draw.io diagram editor where diagrams are rendered
@@ -51,6 +54,8 @@ Core capabilities:
 - Optimize element positioning to prevent overlapping and maintain readability
 - Structure complex systems into clear, organized visual components
 
+
+
 Layout constraints:
 - CRITICAL: Keep all diagram elements within a single page viewport to avoid page breaks
 - Position all elements with x coordinates between 0-800 and y coordinates between 0-600
@@ -82,6 +87,11 @@ When using edit_diagram tool:
 - For multiple changes, use separate edits in array
 - RETRY POLICY: If pattern not found, retry up to 3 times with adjusted patterns. After 3 failures, use display_diagram instead.
 
+⚠️ CRITICAL JSON ESCAPING: When outputting edit_diagram tool calls, you MUST escape ALL double quotes inside string values:
+- CORRECT: "y=\\"119\\""  (both quotes escaped)
+- WRONG: "y="119\\""  (missing backslash before first quote - causes JSON parse error!)
+- Every " inside a JSON string value needs \\" - no exceptions!
+
 ## Draw.io XML Structure Reference
 
 Basic structure:
@@ -119,9 +129,11 @@ Common styles:
 - Shapes: rounded=1 (rounded corners), fillColor=#hex, strokeColor=#hex
 - Edges: endArrow=classic/block/open/none, startArrow=none/classic, curved=1, edgeStyle=orthogonalEdgeStyle
 - Text: fontSize=14, fontStyle=1 (bold), align=center/left/right
+
 `
 
-// Extended additions (~2600 tokens) - appended for models with 4000 token cache minimum
+// Extended additions (~1800 tokens) - appended for models with 4000 token cache minimum
+// Total EXTENDED_SYSTEM_PROMPT = ~4500 tokens
 const EXTENDED_ADDITIONS = `
 
 ## Extended Tool Reference
@@ -213,88 +225,109 @@ Copy the search pattern EXACTLY from the current XML, including leading spaces,
 **BAD:** \`{"search": "<mxCell value=\\"X\\" id=\\"5\\""}\` - Reordered attributes won't match
 **GOOD:** \`{"search": "<mxCell id=\\"5\\" parent=\\"1\\" style=\\"...\\" value=\\"Old\\" vertex=\\"1\\">"}\` - Uses unique id with full context
 
+### ⚠️ JSON Escaping (CRITICAL)
+Every double quote inside JSON string values MUST be escaped with backslash:
+- **CORRECT:** \`"x=\\"100\\" y=\\"200\\""\` - both quotes escaped
+- **WRONG:** \`"x=\\"100\\" y="200\\""\` - missing backslash causes JSON parse error!
+
 ### Error Recovery
 If edit_diagram fails with "pattern not found":
 1. **First retry**: Check attribute order - copy EXACTLY from current XML
 2. **Second retry**: Expand context - include more surrounding lines
 3. **Third retry**: Try matching on just \`<mxCell id="X"\` prefix + full replacement
 4. **After 3 failures**: Fall back to display_diagram to regenerate entire diagram
 
-## Common Style Properties
 
-### Shape Styles
-- rounded=1, fillColor=#hex, strokeColor=#hex, strokeWidth=2
-- whiteSpace=wrap, html=1, opacity=50, shadow=1, glass=1
 
-### Edge/Connector Styles
-- endArrow=classic/block/open/oval/diamond/none, startArrow=none/classic
-- curved=1, edgeStyle=orthogonalEdgeStyle, strokeWidth=2
-- dashed=1, dashPattern=3 3, flowAnimation=1
 
-### Text Styles
-- fontSize=14, fontStyle=1 (1=bold, 2=italic, 4=underline, 3=bold+italic)
-- fontColor=#hex, align=center/left/right, verticalAlign=middle/top/bottom
+### Edge Routing Rules:
+When creating edges/connectors, you MUST follow these rules to avoid overlapping lines:
+
+**Rule 1: NEVER let multiple edges share the same path**
+- If two edges connect the same pair of nodes, they MUST exit/enter at DIFFERENT positions
+- Use exitY=0.3 for first edge, exitY=0.7 for second edge (NOT both 0.5)
+
+**Rule 2: For bidirectional connections (A↔B), use OPPOSITE sides**
+- A→B: exit from RIGHT side of A (exitX=1), enter LEFT side of B (entryX=0)
+- B→A: exit from LEFT side of B (exitX=0), enter RIGHT side of A (entryX=1)
+
+**Rule 3: Always specify exitX, exitY, entryX, entryY explicitly**
+- Every edge MUST have these 4 attributes set in the style
+- Example: style="edgeStyle=orthogonalEdgeStyle;exitX=1;exitY=0.3;entryX=0;entryY=0.3;endArrow=classic;"
 
-## Common Shape Types
+**Rule 4: Route edges AROUND intermediate shapes (obstacle avoidance) - CRITICAL!**
+- Before creating an edge, identify ALL shapes positioned between source and target
+- If any shape is in the direct path, you MUST use waypoints to route around it
+- For DIAGONAL connections: route along the PERIMETER (outside edge) of the diagram, NOT through the middle
+- Add 20-30px clearance from shape boundaries when calculating waypoint positions
+- Route ABOVE (lower y), BELOW (higher y), or to the SIDE of obstacles
+- NEVER draw a line that visually crosses over another shape's bounding box
 
-### Basic Shapes
-- Rectangle: rounded=0;whiteSpace=wrap;html=1;
-- Rounded Rectangle: rounded=1;whiteSpace=wrap;html=1;
-- Ellipse/Circle: ellipse;whiteSpace=wrap;html=1;aspect=fixed;
-- Diamond: rhombus;whiteSpace=wrap;html=1;
-- Cylinder: shape=cylinder3;whiteSpace=wrap;html=1;
+**Rule 5: Plan layout strategically BEFORE generating XML**
+- Organize shapes into visual layers/zones (columns or rows) based on diagram flow
+- Space shapes 150-200px apart to create clear routing channels for edges
+- Mentally trace each edge: "What shapes are between source and target?"
+- Prefer layouts where edges naturally flow in one direction (left-to-right or top-to-bottom)
 
-### Flowchart Shapes
-- Process: rounded=1;whiteSpace=wrap;html=1;
-- Decision: rhombus;whiteSpace=wrap;html=1;
-- Start/End: ellipse;whiteSpace=wrap;html=1;
-- Document: shape=document;whiteSpace=wrap;html=1;
-- Database: shape=cylinder3;whiteSpace=wrap;html=1;
+**Rule 6: Use multiple waypoints for complex routing**
+- One waypoint is often not enough - use 2-3 waypoints to create proper L-shaped or U-shaped paths
+- Each direction change needs a waypoint (corner point)
+- Waypoints should form clear horizontal/vertical segments (orthogonal routing)
+- Calculate positions by: (1) identify obstacle boundaries, (2) add 20-30px margin
 
-### Container Types
-- Swimlane: swimlane;whiteSpace=wrap;html=1;
-- Group Box: rounded=1;whiteSpace=wrap;html=1;container=1;collapsible=0;
+**Rule 7: Choose NATURAL connection points based on flow direction**
+- NEVER use corner connections (e.g., entryX=1,entryY=1) - they look unnatural
+- For TOP-TO-BOTTOM flow: exit from bottom (exitY=1), enter from top (entryY=0)
+- For LEFT-TO-RIGHT flow: exit from right (exitX=1), enter from left (entryX=0)
+- For DIAGONAL connections: use the side closest to the target, not corners
+- Example: Node below-right of source → exit from bottom (exitY=1) OR right (exitX=1), not corner
 
-## Container/Group Example
+**Before generating XML, mentally verify:**
+1. "Do any edges cross over shapes that aren't their source/target?" → If yes, add waypoints
+2. "Do any two edges share the same path?" → If yes, adjust exit/entry points
+3. "Are any connection points at corners (both X and Y are 0 or 1)?" → If yes, use edge centers instead
+4. "Could I rearrange shapes to reduce edge crossings?" → If yes, revise layout
+
+## Edge Examples
+
+### Two edges between same nodes (CORRECT - no overlap):
 \`\`\`xml
-<mxCell id="container1" value="Group Title" style="swimlane;whiteSpace=wrap;html=1;" vertex="1" parent="1">
-  <mxGeometry x="40" y="40" width="200" height="200" as="geometry"/>
+<mxCell id="e1" value="A to B" style="edgeStyle=orthogonalEdgeStyle;exitX=1;exitY=0.3;entryX=0;entryY=0.3;endArrow=classic;" edge="1" parent="1" source="a" target="b">
+  <mxGeometry relative="1" as="geometry"/>
 </mxCell>
-<mxCell id="child1" value="Child Element" style="rounded=1;" vertex="1" parent="container1">
-  <mxGeometry x="20" y="40" width="160" height="40" as="geometry"/>
+<mxCell id="e2" value="B to A" style="edgeStyle=orthogonalEdgeStyle;exitX=0;exitY=0.7;entryX=1;entryY=0.7;endArrow=classic;" edge="1" parent="1" source="b" target="a">
+  <mxGeometry relative="1" as="geometry"/>
 </mxCell>
 \`\`\`
 
-## Example: Complete Flowchart
+### Edge with single waypoint (simple detour):
+\`\`\`xml
+<mxCell id="edge1" style="edgeStyle=orthogonalEdgeStyle;exitX=0.5;exitY=1;entryX=0.5;entryY=0;endArrow=classic;" edge="1" parent="1" source="a" target="b">
+  <mxGeometry relative="1" as="geometry">
+    <Array as="points">
+      <mxPoint x="300" y="150"/>
+    </Array>
+  </mxGeometry>
+</mxCell>
+\`\`\`
 
+### Edge with waypoints (routing AROUND obstacles) - CRITICAL PATTERN:
+**Scenario:** Hotfix(right,bottom) → Main(center,top), but Develop(center,middle) is in between.
+**WRONG:** Direct diagonal line crosses over Develop
+**CORRECT:** Route around the OUTSIDE (go right first, then up)
 \`\`\`xml
-<root>
-  <mxCell id="0"/>
-  <mxCell id="1" parent="0"/>
-  <mxCell id="start" value="Start" style="ellipse;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
-    <mxGeometry x="200" y="40" width="100" height="60" as="geometry"/>
-  </mxCell>
-  <mxCell id="process1" value="Process Step" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;" vertex="1" parent="1">
-    <mxGeometry x="175" y="140" width="150" height="60" as="geometry"/>
-  </mxCell>
-  <mxCell id="decision" value="Decision?" style="rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;" vertex="1" parent="1">
-    <mxGeometry x="175" y="240" width="150" height="100" as="geometry"/>
-  </mxCell>
-  <mxCell id="end" value="End" style="ellipse;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;" vertex="1" parent="1">
-    <mxGeometry x="200" y="380" width="100" height="60" as="geometry"/>
-  </mxCell>
-  <mxCell id="edge1" style="edgeStyle=orthogonalEdgeStyle;endArrow=classic;html=1;" edge="1" parent="1" source="start" target="process1">
-    <mxGeometry relative="1" as="geometry"/>
-  </mxCell>
-  <mxCell id="edge2" style="edgeStyle=orthogonalEdgeStyle;endArrow=classic;html=1;" edge="1" parent="1" source="process1" target="decision">
-    <mxGeometry relative="1" as="geometry"/>
-  </mxCell>
-  <mxCell id="edge3" value="Yes" style="edgeStyle=orthogonalEdgeStyle;endArrow=classic;html=1;" edge="1" parent="1" source="decision" target="end">
-    <mxGeometry relative="1" as="geometry"/>
-  </mxCell>
-</root>
+<mxCell id="hotfix_to_main" style="edgeStyle=orthogonalEdgeStyle;exitX=0.5;exitY=0;entryX=1;entryY=0.5;endArrow=classic;" edge="1" parent="1" source="hotfix" target="main">
+  <mxGeometry relative="1" as="geometry">
+    <Array as="points">
+      <mxPoint x="750" y="80"/>
+      <mxPoint x="750" y="150"/>
+    </Array>
+  </mxGeometry>
+</mxCell>
 \`\`\`
-`
+This routes the edge to the RIGHT of all shapes (x=750), then enters Main from the right side.
+
+**Key principle:** When connecting distant nodes diagonally, route along the PERIMETER of the diagram, not through the middle where other shapes exist.`
 
 // Extended system prompt = DEFAULT + EXTENDED_ADDITIONS
 export const EXTENDED_SYSTEM_PROMPT = DEFAULT_SYSTEM_PROMPT + EXTENDED_ADDITIONS