|
| 1 | +# Event Stream API |
| 2 | + |
| 3 | +Real-time monitoring endpoint for the Web4 Governance plugin. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The event stream provides a JSONL (JSON Lines) file that external clients can tail for real-time monitoring, alerting, and analytics. |
| 8 | + |
| 9 | +**Stream Location**: `~/.web4/events.jsonl` |
| 10 | + |
| 11 | +## Quick Start |
| 12 | + |
| 13 | +### Tail the stream (real-time) |
| 14 | +```bash |
| 15 | +tail -f ~/.web4/events.jsonl | jq . |
| 16 | +``` |
| 17 | + |
| 18 | +### Filter by severity |
| 19 | +```bash |
| 20 | +tail -f ~/.web4/events.jsonl | jq -c 'select(.severity == "alert")' |
| 21 | +``` |
| 22 | + |
| 23 | +### Filter by event type |
| 24 | +```bash |
| 25 | +grep '"type":"policy_decision"' ~/.web4/events.jsonl | jq . |
| 26 | +``` |
| 27 | + |
| 28 | +### Python consumer |
| 29 | +```python |
| 30 | +import json |
| 31 | + |
| 32 | +with open("~/.web4/events.jsonl", "r") as f: |
| 33 | + for line in f: |
| 34 | + event = json.loads(line) |
| 35 | + if event.get("severity") == "alert": |
| 36 | + print(f"ALERT: {event.get('reason')}") |
| 37 | +``` |
| 38 | + |
| 39 | +--- |
| 40 | + |
| 41 | +## Event Schema |
| 42 | + |
| 43 | +Each line in the stream is a self-contained JSON object: |
| 44 | + |
| 45 | +```json |
| 46 | +{ |
| 47 | + "type": "policy_decision", |
| 48 | + "timestamp": "2026-02-05T10:30:00.123456+00:00", |
| 49 | + "severity": "alert", |
| 50 | + "session_id": "sess-abc123", |
| 51 | + "tool": "Bash", |
| 52 | + "target": "rm -rf /tmp/test", |
| 53 | + "category": "command", |
| 54 | + "decision": "deny", |
| 55 | + "reason": "Destructive command blocked by safety preset", |
| 56 | + "rule_id": "deny-destructive-commands" |
| 57 | +} |
| 58 | +``` |
| 59 | + |
| 60 | +### Required Fields |
| 61 | + |
| 62 | +| Field | Type | Description | |
| 63 | +|-------|------|-------------| |
| 64 | +| `type` | string | Event type (see Event Types below) | |
| 65 | +| `timestamp` | string | ISO 8601 UTC timestamp | |
| 66 | +| `severity` | string | Severity level: `debug`, `info`, `warn`, `alert`, `error` | |
| 67 | + |
| 68 | +### Optional Context Fields |
| 69 | + |
| 70 | +| Field | Type | Description | |
| 71 | +|-------|------|-------------| |
| 72 | +| `session_id` | string | Session identifier | |
| 73 | +| `agent_id` | string | Agent/role identifier | |
| 74 | + |
| 75 | +### Event-Specific Fields |
| 76 | + |
| 77 | +| Field | Type | Description | |
| 78 | +|-------|------|-------------| |
| 79 | +| `tool` | string | Tool name (Bash, Read, Edit, etc.) | |
| 80 | +| `target` | string | Target path, URL, or command | |
| 81 | +| `category` | string | Tool category (file_read, file_write, command, network, etc.) | |
| 82 | +| `decision` | string | Policy decision: `allow`, `deny`, `warn` | |
| 83 | +| `reason` | string | Human-readable explanation | |
| 84 | +| `rule_id` | string | ID of matched policy rule | |
| 85 | + |
| 86 | +### Metrics Fields |
| 87 | + |
| 88 | +| Field | Type | Description | |
| 89 | +|-------|------|-------------| |
| 90 | +| `duration_ms` | integer | Operation duration in milliseconds | |
| 91 | +| `count` | integer | Generic count (rate limits, etc.) | |
| 92 | + |
| 93 | +### Trust Fields |
| 94 | + |
| 95 | +| Field | Type | Description | |
| 96 | +|-------|------|-------------| |
| 97 | +| `trust_before` | float | Trust value before update (0.0-1.0) | |
| 98 | +| `trust_after` | float | Trust value after update (0.0-1.0) | |
| 99 | +| `trust_delta` | float | Change in trust value | |
| 100 | + |
| 101 | +### Error Fields |
| 102 | + |
| 103 | +| Field | Type | Description | |
| 104 | +|-------|------|-------------| |
| 105 | +| `error` | string | Error message | |
| 106 | +| `error_type` | string | Error class/type | |
| 107 | + |
| 108 | +### Extensible Metadata |
| 109 | + |
| 110 | +| Field | Type | Description | |
| 111 | +|-------|------|-------------| |
| 112 | +| `metadata` | object | Additional key-value data | |
| 113 | + |
| 114 | +--- |
| 115 | + |
| 116 | +## Event Types |
| 117 | + |
| 118 | +### Session Lifecycle |
| 119 | + |
| 120 | +| Type | Severity | Description | |
| 121 | +|------|----------|-------------| |
| 122 | +| `session_start` | info | New session started | |
| 123 | +| `session_end` | info | Session ended | |
| 124 | + |
| 125 | +**Example:** |
| 126 | +```json |
| 127 | +{"type":"session_start","timestamp":"2026-02-05T10:00:00Z","severity":"info","session_id":"sess-abc123","metadata":{"project":"my-app","atp_budget":100}} |
| 128 | +``` |
| 129 | + |
| 130 | +### Tool Execution |
| 131 | + |
| 132 | +| Type | Severity | Description | |
| 133 | +|------|----------|-------------| |
| 134 | +| `tool_call` | info | Tool invocation started | |
| 135 | +| `tool_result` | info | Tool completed | |
| 136 | + |
| 137 | +**Example:** |
| 138 | +```json |
| 139 | +{"type":"tool_call","timestamp":"2026-02-05T10:01:00Z","severity":"info","session_id":"sess-abc123","tool":"Read","target":"/app/src/main.py","category":"file_read"} |
| 140 | +``` |
| 141 | + |
| 142 | +### Policy Decisions |
| 143 | + |
| 144 | +| Type | Severity | Description | |
| 145 | +|------|----------|-------------| |
| 146 | +| `policy_decision` | varies | Policy evaluated (info=allow, warn=warn, alert=deny) | |
| 147 | +| `policy_violation` | alert | Policy rule violated | |
| 148 | + |
| 149 | +**Example (deny):** |
| 150 | +```json |
| 151 | +{"type":"policy_decision","timestamp":"2026-02-05T10:02:00Z","severity":"alert","session_id":"sess-abc123","tool":"Bash","target":"rm -rf /","decision":"deny","reason":"Destructive command blocked by safety preset","rule_id":"deny-destructive-commands"} |
| 152 | +``` |
| 153 | + |
| 154 | +**Example (warn):** |
| 155 | +```json |
| 156 | +{"type":"policy_decision","timestamp":"2026-02-05T10:03:00Z","severity":"warn","session_id":"sess-abc123","tool":"Bash","target":"rm temp.txt","decision":"warn","reason":"File deletion flagged - use with caution","rule_id":"warn-file-delete"} |
| 157 | +``` |
| 158 | + |
| 159 | +### Rate Limiting |
| 160 | + |
| 161 | +| Type | Severity | Description | |
| 162 | +|------|----------|-------------| |
| 163 | +| `rate_limit_check` | debug | Rate limit checked | |
| 164 | +| `rate_limit_exceeded` | alert | Rate limit exceeded | |
| 165 | + |
| 166 | +**Example:** |
| 167 | +```json |
| 168 | +{"type":"rate_limit_exceeded","timestamp":"2026-02-05T10:04:00Z","severity":"alert","session_id":"sess-abc123","target":"ratelimit:bash:tool","count":6,"metadata":{"max_count":5}} |
| 169 | +``` |
| 170 | + |
| 171 | +### Trust Updates |
| 172 | + |
| 173 | +| Type | Severity | Description | |
| 174 | +|------|----------|-------------| |
| 175 | +| `trust_update` | info | Agent trust level changed | |
| 176 | + |
| 177 | +**Example:** |
| 178 | +```json |
| 179 | +{"type":"trust_update","timestamp":"2026-02-05T10:05:00Z","severity":"info","session_id":"sess-abc123","agent_id":"code-reviewer","trust_before":0.5,"trust_after":0.55,"trust_delta":0.05,"reason":"Successful code review"} |
| 180 | +``` |
| 181 | + |
| 182 | +### Agent Lifecycle |
| 183 | + |
| 184 | +| Type | Severity | Description | |
| 185 | +|------|----------|-------------| |
| 186 | +| `agent_spawn` | info | Agent spawned | |
| 187 | +| `agent_complete` | info | Agent completed | |
| 188 | + |
| 189 | +**Example:** |
| 190 | +```json |
| 191 | +{"type":"agent_spawn","timestamp":"2026-02-05T10:06:00Z","severity":"info","session_id":"sess-abc123","agent_id":"test-runner","metadata":{"capabilities":{"can_write":true,"can_execute":true}}} |
| 192 | +``` |
| 193 | + |
| 194 | +### Audit Events |
| 195 | + |
| 196 | +| Type | Severity | Description | |
| 197 | +|------|----------|-------------| |
| 198 | +| `audit_record` | info | Standard audit record | |
| 199 | +| `audit_alert` | alert | High-priority audit event (credential access, etc.) | |
| 200 | + |
| 201 | +**Example (credential access alert):** |
| 202 | +```json |
| 203 | +{"type":"audit_alert","timestamp":"2026-02-05T10:07:00Z","severity":"alert","session_id":"sess-abc123","tool":"Read","target":"/home/user/.aws/credentials","category":"credential_access","reason":"Credential file access detected"} |
| 204 | +``` |
| 205 | + |
| 206 | +### System Events |
| 207 | + |
| 208 | +| Type | Severity | Description | |
| 209 | +|------|----------|-------------| |
| 210 | +| `system_info` | info | System information | |
| 211 | +| `system_error` | error | System error | |
| 212 | + |
| 213 | +**Example:** |
| 214 | +```json |
| 215 | +{"type":"system_error","timestamp":"2026-02-05T10:08:00Z","severity":"error","error":"Database connection failed","error_type":"sqlite3.OperationalError"} |
| 216 | +``` |
| 217 | + |
| 218 | +--- |
| 219 | + |
| 220 | +## Severity Levels |
| 221 | + |
| 222 | +| Level | When Used | Action | |
| 223 | +|-------|-----------|--------| |
| 224 | +| `debug` | Verbose debugging | Usually filtered | |
| 225 | +| `info` | Normal operations | Log/monitor | |
| 226 | +| `warn` | Potential issues | Review | |
| 227 | +| `alert` | Security events, policy violations | Immediate attention | |
| 228 | +| `error` | System errors | Investigate | |
| 229 | + |
| 230 | +--- |
| 231 | + |
| 232 | +## File Rotation |
| 233 | + |
| 234 | +The stream file automatically rotates at 100MB: |
| 235 | +- Current: `~/.web4/events.jsonl` |
| 236 | +- Rotated: `~/.web4/events.jsonl.1` |
| 237 | + |
| 238 | +Only one backup is kept. For long-term retention, configure an external log collector. |
| 239 | + |
| 240 | +--- |
| 241 | + |
| 242 | +## Integration Examples |
| 243 | + |
| 244 | +### Forward to syslog |
| 245 | +```bash |
| 246 | +tail -f ~/.web4/events.jsonl | while read line; do |
| 247 | + logger -t web4-governance "$line" |
| 248 | +done |
| 249 | +``` |
| 250 | + |
| 251 | +### Send alerts to Slack |
| 252 | +```python |
| 253 | +import json |
| 254 | +import requests |
| 255 | + |
| 256 | +WEBHOOK_URL = "https://hooks.slack.com/services/..." |
| 257 | + |
| 258 | +with open("~/.web4/events.jsonl", "r") as f: |
| 259 | + f.seek(0, 2) # Seek to end |
| 260 | + while True: |
| 261 | + line = f.readline() |
| 262 | + if line: |
| 263 | + event = json.loads(line) |
| 264 | + if event.get("severity") == "alert": |
| 265 | + requests.post(WEBHOOK_URL, json={ |
| 266 | + "text": f":warning: {event.get('type')}: {event.get('reason')}" |
| 267 | + }) |
| 268 | +``` |
| 269 | + |
| 270 | +### Prometheus metrics (conceptual) |
| 271 | +```python |
| 272 | +from prometheus_client import Counter |
| 273 | + |
| 274 | +policy_decisions = Counter('web4_policy_decisions', 'Policy decisions', ['decision']) |
| 275 | + |
| 276 | +# In your event consumer: |
| 277 | +if event["type"] == "policy_decision": |
| 278 | + policy_decisions.labels(decision=event["decision"]).inc() |
| 279 | +``` |
| 280 | + |
| 281 | +### Structured logging (JSON to stdout) |
| 282 | +```bash |
| 283 | +tail -f ~/.web4/events.jsonl | jq -c '{ |
| 284 | + time: .timestamp, |
| 285 | + level: .severity, |
| 286 | + msg: .reason // .type, |
| 287 | + tool: .tool, |
| 288 | + session: .session_id |
| 289 | +}' |
| 290 | +``` |
| 291 | + |
| 292 | +--- |
| 293 | + |
| 294 | +## Configuration |
| 295 | + |
| 296 | +The event stream can be configured when initializing: |
| 297 | + |
| 298 | +```python |
| 299 | +from governance.event_stream import EventStream, Severity |
| 300 | + |
| 301 | +# Custom location and minimum severity |
| 302 | +stream = EventStream( |
| 303 | + storage_path="~/.my-app/governance", |
| 304 | + filename="audit-events.jsonl", |
| 305 | + min_severity=Severity.WARN # Only emit WARN and above |
| 306 | +) |
| 307 | +``` |
| 308 | + |
| 309 | +--- |
| 310 | + |
| 311 | +## Best Practices |
| 312 | + |
| 313 | +1. **Use `tail -f`** for real-time monitoring rather than polling |
| 314 | +2. **Filter by severity** to reduce noise (`alert` for critical events) |
| 315 | +3. **Use `jq`** for ad-hoc queries and formatting |
| 316 | +4. **Configure external log rotation** for long-term retention |
| 317 | +5. **Register callbacks** for in-process alerting (low latency) |
| 318 | + |
| 319 | +--- |
| 320 | + |
| 321 | +## Version History |
| 322 | + |
| 323 | +| Version | Changes | |
| 324 | +|---------|---------| |
| 325 | +| 0.4.0 | Initial event stream API | |
0 commit comments