Auto sync permissions ACL support for Jira + Confluence knowledge connectors

[joeyorlando]

Feature Request

Add support for the Auto Sync Permissions visibility mode on knowledge bases, enabling automatic synchronization of access control lists (ACLs) from source systems (connectors) into Archestra knowledge bases.

Background

Knowledge bases currently support two visibility modes:

• Org-wide — all documents accessible to all users in the organization
• Team-scoped — documents accessible only to members of assigned teams

There would be a third Auto Sync Permissions option added here.

Requirements

• To start, let's implement this for Jira and Confluence. These connectors should extract permission/ACL data from these systems during sync.
• Currently there is a ARCHESTRA_ENTERPRISE_LICENSE_KNOWLEDGE_BASE_ACTIVATED, which was modified in chore: knowledge base - make advanced-access control an enterprise feature #3783 - this PR should build on top of that (basically RAG should be available as open-source, while only this new access control feature will be behind this ARCHESTRA_ENTERPRISE_LICENSE_KNOWLEDGE_BASE_ACTIVATED flag)
• When a knowledge connector uses auto-sync-permissions visibility, the query_knowledge_sources tool must filter chunks based on the querying user's identity and which documents they are allowed to see in the upstream system
• Should be built in such a way that adding this capability to more knowledge connector types in the future won't require a lot of duplication/refactoring (consider adding this as part of BaseConnector)
• Need to consider how this will work should the access control config change in the "upstream" source system
• Document this new functionality + ensure there is adequate tests in place

As for how the "user identity mapping" (ie. matching users between Jira/Confluence and Archestra) will work, is up for research/suggestions (ex. team based, user based, or a mix of both) 🙂

Definition of Done

Demo video showcasing that the feature works as described above.

[London-Cat]

📊 Reputation Summary

User	Rep	Pull Requests	Activity	Assigned	Core Reactions
joeyorlando	⚡ 2299	92✅ 3🔄 5❌	100 issues, 50 comments	4	—
Dev0907	⚡ 3	0✅ 1🔄 0❌	0 issues, 3 comments	0	—

---

How is the score calculated? Read about it in the Reputation Bot repository 🤖

[Dev0907]

This is a solid feature and aligns well with how enterprise knowledge systems handle access control. I’d like to propose a concrete implementation approach to make this scalable and connector-agnostic.

---

High-Level Architecture

Split the system into three clear layers:

1. Connector-level ACL extraction
2. Normalized ACL storage in KB
3. Query-time enforcement layer

---

1. Connector-Level ACL Extraction

Each connector should extend its sync pipeline to emit ACL metadata alongside documents.

Proposed interface addition:

interface SyncedDocument {
  id: string;
  content: string;
  metadata: Record<string, any>;
  acl?: {
    users?: string[];   // emails
    groups?: string[];  // team/org identifiers
    isPublic?: boolean;
  };
}

Connector-specific mapping:

• Jira → project roles, issue-level security
• Confluence → space permissions, page restrictions
• GitHub → repo visibility + team access
• GitLab → project/group membership

---

2. Normalized ACL Storage

Store ACLs at document (or chunk) level in the knowledge base.

Suggested schema:

{
  documentId: string;
  acl: {
    users: string[];
    groups: string[];
    isPublic: boolean;
    lastSyncedAt: string;
  };
}

Key considerations:

• Incremental updates using lastSyncedAt
• Overwrite ACLs only when source last_edited_time changes
• Keep ACL storage decoupled from embeddings for flexibility

---

3. Identity Mapping Layer

Before query execution:

• Resolve user identity:

  {
    email: string;
    groups: string[];
  }

• Map:

  • Archestra user → email
  • External systems → same email (primary key)
  • Groups via:
    • cached memberships OR
    • connector-level fetch (with caching)

---

4. Query-Time Filtering (Critical)

Modify query_knowledge_base:

results.filter(doc => {
  if (doc.acl?.isPublic) return true;

  if (doc.acl?.users?.includes(user.email)) return true;

  if (doc.acl?.groups?.some(g => user.groups.includes(g))) return true;

  return false;
});

Optimization:

• Apply filtering before final ranking if possible
• Or enforce strict post-filter before returning results

---

5. Incremental Sync Strategy

• Track per-connector checkpoint
• Only re-fetch ACLs when:
  • document updated
  • permissions changed (if detectable)

---

6. Edge Cases

• Private docs with no explicit ACL → treat as restricted
• Missing group resolution → fallback to user-only matching
• Large orgs → require caching layer for group memberships
• Revoked access → must remove ACL entries on next sync

---

7. Security Considerations

• Never rely only on frontend filtering
• Enforce ACL strictly at backend query layer
• Avoid leaking metadata of restricted documents

---

8. Suggested Rollout

1. Start with GitHub connector (simplest ACL model)
2. Add schema + query filtering
3. Extend to Jira/Confluence (more complex ACLs)

---

This design keeps connectors loosely coupled while ensuring consistent enforcement at query time. Happy to take a first pass at implementing the base ACL pipeline + GitHub connector support if aligned.

[Dev0907]

I’d like to take a more implementation-driven approach to this and propose a concrete backend design that fits cleanly into the existing connector + RAG pipeline.

---

Core Design Principle

ACL enforcement should be fully backend-driven and applied before result exposure, not as a UI concern. Connectors should only emit ACLs, while the knowledge base enforces them.

---

1. Connector Contract Extension

Extend the existing connector sync output to include ACL metadata in a normalized format:

interface ConnectorDocument {
  id: string;
  content: string;
  metadata: Record<string, any>;
  acl?: {
    users: string[];     // normalized emails
    groups: string[];    // connector-specific group identifiers
    visibility?: "public" | "restricted";
  };
  lastModifiedAt: string;
}

Important:

• Normalize emails to lowercase early
• Avoid connector-specific ACL formats leaking beyond this layer

---

2. Ingestion Pipeline Changes

During sync:

• Extract ACLs per document
• Attach ACL to document BEFORE chunking
• Propagate ACL to all derived chunks

chunk.acl = parentDocument.acl;

This avoids recomputing ACL at query time per chunk.

---

3. Storage Model

ACL should be stored alongside chunks (not only documents) for efficient filtering.

Example (vector DB / document store):

{
  chunkId: string;
  content: string;
  embedding: number[];
  acl: {
    users: string[];
    groups: string[];
    visibility: "public" | "restricted";
  };
}

Optional optimization:

• Store users and groups as hashed sets for faster lookup

---

4. Identity Resolution Layer

Before querying:

interface RequestContext {
  email: string;
  groups: string[];
}

Resolution strategy:

• Email → primary identifier (must match connector data)
• Groups:
  • cached per user
  • optionally refreshed periodically

---

5. Query Pipeline Modification

Modify query_knowledge_base flow:

Current (simplified):

query → vector search → results → return

Proposed:

query → vector search → ACL filter → rank → return

Filtering logic:

function hasAccess(docAcl, user) {
  if (!docAcl) return false;

  if (docAcl.visibility === "public") return true;

  if (docAcl.users?.includes(user.email)) return true;

  if (docAcl.groups?.some(g => user.groups.includes(g))) return true;

  return false;
}

Critical:

• Filtering must happen before final response
• Prefer pushing ACL filtering into DB query if supported (e.g., metadata filters in vector DB)

---

6. Incremental Sync + Consistency

• Use lastModifiedAt to detect document changes
• Recompute ACL only when document or permissions change
• Ensure revoked access removes users/groups on next sync

---

7. Connector-Specific Notes

• GitHub: repo visibility + team slugs → easiest starting point
• Jira/Confluence: more complex (roles, inheritance)
• Normalize all into flat users/groups

---

8. Edge Cases

• Missing ACL → treat as restricted (fail-safe)
• Large group memberships → require caching layer
• Partial sync failures → do not expose stale ACLs

---

9. Security Guarantees

• No ACL enforcement in frontend
• No leakage of restricted metadata (titles/snippets)
• Strict deny-by-default policy

---

Suggested Incremental Rollout

1. Add ACL schema + query filtering
2. Implement GitHub connector ACL extraction
3. Expand to Jira/Confluence with richer mappings

---

I can start by implementing the base ACL pipeline (schema + query filtering) and wire it with the GitHub connector as a reference implementation.