Skip to content

Fixes branch#146

Merged
MasumRab merged 146 commits intomainfrom
fixes-branch
Oct 15, 2025
Merged

Fixes branch#146
MasumRab merged 146 commits intomainfrom
fixes-branch

Conversation

@MasumRab
Copy link
Copy Markdown
Owner

@MasumRab MasumRab commented Oct 14, 2025

Summary by CodeRabbit

  • New Features
    • Introduces a node-based workflow engine with pluggable email-processing nodes, security/audit/resource controls, and persistence.
    • Adds APIs to create, manage, execute, and monitor workflows, plus versioned v1 email/category endpoints.
    • Provides a Gradio-based workflow editor UI and enhanced analysis UI.
    • Adds background model training endpoints and simplified model discovery/management.
    • Upgrades NLP analysis with new pipelines and fallbacks; adds a JSON-backed data store.
  • Documentation
    • Major rewrites of README and architecture; new advanced workflow, deployment, and guidance docs.
  • Tests
    • Extensive integration, security, migration, and route tests.
  • Chores
    • Expanded ignore rules, LFS for model assets, CI “[skip ci]” guard, increased LLM token limits.

# Conflicts:
#	backend/python_backend/gradio_app.py
#	requirements.txt
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We failed to fetch the diff for pull request #146

You can try again by commenting this pull request with @sourcery-ai review, or contact us for help.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Oct 14, 2025

Caution

Review failed

The pull request is closed.

Walkthrough

Adds a full node-based workflow system (engine, nodes, security, persistence, library), FastAPI routes for legacy and node-based workflows, services layer (email/category), Pydantic v2 model updates, JSON DB, NLP pipeline refactor, Gmail service/retrieval rework, training endpoints/tests, Gradio UI overhaul, CI/test updates, configuration/docs expansion, and LFS for safetensors.

Changes

Cohort / File(s) Summary
Git & CI config
/.gitattributes, /.gitignore, /.github/workflows/ci.yml
LFS tracking for .safetensors; expanded ignores; CI skip guard and broadened pytest/coverage targets.
Node workflow core
backend/node_engine/node_base.py, .../workflow_engine.py, .../security_manager.py, .../workflow_manager.py, .../node_library.py, .../email_nodes.py
New node framework: base types, execution engine with security/sandbox/audit, persistence manager, node registry/library, and email-processing nodes.
Node workflow utilities & tests
backend/node_engine/migration_utils.py, .../test_nodes.py, .../test_integration.py, .../test_migration.py, .../test_security.py, .../test_sanitization.py
Legacy-to-node migration utilities and comprehensive async tests for execution, security, migration, and sanitization.
Backend app wiring
backend/python_backend/main.py, .../settings.py, .../dependencies.py, .../exceptions.py
App initialized via settings; DI-based services/providers; unified API error model and handlers.
Routes: legacy + node + v1
backend/python_backend/email_routes.py, .../category_routes.py, .../filter_routes.py, .../workflow_routes.py, .../advanced_workflow_routes.py, .../node_workflow_routes.py, .../routes/v1/*.py, .../performance_routes.py
Shift to service layer; add node-based and advanced workflow endpoints; versioned v1 email/category APIs; performance endpoint import tidy.
Services layer
backend/python_backend/services/base_service.py, .../services/email_service.py, .../services/category_service.py
Introduces BaseService/BaseResponse; email and category service implementations with lazy DB access and standardized responses.
Database & JSON DB
backend/python_backend/database.py, .../json_database.py
New JSONDataManager with compressed storage; DatabaseManager gains get/search/update methods; decorator usage updated.
Performance logging
backend/python_backend/performance_monitor.py, deployment/test_stages.py
Unified log_performance decorator for sync/async; test stages updated, security test param renamed.
Models (Pydantic v2)
backend/python_backend/models.py
Model updates to ConfigDict, field aliases, added enums/constants, expanded response models, validator migration.
Model management & training
backend/python_backend/model_manager.py, .../training_routes.py, .../tests/test_training_routes.py
Directory-based model discovery/metadata; training API with background jobs and persistence; tests added.
NLP pipeline & components
backend/python_nlp/nlp_engine.py, .../analysis_components/*, .../text_utils.py
NLP refactor to HF pipelines; new intent/urgency components; text cleaning utility; adjusted internal aggregation.
Gmail service & retrieval
backend/python_nlp/gmail_service.py, .../gmail_integration.py, .../smart_retrieval.py
Gmail service expanded (async command exec, analysis, training, retrieval helpers); retrieval system enhanced with OAuth, quotas, analytics; integration simplified placeholder.
Plugins
backend/plugins/*.py
Import order tweaks; added gradio import for visualizer; no API changes.
Gradio UI
backend/python_backend/gradio_app.py, .../workflow_editor_ui.py
Multi-tab UI with analysis, workflows, training; new workflow editor/advanced UI using Blocks.
Backend workflow engine (legacy)
backend/python_backend/workflow_engine.py, .../workflow_manager.py
BaseWorkflow gets DI constructor; logging import reorder.
Tests adjustments
backend/python_backend/tests/*, backend/python_nlp/tests/test_nlp_engine.py
Path/import fixes, encoding handling, updated expectations for new APIs and constants.
Docs overhaul
README.md, GEMINI.md, QWEN.md, AGENTS.md, docs/*, CODEREVIEW_REPORT.md, STATIC_ANALYSIS_REPORT.md, .qwen/PROJECT_SUMMARY.md, branch_alignment_report.md, backend/python_backend/README.md (removed)
Broad documentation rewrite/additions for architecture, workflows, deployment, hardening, reports; legacy backend README removed.
Configs
config/llm_guidelines*.json
max_tokens increased to 4096 across configs.
Misc backend pkg
backend/python_backend/__init__.py, .../enhanced_routes.py, backend/plugins/__init__.py, backend/plugins/plugin_manager.py
Deprecation and all updates; import reorganizations; minor typing annotation.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor User
  participant API as FastAPI Routes
  participant WM as WorkflowManager
  participant WE as WorkflowEngine
  participant Sec as SecurityManager/Audit
  participant Nodes as Nodes (EmailSource→Preprocess→AI→Filter→Action)

  User->>API: POST /api/nodes/workflows/{id}/execute
  API->>WM: load_workflow(id)
  WM-->>API: Workflow
  API->>WE: execute_workflow(workflow, inputs, user)
  WE->>Sec: audit: workflow_start, acquire resources
  loop Topological order
    WE->>Nodes: execute(node, ctx) with timeout
    Nodes-->>WE: outputs/status
    WE->>Sec: audit: node_execution(status)
  end
  WE->>Sec: audit: workflow_end, release resources
  WE-->>API: ExecutionContext (status, path, errors, outputs)
  API-->>User: Response
Loading
sequenceDiagram
  autonumber
  actor Admin
  participant MU as MigrationUtils
  participant WM as WorkflowManager
  participant NB as NodeBuilder

  Admin->>MU: migrate_workflow_file(legacy.json)
  MU->>MU: parse legacy config
  MU->>NB: build nodes/connections (Std pipeline)
  NB-->>MU: NodeWorkflow
  MU->>WM: save_workflow(new_id)
  WM-->>MU: path/id
  MU-->>Admin: migration report + id
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~150 minutes

Possibly related issues

Possibly related PRs

Suggested labels

enhancement

Poem

A rabbit taps keys with a whiskered grin,
New nodes hop in line—let workflows begin!
Security stands guard, audits take note,
NLP hums softly, models now float.
CI skips when told, docs bloom like spring—
Ship it, I thump; what a beautiful thing! 🐇✨

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fixes-branch

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 0c67476 and caad179.

⛔ Files ignored due to path filters (3)
  • models/test_model_63ba9a7c-088f-48d7-8996-97ab99f871a1.pkl is excluded by !**/*.pkl
  • models/test_model_d4ef6ccb-59e7-4862-be91-7608b350c863.pkl is excluded by !**/*.pkl
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (95)
  • .gitattributes (1 hunks)
  • .github/workflows/ci.yml (2 hunks)
  • .gitignore (1 hunks)
  • .qwen/PROJECT_SUMMARY.md (2 hunks)
  • AGENTS.md (1 hunks)
  • CODEREVIEW_REPORT.md (1 hunks)
  • GEMINI.md (1 hunks)
  • JULES_WIP_ANALYSIS.md (5 hunks)
  • QWEN.md (1 hunks)
  • README.md (1 hunks)
  • STATIC_ANALYSIS_REPORT.md (1 hunks)
  • backend/node_engine/email_nodes.py (1 hunks)
  • backend/node_engine/migration_utils.py (1 hunks)
  • backend/node_engine/node_base.py (1 hunks)
  • backend/node_engine/node_library.py (1 hunks)
  • backend/node_engine/security_manager.py (1 hunks)
  • backend/node_engine/test_integration.py (1 hunks)
  • backend/node_engine/test_migration.py (1 hunks)
  • backend/node_engine/test_nodes.py (1 hunks)
  • backend/node_engine/test_sanitization.py (1 hunks)
  • backend/node_engine/test_security.py (1 hunks)
  • backend/node_engine/workflow_engine.py (1 hunks)
  • backend/node_engine/workflow_manager.py (1 hunks)
  • backend/plugins/__init__.py (1 hunks)
  • backend/plugins/base_plugin.py (1 hunks)
  • backend/plugins/email_filter_node.py (1 hunks)
  • backend/plugins/email_visualizer_plugin.py (1 hunks)
  • backend/plugins/plugin_manager.py (1 hunks)
  • backend/python_backend/README.md (0 hunks)
  • backend/python_backend/__init__.py (1 hunks)
  • backend/python_backend/advanced_workflow_routes.py (1 hunks)
  • backend/python_backend/category_routes.py (3 hunks)
  • backend/python_backend/database.py (8 hunks)
  • backend/python_backend/dependencies.py (1 hunks)
  • backend/python_backend/email_routes.py (11 hunks)
  • backend/python_backend/enhanced_routes.py (1 hunks)
  • backend/python_backend/exceptions.py (1 hunks)
  • backend/python_backend/filter_routes.py (2 hunks)
  • backend/python_backend/gradio_app.py (11 hunks)
  • backend/python_backend/json_database.py (1 hunks)
  • backend/python_backend/main.py (3 hunks)
  • backend/python_backend/model_manager.py (1 hunks)
  • backend/python_backend/model_routes.py (1 hunks)
  • backend/python_backend/models.py (18 hunks)
  • backend/python_backend/node_workflow_routes.py (1 hunks)
  • backend/python_backend/performance_monitor.py (6 hunks)
  • backend/python_backend/performance_routes.py (1 hunks)
  • backend/python_backend/plugin_manager.py (3 hunks)
  • backend/python_backend/routes/v1/category_routes.py (1 hunks)
  • backend/python_backend/routes/v1/email_routes.py (1 hunks)
  • backend/python_backend/services/base_service.py (1 hunks)
  • backend/python_backend/services/category_service.py (1 hunks)
  • backend/python_backend/services/email_service.py (1 hunks)
  • backend/python_backend/settings.py (1 hunks)
  • backend/python_backend/tests/conftest.py (1 hunks)
  • backend/python_backend/tests/test_ai_engine.py (2 hunks)
  • backend/python_backend/tests/test_database_optimizations.py (1 hunks)
  • backend/python_backend/tests/test_email_routes.py (1 hunks)
  • backend/python_backend/tests/test_filter_routes.py (1 hunks)
  • backend/python_backend/tests/test_model_manager.py (5 hunks)
  • backend/python_backend/tests/test_training_routes.py (1 hunks)
  • backend/python_backend/tests/test_workflow_engine.py (4 hunks)
  • backend/python_backend/tests/test_workflow_routes.py (3 hunks)
  • backend/python_backend/training_routes.py (1 hunks)
  • backend/python_backend/workflow_editor_ui.py (1 hunks)
  • backend/python_backend/workflow_engine.py (2 hunks)
  • backend/python_backend/workflow_manager.py (1 hunks)
  • backend/python_backend/workflow_routes.py (3 hunks)
  • backend/python_nlp/ai_training.py (2 hunks)
  • backend/python_nlp/analysis_components/intent_model.py (1 hunks)
  • backend/python_nlp/analysis_components/sentiment_model.py (1 hunks)
  • backend/python_nlp/analysis_components/topic_model.py (1 hunks)
  • backend/python_nlp/analysis_components/urgency_model.py (1 hunks)
  • backend/python_nlp/gmail_integration.py (1 hunks)
  • backend/python_nlp/gmail_service.py (6 hunks)
  • backend/python_nlp/nlp_engine.py (15 hunks)
  • backend/python_nlp/protocols.py (1 hunks)
  • backend/python_nlp/retrieval_monitor.py (0 hunks)
  • backend/python_nlp/smart_filters.py (6 hunks)
  • backend/python_nlp/smart_retrieval.py (8 hunks)
  • backend/python_nlp/tests/test_nlp_engine.py (1 hunks)
  • backend/python_nlp/text_utils.py (1 hunks)
  • branch_alignment_report.md (2 hunks)
  • config/llm_guidelines.json (1 hunks)
  • config/llm_guidelines.qwen.json (1 hunks)
  • config/llm_guidelines.scientific.json (1 hunks)
  • deployment/test_stages.py (4 hunks)
  • docs/advanced_workflow_system.md (1 hunks)
  • docs/application_launch_hardening_strategy.md (1 hunks)
  • docs/changes_report.md (1 hunks)
  • docs/deployment_guide.md (2 hunks)
  • docs/git_workflow_plan.md (1 hunks)
  • docs/node_architecture.md (1 hunks)
  • docs/project_documentation_guide.md (1 hunks)
  • docs/project_structure_comparison.md (1 hunks)

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

MasumRab and others added 4 commits October 15, 2025 11:06
…in path expression

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…in path expression

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@MasumRab MasumRab marked this pull request as ready for review October 15, 2025 04:44
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We failed to fetch the diff for pull request #146

You can try again by commenting this pull request with @sourcery-ai review, or contact us for help.

@MasumRab MasumRab merged commit c340f43 into main Oct 15, 2025
5 of 6 checks passed
MasumRab added a commit that referenced this pull request Oct 29, 2025
@MasumRab MasumRab deleted the fixes-branch branch November 1, 2025 07:51
MasumRab added a commit that referenced this pull request Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants