source-shopify-native: support multiple Shopify stores#3884
Open
JustinASmith wants to merge 1 commit intomainfrom
Open
source-shopify-native: support multiple Shopify stores#3884JustinASmith wants to merge 1 commit intomainfrom
JustinASmith wants to merge 1 commit intomainfrom
Conversation
2017075 to
75f2634
Compare
nicolaslazo
approved these changes
Feb 10, 2026
Contributor
nicolaslazo
left a comment
There was a problem hiding this comment.
I haven't had to deal with config migrations yet so my experience is limited here, but the logic looks solid 👍
75f2634 to
87fdfe4
Compare
87fdfe4 to
6bc377f
Compare
Adds multi-store support allowing a single capture to replicate data from multiple Shopify stores. Each store uses Access Token authentication with independent rate limiting and HTTP sessions. Config & models: - StoreConfig model with per-store credentials (Access Token only) - Legacy single-store configs auto-migrated via wrap model_validator - Duplicate store name validation - OAuth temporarily removed; will revisit when Flow/UI supports per-store OAuth (see git history for previous OAUTH2_SPEC) Document keys & state: - Collection keys use ["/_meta/store", "/id"] for cross-store uniqueness - Discover always proposes composite keys for forward compatibility; legacy captures may have mixed keys (old bindings retain ["/id"], new bindings get composite) to reduce backfill scope on store addition - Validation requires backfill when transitioning to multi-store keys - Dict-based state format keyed by store ID with automatic migration from legacy flat format during open Store context: - StoreValidationContext dataclass injected via Pydantic validation context; mode="before" validator + validate_assignment=True ensures store field survives model_dump(exclude_unset=True) - Per-store HTTP wrapper (StoreHTTP) shares parent session for connection pooling with independent rate limiters - Parallel store initialization and credential validation via asyncio.gather - Fail-fast on any store init failure (TODO: graceful degradation) Infrastructure: - Extracted dt_to_str/str_to_dt into utils.py to break circular import chain: models.py -> graphql/__init__.py -> bulk_job_manager -> models - Removed eager bulk_job_manager import from graphql/__init__.py; BulkJobManager now imported directly where needed - BulkJobManager: public check_connectivity() method, cancel timeout using time.monotonic(), store-prefixed log messages - GraphQL client accepts typed StoreValidationContext for document parsing Tests: - State migration: flat-to-dict, idempotency, multi-store addition, removed store preservation, error cases - Binding key validation: multi-store transition, backfill acknowledgment, new binding handling - _should_use_store_in_key: all branches including key regression - EndpointConfig migration: legacy format, multi-store, duplicate rejection - StoreValidationContext: serialization, exclude_unset, JSON parsing Closes: #3508
6bc377f to
88c5222
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Summary
Adds multi-store support to
source-shopify-native, allowing a single capture to replicate data from multiple Shopify stores. Each store authenticates independently via Access Token with its own rate limiter and HTTP session.Key changes
StoreConfigmodel wrapping per-storestorename andcredentials. Legacy single-store configs ({"store": "...", "credentials": {...}}) are auto-migrated to the newstoreslist format via a Pydanticmode="wrap"validator, with the original config persisted on nextopenviaconfig_updateevent.["/_meta/store", "/id"]to ensure cross-store document uniqueness. Discover always proposes composite keys for forward compatibility. Legacy captures may have mixed keys — old bindings retain["/id"]while newly discovered bindings get["/_meta/store", "/id"]— reducing the number of bindings requiring backfill if a store is added later.["/id"]bindings so the key structure can change to include the store identifier.{"inc": {"store-id": {"cursor": "..."}}}). Legacy flat state is automatically migrated duringopenand checkpointed.StoreHTTPwrapper shares the parentaiohttp.ClientSessionfor connection pooling while providing independentRateLimiterandTokenSourceper store. Store initialization and credential validation run in parallel viaasyncio.gather.StoreValidationContextdataclass is passed through Pydantic's validation context to inject the_meta.storefield on every document. Amode="before"model validator combined withvalidate_assignment=Trueensures the field survivesmodel_dump(exclude_unset=True).dt_to_str/str_to_dtintoutils.pyand removed the eagerbulk_job_managerimport fromgraphql/__init__.pyto break the cycle:models.py→graphql/__init__.py→bulk_job_manager→models.py.StoreInitError: fail-fast behavior documented with TODO for graceful degradation.Other improvements
BulkJobManager: publiccheck_connectivity()method (replacing private_get_running_jobs()usage for validation), cancel timeout usingtime.monotonic(), store-prefixed log messages, typo fix.Closes #3508
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
TODO: Needs updated documentation PR.
Notes for reviewers:
Test plan
poetry run pytest tests/test_state_migration.py— 41 tests covering:_should_use_store_in_key(all branches including key regression prevention)EndpointConfigmigration (legacy format, multi-store, duplicate rejection)StoreValidationContext(serialization,exclude_unset, JSON parsing, existing meta preserved)poetry run pytest tests/test_snapshots.py --insta=update— spec, discover, capture snapshots pass