-
Notifications
You must be signed in to change notification settings - Fork 337
Description
Problem
When defining Snowflake dynamic tables in Atlas HCL, there is no way to specify which warehouse should be used for the dynamic table's refresh operations. The warehouse is always inherited from the connection URL's ?warehouse= query parameter.
In Snowflake, CREATE DYNAMIC TABLE supports a WAREHOUSE clause:
CREATE DYNAMIC TABLE my_table
TARGET_LAG = '20 seconds'
WAREHOUSE = WH_DTS_PROD
AS SELECT ...This is important because organizations typically use different warehouses for different workloads — e.g., a small warehouse for Atlas migrations/DDL operations and a larger compute-optimized warehouse for dynamic table refreshes.
Proposed Solution
Add a warehouse attribute to the dynamic_table block in the Snowflake HCL dialect:
dynamic_table "DIM_ACCOUNTS" {
schema = schema.PROCESSING
target_lag = "20 seconds"
warehouse = "WH_DTS_PROD"
refresh_mode = INCREMENTAL
as = var.dim_accounts_query
}This would generate:
CREATE DYNAMIC TABLE "PROCESSING"."DIM_ACCOUNTS"
TARGET_LAG = '20 seconds'
WAREHOUSE = "WH_DTS_PROD"
REFRESH_MODE = INCREMENTAL
AS ...When warehouse is omitted, the current behavior (inheriting from the connection) should remain.
Current Workaround
Running ALTER DYNAMIC TABLE ... SET WAREHOUSE = '...' as a post-migration step, which is fragile and not declarative.
Additional Context
- The Snowflake HCL reference documents
dynamic_tablewithtarget_lag,refresh_mode,initialize, andas— but nowarehouse. - Snowflake's
CREATE DYNAMIC TABLEdocumentation listsWAREHOUSEas a required clause. - This also causes false positives in schema drift checks, since the live DB reflects the actual warehouse while Atlas normalizes to the connection warehouse.