[Bug] sqlparse 0.5.5 results in failed unit test

[FrankTub]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I have some unit tests, on our largest model (~1100 lines of code and 26 direct upstream dependencies). Since today the following code returns an error:

dbt test -s my-unit-test

The error:

Unhandled error while executing
Maximum number of tokens exceeded (10000).

We have pinpointed it to that it happens when you have sqlparse 0.5.5 in your environment, when you pin sqlparse to 0.5.4 the code works just fine.

The hard part is that it does not happen to all our unit tests, only our largest one

Expected Behavior

Be able to run all unit tests without having to worry about a dependent package.

Steps To Reproduce

1. Create a big model with at least 26 dependencies
2. Create an unit tests. E.g.

unit_tests:
  - name: unit_test_organizations
    description: Check all organization/one obligor outstanding buckets are determined correctly
    model: organizations
    overrides:
      macros:
        get_current_date_or_last_business_day: to_date('01-01-2024','dd-mm-yyyy')
    given:
      - input: ref('salesforce_account')
        rows:
          - {account_id: abc}
      - input: ref('active_customers')
        rows:
          - {customer_organization_id: 1}
      - input: ref('active_ecl_stage_3_organizations')
        rows:
          - {customer_organization_id: 4, is_currently_stage_3: true}
      - input: ref('all_pd_ratings')
        rows:
          - {year: 1}
      - input: ref('beehive_approvals')
        rows:
          - {owner_id: 1}
      - input: ref('beehive_bank_accounts')
        rows:
          - {bank_account_id: 1, organization_id: 1, deactivated_at: null, bank_account_number: NL123, primary: true}
          - {bank_account_id: 2, organization_id: 1, deactivated_at: 2024-01-01, bank_account_number: NL456, primary: true}
          - {bank_account_id: 3, organization_id: 1, deactivated_at: null, bank_account_number: NL789, primary: false}
          - {bank_account_id: 4, organization_id: 1, deactivated_at: 2024-01-01, bank_account_number: NLXXX, primary: false}
      - input: ref('beehive_parties')
        rows:
          - {id: 1, name: test1, organization_no: C1, type: Organization, organization_group_id: 1}
          - {id: 2, name: test2, organization_no: C2, type: Organization, organization_group_id: 1}
          - {id: 3, name: test3, organization_no: C3, type: Organization, organization_group_id: 2}
          - {id: 4, name: test4, organization_no: C4, type: Organization, organization_group_id: 3}
      - input: ref('public_beehive_labels')
        rows:
          - {attribute: organization.customer_segments, attribute_code: small_company, value: Klein bedrijf}
      - input: ref('beehive_organization_groups')
        rows:
          - {organization_group_id: 1, name: test1}
          - {organization_group_id: 2, name: test2}
          - {organization_group_id: 3, name: test3}
      - input: ref('beehive_collection_mandates')
        rows:
          - {organization_id: 1}
      - input: ref('customer_forbearance_measures')
        rows:
          - {customer_organization_id: 1}
      - input: ref('customer_ledger_entries')
        rows:
          - {customer_no: C1}
      - input: ref('daily_customer_balances')
        rows:
          - {customer_no: C1, date_actual: 2024-01-01, overdue_amount_eur: 1000, days_past_due: 10, customer_performance_stage_id: 2}
          - {customer_no: C2, date_actual: 2024-01-01, overdue_amount_eur: 99, days_past_due: 93, customer_performance_stage_id: 1}
          - {customer_no: C3, date_actual: 2024-01-01, overdue_amount_eur: 0, days_past_due: 0, customer_performance_stage_id: 1}
          - {customer_no: C4, date_actual: 2024-01-01, overdue_amount_eur: 0, days_past_due: 0, customer_performance_stage_id: 1}
      - input: ref('beehive_deals')
        rows:
          - {customer_organization_id: 1, kvo_organization_id: 2}
      - input: ref('ecl_stage_changes')
        rows:
          - {customer_organization_id: 1}
      - input: ref('entity_legal_form')
        rows:
          - {elf_code: CDOV, language: English}
      - input: ref('installments')
        rows:
          - {customer_no: C1}
      - input: ref('internal_users')
        rows:
          - {user_id: 1}
          - {user_id: 2}
      - input: ref('kycs')
        rows:
          - {party_id: 1}
      - input: ref('beehive_leases')
        rows:
          - {vendor_organization_id: 2}
      - input: ref('lease_contracts')
        rows:
          - {customer_organization_id: 1, current_outstanding_eur: 100000}
          - {customer_organization_id: 1, current_outstanding_eur: 100000}
          - {customer_organization_id: 1, current_outstanding_eur: 100000}
          - {customer_organization_id: 2, current_outstanding_eur: 500000}
          - {customer_organization_id: 3, current_outstanding_eur: 0}
          - {customer_organization_id: 4, current_outstanding_eur: 10000000}
      - input: ref('monthly_expected_loss_leases')
        rows:
          - {customer_organization_id: 1}
      - input: ref('contacts_on_organizations')
        rows:
          - {organization_id: 1}
      - input: ref('sbi_codes')
        rows:
          - {sbi_code: 0111}
      - input: ref('base_snapshot_parties')
        rows:
          - {id: 1}
      - input: ref('soft4lease_beequip_contact')
        rows:
          - {organization_no: C1, name: test1}
          - {organization_no: C2, name: test2}
          - {organization_no: C3, name: test3}
          - {organization_no: C4, name: test4}
      - input: ref('soft4lease_beequip_vendor_ledger_entry')
        rows:
          - {vendor_no: C3}
    expect:
      rows:
        - {id: 1, customer_outstanding_eur_bucket_id: 4, one_obligor_outstanding_eur_bucket_id: 6, current_customer_performance_stage_id: 2, bank_account_number: NL123}
        - {id: 2, customer_outstanding_eur_bucket_id: 5, one_obligor_outstanding_eur_bucket_id: 6, current_customer_performance_stage_id: 1, bank_account_number: null}
        - {id: 3, customer_outstanding_eur_bucket_id: 0, one_obligor_outstanding_eur_bucket_id: 0, current_customer_performance_stage_id: 1, bank_account_number: null}
        - {id: 4, customer_outstanding_eur_bucket_id: 10, one_obligor_outstanding_eur_bucket_id: 10, current_customer_performance_stage_id: 4, bank_account_number: null}

Run:

dbt test -s my-unit-test

Relevant log output

Environment

- OS: MacOS
- Python: 3.11.9
- dbt: 1.9.4 (Snowflake)

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[DmytryEmery]

This broke our workflows last night, we get an interesting error: Maximum number of tokens exceeded {number}

pretty confident this broke it:
andialbrecht/sqlparse@da67ac1

[bk-ww]

Following. This took our production down as well.

[mh-avant]

The error is occurring in dbt cloud with the Databricks adapter as well. The query that it's failing on is very small.

[DmytryEmery]

just FYI, downgrading to sqlparse 0.5.4 does resolve the issue in my case.

[cheyuriy]

DBT cloud with BigQuery - same, since an hour ago