fix(bigquery): limit result set size to prevent browser memory crashes by rusackas · Pull Request #38588 · apache/superset

rusackas · 2026-03-11T20:21:19Z

SUMMARY

Adopts and improves the fix from #36387 (originally by @ethan-l-geotab). Fixes #36385.

BigQuery queries returning huge result sets (950+ MB) crash Chrome by loading everything into browser memory at once. This PR implements memory-aware progressive fetching in BigQueryEngineSpec.fetch_data:

Progressive fetch: Samples an initial batch (1000 rows) to estimate row size, then extrapolates how many total rows fit within the BQ_FETCH_MAX_MB config limit (default 200 MB)
Warning propagation: When results are truncated, a warning is passed through Flask g to the query context processor, which adds it to the response payload
Frontend toast: The chart action handler displays a warning toast to the user when results were truncated
Graceful fallback: On any error in the progressive fetch, falls back to the parent BaseEngineSpec.fetch_data implementation

Key differences from the original PR (#36387):

Removed the BQ_MEMORY_LIMIT_FETCH feature flag -- the fix is always-on
The BQ_FETCH_MAX_MB config constant (default 200 MB) is the only operator-level knob
Added comprehensive unit tests for the new BigQuery fetch logic and warning propagation
Applied to the renamed chartAction.ts (was .js in the original PR)

Files changed:

superset/db_engine_specs/bigquery.py -- Memory-aware progressive fetch implementation
superset/common/query_context_processor.py -- Warning propagation via Flask g
superset/config.py -- BQ_FETCH_MAX_MB = 200 config constant
superset-frontend/src/components/Chart/chartAction.ts -- Warning toast display
superset-frontend/packages/superset-ui-core/src/query/types/QueryResponse.ts -- warning field on ChartDataResponseResult
tests/unit_tests/db_engine_specs/test_bigquery.py -- 5 new test cases
tests/unit_tests/common/test_query_context_processor.py -- 2 new test cases

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before: Chrome crashes or becomes unresponsive when BigQuery returns 950+ MB of data.

After: Results are truncated to fit within the configured memory budget, and a warning toast informs the user.

TESTING INSTRUCTIONS

Configure a BigQuery datasource in Superset
Run a query that returns a very large result set (millions of rows)
Verify that results are truncated and a warning toast appears
Set BQ_FETCH_MAX_MB to a smaller value (e.g., 10) in superset_config.py to test truncation with smaller datasets
Verify that queries returning small result sets work normally without any warning
Run the new unit tests: pytest tests/unit_tests/db_engine_specs/test_bigquery.py -k test_fetch_data -v

ADDITIONAL INFORMATION

Has associated issue: Fixes Large chart data doesn't return a helpful error message #36385
Required feature flags:
Changes UI
Includes DB Migration
Introduces new feature or API
Removes existing feature or API

Co-authored-by: ethan-l-geotab ethanliong@geotab.com

@ethan-l-geotab

Implement memory-aware progressive fetching in BigQuery's fetch_data method. Large result sets (950+ MB) previously crashed Chrome by loading everything into memory at once. The fix samples an initial batch to estimate row size, then fetches only as many rows as fit within the BQ_FETCH_MAX_MB config limit (default 200 MB). A warning toast is shown to users when results are truncated. This is always-on with no feature flag -- operators control the budget via the BQ_FETCH_MAX_MB config constant. Originally by @ethan-l-geotab in #36387. Co-authored-by: ethan-l-geotab <ethanliong@geotab.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

bito-code-review

Code Review Agent Run #e5147f

Actionable Suggestions - 3

superset/db_engine_specs/bigquery.py - 2
- Inaccurate memory estimation · Line 335-335
- Avoid blind exception catch · Line 367-367
superset-frontend/packages/superset-ui-core/src/query/types/QueryResponse.ts - 1
- Backend schema missing warning field · Line 80-80

Review Details

Files reviewed - 7 · Commit Range: 1773531..1773531
- superset-frontend/packages/superset-ui-core/src/query/types/QueryResponse.ts
- superset-frontend/src/components/Chart/chartAction.ts
- superset/common/query_context_processor.py
- superset/config.py
- superset/db_engine_specs/bigquery.py
- tests/unit_tests/common/test_query_context_processor.py
- tests/unit_tests/db_engine_specs/test_bigquery.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- Eslint (Linter) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

/review - Manually triggers a full AI review.
/pause - Pauses automatic reviews on this pull request.
/resume - Resumes automatic reviews.
/resolve - Marks all Bito-posted review comments as resolved.
/abort - Cancels all in-progress reviews.

Refer to the documentation for additional commands.

Configuration

This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.

Documentation & Help

AI Code Review powered by

bito-code-review · 2026-03-11T20:41:09Z

superset/db_engine_specs/bigquery.py

+                first_batch = [r.values() for r in first_batch]
+
+            # Estimate how many rows fit in the memory budget
+            first_batch_bytes = sys.getsizeof(str(first_batch))


Inaccurate memory estimation

The memory size estimation uses sys.getsizeof(str(first_batch)), but str() creates a string representation that can be significantly larger than the actual in-memory size of the list object. This leads to overestimating memory usage and potentially fetching fewer rows than the budget allows. Use sys.getsizeof(first_batch) for correct memory budgeting.

Code suggestion

Check the AI-generated fix before applying

Suggested change

first_batch_bytes = sys.getsizeof(str(first_batch))

first_batch_bytes = sys.getsizeof(first_batch)

Code Review Run #e5147f

Should Bito avoid suggestions like this for future reviews? (Manage Rules)

Yes, avoid them

bito-code-review · 2026-03-11T20:41:09Z

superset/db_engine_specs/bigquery.py

+            g.bq_memory_limited_row_count = len(data)
+            return data
+
+        except Exception:  # pylint: disable=broad-except


Avoid blind exception catch

Replace the broad except Exception: at line 367 with specific exception types that are expected from cursor operations. This improves error handling clarity and prevents masking unexpected errors.

Code suggestion

Check the AI-generated fix before applying

Suggested change

except Exception: # pylint: disable=broad-except

except (DatabaseError, OperationalError, Exception): # pylint: disable=broad-except

Code Review Run #e5147f

Should Bito avoid suggestions like this for future reviews? (Manage Rules)

Yes, avoid them

bito-code-review · 2026-03-11T20:41:09Z

superset-frontend/packages/superset-ui-core/src/query/types/QueryResponse.ts

  // TODO(hainenber): define proper type for below attributes
  rejected_filters?: any[];
  applied_filters?: any[];
+  warning?: string | null;


Backend schema missing warning field

The added warning field matches backend response data, but the schema lacks it, potentially causing validation issues. Add warning = fields.String(allow_none=True) to ChartDataResponseResult.

Code Review Run #e5147f

Should Bito avoid suggestions like this for future reviews? (Manage Rules)

Yes, avoid them

pull-request-size bot added the size/L label Mar 11, 2026

github-actions bot added the packages label Mar 11, 2026

dosubot bot added the data:connect:googlebigquery Related to BigQuery label Mar 11, 2026

bito-code-review bot reviewed Mar 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(bigquery): limit result set size to prevent browser memory crashes#38588

fix(bigquery): limit result set size to prevent browser memory crashes#38588
rusackas wants to merge 1 commit intomasterfrom
adopt-pr-36387-bq-memory-limit

rusackas commented Mar 11, 2026

Uh oh!

bito-code-review bot left a comment •

edited

Loading

Uh oh!

bito-code-review bot Mar 11, 2026

Uh oh!

bito-code-review bot Mar 11, 2026

Uh oh!

bito-code-review bot Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	first_batch_bytes = sys.getsizeof(str(first_batch))
	first_batch_bytes = sys.getsizeof(first_batch)

	except Exception: # pylint: disable=broad-except
	except (DatabaseError, OperationalError, Exception): # pylint: disable=broad-except

Conversation

rusackas commented Mar 11, 2026

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Uh oh!

bito-code-review bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Code Review Agent Run #e5147f

Uh oh!

bito-code-review bot Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

bito-code-review bot Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

bito-code-review bot Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bito-code-review bot left a comment •

edited

Loading