[WIP] feat(narwhals): implement the new unifying backend by deepyaman · Pull Request #2223 · unionai-oss/pandera

deepyaman · 2026-03-10T02:50:30Z

Tip

I've used Claude to significantly squash the commits and remove GSD artifacts; however; I'm still working off of the branch that has all of those, now pushed to https://github.com/deepyaman/pandera/tree/feat/narwhals/create-backend-all-artifacts

Prologue

Most of the below description, as well as almost all of the PR, is AI-generated. However, I have been very closely guiding the process and reviewing each step of the way. I have still not deeply reviewed the code in it's entirety, which I plan to do.

I've also verified the functionality at a high level manually. With this change, you can run checks using the existing Polars and Ibis APIs, and they leverage the newly-added Narwhals backend-pretty good for a first pass!

I've currently started exploring adding support for the PySpark backend, but that could be a separate PR. What I think are necessary steps in this PR, before merging:

Proper review of the generated code (of course)
Removal of artifacts from .planning/ (I left them there for now, both as a backup and in case wanted to share the context)
Adding the CI workflow, that will test using both the existing backend and the Narwhals backend for Polars and Ibis (probably the biggest remaining piece)

Turning it over to my agentic intern...

Warning

The below is outdated and needs updating for milestone v1.1.

Narwhals backend for Polars and Ibis (v1.0)

This PR introduces a unified Narwhals-backed validation engine that replaces library-specific backends for Polars and Ibis with a single shared implementation. Users continue to pass native frames — pandera routes them through Narwhals internally.

Scope: 14 new files, ~2,950 lines of new production code and tests across 5 implementation phases.

What was built

New packages and modules

File	Description
`pandera/api/narwhals/types.py`	`NarwhalsData` NamedTuple (`frame`, `key`) — the data container passed to builtin checks
`pandera/api/narwhals/utils.py`	`_to_native()` helper to unwrap narwhals wrappers
`pandera/engines/narwhals_engine.py`	Narwhals dtype engine with 18 registered types (Int8–UInt64, Float32/64, String, Bool, Date, DateTime, Duration, Categorical, List, Struct)
`pandera/backends/narwhals/checks.py`	`NarwhalsCheckBackend` — routes builtins through `NarwhalsData`, user-defined checks through native frame unwrapping or ibis delegation
`pandera/backends/narwhals/builtin_checks.py`	14 builtin checks implemented against `nw.Expr`: `equal_to`, `not_equal_to`, `greater_than`, `greater_than_or_equal_to`, `less_than`, `less_than_or_equal_to`, `in_range`, `isin`, `notin`, `str_matches`, `str_contains`, `str_startswith`, `str_endswith`, `str_length`
`pandera/backends/narwhals/components.py`	`ColumnBackend` — per-column validation (dtype check, nullable, unique, run_checks)
`pandera/backends/narwhals/container.py`	`DataFrameSchemaBackend` — full container validation (parsers, checks, strict/ordered, lazy mode, drop_invalid_rows)
`pandera/backends/narwhals/base.py`	`NarwhalsSchemaBackend` — shared helpers: `run_check`, `subsample`, `failure_cases_metadata`, `drop_invalid_rows`, `is_float_dtype`

Modified files

File	Change
`pandera/backends/polars/register.py`	Auto-detects narwhals at registration time; swaps all Polars backends for narwhals equivalents when narwhals is installed. Falls back to native Polars backends otherwise. Emits `UserWarning`.
`pandera/backends/ibis/register.py`	Same auto-detection pattern for Ibis. Adds `@lru_cache` (was missing). Registers `NarwhalsCheckBackend` for `ibis.Table` / `ibis.Column` / `nw.LazyFrame`.

Test suite

tests/backends/narwhals/ — backend-agnostic, parameterized against both Polars and Ibis:

conftest.py — make_narwhals_frame fixture producing nw.LazyFrame from either pl.LazyFrame or ibis.memtable; autouse fixture that calls both register functions
test_checks.py — 14 builtin checks × 2 backends (valid + invalid data paths)
test_components.py — column-level dtype, nullable, unique, and check validation
test_container.py — full container validation: strict, ordered, lazy mode, failure cases, drop_invalid_rows
test_parity.py — behavioral parity between Polars and Ibis paths: valid, invalid, lazy, strict, filter, decorator, DataFrameModel
test_narwhals_dtypes.py — dtype engine registration and coerce/try_coerce

Architecture decisions

Narwhals is internal plumbing, not a user-facing API. Users pass pl.DataFrame, pl.LazyFrame, or ibis.Table — no changes to call sites.

Auto-detection over configuration. register_polars_backends() and register_ibis_backends() check for narwhals via try/import and swap backends transparently. No config flag needed.

SQL-lazy element_wise raises NotImplementedError. Row-level Python functions cannot be applied to lazy query plans (Ibis, DuckDB, PySpark). The error is surfaced as a SchemaError with CHECK_ERROR reason code. NOTE(@deepyaman): See narwhals-dev/narwhals#3512

Ibis drop_invalid_rows delegates to IbisSchemaBackend. Narwhals has no positional-join / row_number abstraction for SQL-lazy backends. NOTE(@deepyaman): I'll verify this later.

How to verify

Install with narwhals:

pip install pandera[<dataframe API you're using>,narwhals]

Polars — object API

import polars as pl
import pandera.polars as pa

schema = pa.DataFrameSchema({
    "name": pa.Column(pl.String, pa.Check.str_startswith("A")),
    "age":  pa.Column(pl.Int64,  pa.Check.greater_than(0)),
})

df = pl.DataFrame({"name": ["Alice", "Bob"], "age": [30, 25]})
schema.validate(df)  # raises SchemaError: "Bob" fails str_startswith("A")

Polars — class-based model

import polars as pl
import pandera.polars as pa
from pandera.typing.polars import DataFrame, Series

class UserSchema(pa.DataFrameModel):
    name: Series[str]
    age:  Series[int]

    @pa.check("age")
    def age_positive(cls, series):
        return series > 0

UserSchema.validate(pl.DataFrame({"name": ["Alice"], "age": [30]}))

Polars — lazy mode (collect all errors before raising)

import polars as pl
import pandera.polars as pa
from pandera.errors import SchemaErrors

schema = pa.DataFrameSchema({
    "a": pa.Column(pl.Int64, pa.Check.greater_than(10)),
    "b": pa.Column(pl.Int64, pa.Check.greater_than(10)),
})

try:
    schema.validate(pl.DataFrame({"a": [1, 2], "b": [3, 4]}), lazy=True)
except SchemaErrors as e:
    print(len(e.schema_errors))  # 2 — both columns collected

Ibis — same schema, different backend

import pandas as pd
import ibis
import pandera.ibis as pa
import ibis.expr.datatypes as dt

schema = pa.DataFrameSchema({
    "a": pa.Column(dt.int64, pa.Check.greater_than(0)),
})

t = ibis.memtable(pd.DataFrame({"a": [1, 2, 3]}))
result = schema.validate(t)
result.execute()

Ibis — user-defined check (delegates to IbisCheckBackend)

from pandera.api.ibis.types import IbisData
import ibis.selectors as s
from ibis import _ as D

def check_positive(data: IbisData):
    return data.table.select(s.across(s.all(), D > 0))

schema = pa.DataFrameSchema({"a": pa.Column(dt.int64, pa.Check(check_positive))})
schema.validate(ibis.memtable(pd.DataFrame({"a": [1, 2, 3]})))

Known gaps and next steps

Gap	Notes
`coerce` for Ibis	`xfail(strict=True)` — intentionally deferred; will break CI when implemented so the mark gets cleaned up
pandas via narwhals backend	Not yet wired; pandas currently still uses its own backend
PySpark via narwhals backend	Feasibility TBD
`add_missing_columns` parser	Deferred to next milestone
`set_default` for Column fields	Deferred to next milestone
Groupby-based checks	`group_by().agg()` pattern designed; not implemented
Schema IO (YAML/JSON)	Deferred
Hypothesis strategies	Deferred
`narwhals stable.v2` migration	Monitor narwhals releases; migrate when stable
`sample=` subsampling	Raises `NotImplementedError`; only `head=`/`tail=` are supported

codecov · 2026-03-10T02:57:27Z

Codecov Report

❌ Patch coverage is 3.18907% with 850 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.00%. Comparing base (7614754) to head (44967b2).
⚠️ Report is 15 commits behind head on dev/narwhals.

Files with missing lines	Patch %	Lines
pandera/backends/narwhals/container.py	0.00%	204 Missing ⚠️
pandera/backends/narwhals/base.py	0.00%	171 Missing ⚠️
pandera/engines/narwhals_engine.py	0.00%	149 Missing ⚠️
pandera/backends/narwhals/components.py	0.00%	122 Missing ⚠️
pandera/backends/narwhals/checks.py	0.00%	91 Missing ⚠️
pandera/backends/narwhals/builtin_checks.py	0.00%	63 Missing ⚠️
pandera/backends/polars/register.py	52.17%	11 Missing ⚠️
pandera/api/narwhals/types.py	0.00%	10 Missing ⚠️
pandera/api/narwhals/utils.py	0.00%	10 Missing ⚠️
pandera/backends/ibis/register.py	60.00%	10 Missing ⚠️
... and 1 more

Additional details and impacted files

@@               Coverage Diff                @@
##           dev/narwhals    #2223      +/-   ##
================================================
- Coverage         83.76%   78.00%   -5.77%     
================================================
  Files               137      147      +10     
  Lines             10764    11962    +1198     
================================================
+ Hits               9017     9331     +314     
- Misses             1747     2631     +884

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cosmicBboy · 2026-03-16T13:50:41Z

hey @deepyaman let me know if you need any help here

deepyaman · 2026-03-16T14:26:21Z

hey @deepyaman let me know if you need any help here

Hey @cosmicBboy, updated with a rough description of where things are at/what's necessary before moving forward with this.

cosmicBboy

hey @deepyaman how are you thinking about the .planning directory? should those be checked in for posterity (and future agent reference)?

I also made a dev/narwhals branch that we can use to merge this PR... can you do an interactive rebase to squash some of the commits into smaller chunks (so as to not lose granularity of commits)?

deepyaman · 2026-03-20T13:44:19Z

hey @deepyaman how are you thinking about the .planning directory? should those be checked in for posterity (and future agent reference)?

I don't intend to check them in, but I thought it could be useful to share for now. From my list of pre-merge TODOs above:

Removal of artifacts from .planning/ (I left them there for now, both as a backup and in case wanted to share the context)

I don't know what you think, but I don't think it makes sense to leave project-level artifacts just for agents, even if want to support them? TBH I'm not super familiar on this.

I also made a dev/narwhals branch that we can use to merge this PR... can you do an interactive rebase to squash some of the commits into smaller chunks (so as to not lose granularity of commits)?

Sure, will look into doing this.

deepyaman

There seem to be a number of potential issues, most of which fall into two categories: executing too eagerly and backend-specific logic.

deepyaman · 2026-03-21T20:06:33Z

pandera/api/base/error_handler.py

+        # Import is guarded so ibis remains an optional dependency.
+        try:
+            import ibis as _ibis
+            if isinstance(failure_cases, _ibis.Table):


Suggested change

if isinstance(failure_cases, _ibis.Table):

import ibis

if isinstance(failure_cases, ibis.Table):

Why alias, why not just import ibis?

deepyaman · 2026-03-21T20:18:22Z

pandera/api/base/error_handler.py

        if isinstance(failure_cases, str):  # Avoid returning str length
            return 1

+        # ibis.Table raises ExpressionError for len(); use .count().execute() instead.


It seems wrong to add an Ibis-specific branch to the base ErrorHandler; isinstance(failure_cases, ibis.table) made a lot more sense in the Ibis ErrorHandler, and this doesn't seem like the right way to handle it for Narwhals. Maybe the Narwhals backend needs it's own ErrorHandler, and that can dispatch based on type—or, much better, just use the Narwahls way of counting, not sure why this wouldn't work...

deepyaman · 2026-03-21T20:19:55Z

pandera/backends/ibis/register.py

+    Auto-detects narwhals: if narwhals is installed, registers narwhals backends
+    (NarwhalsCheckBackend, narwhals ColumnBackend, narwhals DataFrameSchemaBackend)
+    and emits a UserWarning. If narwhals is not installed, registers the native


Nit: Don't know why we're not capitalizing Narwhals.

Suggested change

Auto-detects narwhals: if narwhals is installed, registers narwhals backends

(NarwhalsCheckBackend, narwhals ColumnBackend, narwhals DataFrameSchemaBackend)

and emits a UserWarning. If narwhals is not installed, registers the native

Auto-detects Narwhals: if Narwhals is installed, registers Narwhals backends

(NarwhalsCheckBackend, Narwhals ColumnBackend, Narwhals DataFrameSchemaBackend)

and emits a UserWarning. If Narwhals is not installed, registers the native

deepyaman · 2026-03-21T20:21:07Z

pandera/backends/narwhals/base.py

+import narwhals.stable.v1 as nw
+import polars as pl
+
+from pandera.api.base.error_handler import ErrorHandler


Again, seems like we should be importing the Narwhals ErrorHandler here, rather than modifying the base one.

deepyaman · 2026-03-21T20:26:27Z

pandera/backends/narwhals/checks.py

+        )
+
+    @staticmethod
+    def _materialize(frame) -> nw.DataFrame:


This is eagerly executing? I think executing early for the check output is not ideal, but still reasonable. However, this is also getting called elsewhere above. Furthermore, the conditional logic seems overly complicated—not sure why this is needed.

deepyaman · 2026-03-21T22:13:26Z