Implement parameter validation framework

[ravi-kumar-pilla]

Description

Implement native parameter validation + typed instantiation in Kedro core, following the consolidated validation framework design. The core deliverable is that accessing context.params performs fail-fast validation and (when applicable) returns typed objects (e.g., Pydantic models / dataclasses) rather than raw dicts.

Context

The KEP proposes a validation framework integrated at the context level (KedroContext.params as the entry point), orchestrated by ParameterValidator, which:

1. Merges config parameters with runtime params
2. Discovers required types from pipeline node signatures via TypeExtractor + source filters
3. Instantiates typed objects via ModelFactory
4. Caches results for performance.

Possible Implementation

1. Context integration

• Implement _get_validated_params() (or equivalent) and wire it to KedroContext.params as described in the KEP
• Add a cache keyed by environment + runtime params hash (or a safe equivalent) so repeated context.params calls are cheap.

2. Type discovery

• Implement TypeExtractor to walk the active pipeline(s) and inspect node function signatures + annotations.
• Use a SourceFilter abstraction; in this ticket, focus on the ParameterFilter path (datasets can be added in future).

3. Model instantiation

• Implement ModelFactory to support:
  • Pydantic models when installed (optional dependency).
  • Python dataclasses.
  • Fallback behavior: if no type requirement exists, return raw values untouched.
• Provide clear errors via custom exceptions (e.g., a specialized ValidationError that includes param path + expected type + source node info).

Draft implementation - https://github.com/kedro-org/kedro/tree/test/validate_no_hook/kedro/framework/validation

Possible Alternatives

NA

[SajidAlamQB]

Parameter Validation Framework: Implementation Plan

Following discussion with @merelcht and @ravi-kumar-pilla, the original draft PR (#5371) is being broken down into smaller, incremental PRs each containing a single component with its tests.

Architecture

graph TD
    A[kedro run] --> B[KedroSession]
    B --> C[KedroContext]
    C --> D[_get_validated_params]
    D --> E[ParameterValidator]
    D --> F["config_loader['parameters']"]
    F --> G["OmegaConf.merge (runtime params)"]
    E --> H[TypeExtractor]
    H --> I["ParameterSourceFilter"]
    H --> J[Pipeline Node Signatures]
    E --> K[ModelFactory]
    K --> L[Pydantic Models]
    K --> M[Dataclasses]
    K --> N["Raw Values (no type hint)"]
    D --> O[Validated Params Cache]
    C --> P[_get_parameters]
    P --> Q["Catalog entries (params:key)"]

Loading

How it works:

1. KedroContext._get_validated_params() loads parameters from config (already merged with runtime --params by OmegaConf)
2. ParameterValidator asks TypeExtractor to walk all pipeline node signatures via ParameterSourceFilter to find typed parameter inputs (e.g. params:model_options with a Pydantic type hint)
3. For each typed parameter found, ModelFactory instantiates the appropriate model (Pydantic, dataclass, or returns raw value for unsupported types)
4. Validated parameters are cached and registered in the catalog as MemoryDataset entries

Validation is opt-in by design — projects without type-hinted parameters are completely unaffected.

PR Checklist

PRs are ordered by dependency each builds on the previous. All PRs target main and live under kedro/validation.

• [PR 1: Exceptions + utils](Add validation framework exceptions and pydantic utilities #5403) — ValidationError, ModelInstantiationError, is_pydantic_model, is_pydantic_class (no dependencies)
• [PR 2: Source filters + tests](Add validation source filters #5404) — SourceFilter ABC, ParameterSourceFilter (depends on PR 1)
• [PR 3: Model factory + tests](Add validation model factory #5419) — ModelFactory with Pydantic/dataclass/raw value handling (depends on PR 1)
• [PR 4: Type extractor + tests](Add validation type extractor #5421) — TypeExtractor, pipeline walking, node signature inspection, nested path helpers (depends on PR 1, 2)
• [PR 5: Parameter validator + tests](Add validation parameter validator #5422) — ParameterValidator orchestrating extraction and validation (depends on PR 1–4)
• [PR 6: Context integration + tests](Add validation context integration #5423) — KedroContext changes, _get_validated_params, _get_parameters nested path registration (depends on PR 5)

Reference

• Draft PR with full implementation: [Reference] Implement parameter validation framework #5371
• Stress test project: https://github.com/SajidAlamQB/kedro-parameter-validation-test
• KEP discussion: [KEP-1] Parameter Validation #5234
• Original spike: Spike: Explore implementation options for native Pydantic parameter validation #5161
• Tests PR (will be absorbed into individual PRs): Add tests for validation framework #5386
• Docs: Documentation for Parameter Validation framework #5315

Key design decisions

• Opt-in: No typed params = no validation, zero overhead
• Fail-fast: Bad data caught during catalog creation, before any node runs
• Config loader handles merging: OmegaConf.merge deep-merges runtime params, validator receives a single merged dict
• Nested types preserved: Shallow field access (not model_dump()/asdict()) preserves nested Pydantic sub-model types in catalog entries
• Conflicting types warned: If two pipelines define different types for the same parameter key, a warning is logged
• MemoryDataset deepcopies: Pydantic/dataclass instances are deepcopied on load, so nodes don't share mutable state