[FEATURE] Create a PPL compatibility verification framework #32
Description
Is your feature request related to a problem?
Yes. When implementing the PPL language on top of various query engines (e.g., OpenSearch, Spark, Trino), it’s crucial to assess whether the behavior conforms to the PPL specification. Currently, there is no standardized framework to validate PPL query compatibility across engines. This makes it hard to detect semantic mismatches in areas like type coercion, null handling, expression evaluation, or operator behavior.
Without such a framework, each backend must create its own ad hoc tests, leading to duplicated effort, inconsistent coverage, and gaps in spec compliance.
What solution would you like?
Introduce a PPL compatibility test framework that includes:
- A reference test dataset shared across engines
- A suite of PPL queries covering core syntax and semantics (e.g., source, eval, where, stats, sort, etc.)
- Expected result files for each query
- A lightweight test runner that executes the queries against a given backend and compares the actual output to the expected results
This framework would help backend implementers verify conformance to the PPL language spec and avoid regressions when introducing new features.
What alternatives have you considered?
- Each backend sticks with its current backend-specific test suites as below
Do you have any additional context?
References
- PartiQL test suite: https://github.com/partiql/partiql-tests
- ZetaSQL compliance test suite: https://github.com/google/zetasql
Current PPL IT
There are currently several PPL-related test suites across the opensearch-sql and opensearch-spark repositories. However, these are tightly coupled to specific engines and not designed as a general-purpose compatibility framework.
- OpenSearch
- Spark
- Spark PPL IT: https://github.com/opensearch-project/opensearch-spark/tree/main/integ-test/src/integration/scala/org/opensearch/flint/spark/ppl
- Spark PPL e2e IT: https://github.com/opensearch-project/opensearch-spark/blob/main/e2e-test/src/test/scala/org/opensearch/spark/e2e/EndToEndITSuite.scala