feat: add column mapping support when reading tables#3954
feat: add column mapping support when reading tables#3954JonatanMartens wants to merge 7 commits intodelta-io:mainfrom
Conversation
JonatanMartens
requested review from
hntd187,
ion-elgreco,
roeap and
rtyler
as code owners
github-actions
bot
added
binding/python
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
JonatanMartens
changed the title
| if ( | ||
| table_protocol.reader_features | ||
| and "columnMapping" in table_protocol.reader_features | ||
| ): | ||
| raise DeltaProtocolError( | ||
| "The table requires reader feature 'columnMapping' " | ||
| "but this is not supported using pyarrow Datasets." | ||
| ) | ||
|
|
||
| if ( |
There was a problem hiding this comment.
Why is this removed? The pr itself doesn't provide support for reading column mapping with pyarrow
There was a problem hiding this comment.
My primary use case is loading delta tables with datafusion (in python not rust), so must have missed it.
Will look into adding support for reading with pyarrow
There was a problem hiding this comment.
I have opened a pr to arrow apache/arrow#48289, once that is merged we can support column mapping in datasets.
There was a problem hiding this comment.
I think it's worthwhile to get this merged in and throw an error on pyarrow usage, I have no clue what percentage of using pyarrow but I think it's likely considered legacy
There was a problem hiding this comment.
@abhiaagarwal sounds good to me.
@ion-elgreco what do you think? We could print a warning when initializing the table in python (if it column mapping is enabled) and throw an error if the user tries to use the pyarrow methods
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3954 +/- ##
===========================================
+ Coverage 25.76% 37.27% +11.51%
===========================================
Files 127 87 -40
Lines 20540 13257 -7283
Branches 20540 13257 -7283
===========================================
- Hits 5292 4942 -350
+ Misses 14886 7984 -6902
+ Partials 362 331 -31 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
JonatanMartens
force-pushed
the
column-mapping
branch
from
4a53ecc to
edb5739
Compare
3 participants
Description
This pr adds the ability to read tables with the column mapping feature enabled.
Related Issue(s)
Partially implements #930, but lacks the ability to write to tables with column mapping.
Documentation