Add Athena partition projection docs#13238
Conversation
There was a problem hiding this comment.
is there a default somehow for each table?
There was a problem hiding this comment.
From looking at the code, the default value is null, but that null behaves identically to false because it's read in as an Optional and then used in some checks with <property>.orElse(false). So you do need to set this to true to enable it for a table. Should we say it's default false in the docs (because for our users, it may as well be), then submit a follow-up PR to explicitly set a default of false and get rid of the Optional? (I can handle that)
cc @aczajkowski
There was a problem hiding this comment.
It's a little more complex even. Value is optional as we are not owners of it. It may be pre existing in metastore.
So we read it as optional and:
- if Optional is empty - Projection is not active. Thats why in condition there is
<property>.orElse(false) - if Optional is not empty
- if value is false - Projection is there but disabled. We only validate all Projection table and column Properties. If there is some issue users may add Trino owned property
partition_projection_ignore=trueto by pass compatibility issues - if value is true - Projection is enabled. We validate all Projection table and column Properties and use Partition Projection
- if value is false - Projection is there but disabled. We only validate all Projection table and column Properties. If there is some issue users may add Trino owned property
All above is due to fact that when in AWS Athena projection.enabled=false other properties are not validated and may contain invalid values.
And as we map them to Trino properties and the other way round we may run into issues.
So we prefere to always validate if projection.enabled is there. Or tell Trino to ignore that table projection at all.
035db4e to
b11650d
Compare
b11650d to
7e43705
Compare
|
Can you maybe look at this and merge @kokosing ? |
|
CI fails |
|
@colebow can you confirm that you built the docs locally and the CI failure is a false alarm? |
|
@mosabua Ive builded locally as well. |
|
I restarted the job. Please report the flaky test as GitHub issue. We should not just ignore flaky failures. |
|
@kokosing it passed this time around. I'll try to look into why that test might be flaky. |
Description
Documentation for Athena partition projection feature. Edit of #13232
Related issues, pull requests, and links
Documentation
( ) No documentation is needed.
(x) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.
Release notes
(x) No release notes entries required.
( ) Release notes entries required with the following suggested text: