Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
bd686fd
partition projection
badmintoncryer Oct 2, 2025
04e0cd4
update
badmintoncryer Oct 4, 2025
9fec278
refactor
badmintoncryer Oct 4, 2025
0e37abc
update
badmintoncryer Oct 4, 2025
37fa29e
refactor
badmintoncryer Oct 5, 2025
5c82132
refactor
badmintoncryer Oct 5, 2025
daa43b6
style
badmintoncryer Oct 9, 2025
ea65c76
update unit test
badmintoncryer Oct 11, 2025
0bff688
fix
badmintoncryer Oct 11, 2025
ff7100d
Merge branch 'main' into partition-projection
badmintoncryer Oct 11, 2025
1616b9a
Merge branch 'main' into partition-projection
badmintoncryer Oct 14, 2025
d0faff7
Merge branch 'main' into partition-projection
badmintoncryer Oct 16, 2025
74a732d
Merge branch 'main' into partition-projection
badmintoncryer Oct 23, 2025
981f9f7
Merge branch 'main' into partition-projection
badmintoncryer Oct 25, 2025
0bafc53
Merge branch 'main' into partition-projection
badmintoncryer Nov 11, 2025
aaa5478
Merge branch 'main' into partition-projection
badmintoncryer Nov 13, 2025
fa37108
Merge branch 'main' into partition-projection
badmintoncryer Nov 19, 2025
c2127cb
Merge branch 'main' into partition-projection
badmintoncryer Nov 28, 2025
9205962
Merge branch 'main' into partition-projection
badmintoncryer Dec 6, 2025
56f6497
Merge branch 'main' into partition-projection
badmintoncryer Dec 14, 2025
468a678
Merge branch 'main' into partition-projection
badmintoncryer Dec 21, 2025
10c2a85
Merge branch 'main' into partition-projection
badmintoncryer Dec 31, 2025
88a2a9e
Merge branch 'main' into partition-projection
badmintoncryer Jan 12, 2026
26c3ed6
chore: add .gitignore and project configuration for Serena
badmintoncryer Jan 18, 2026
66d80fb
Merge branch 'main' into partition-projection
badmintoncryer Jan 18, 2026
7bd7575
Merge remote-tracking branch 'origin/main' into partition-projection
badmintoncryer Feb 3, 2026
06ad88b
fix(glue): add validation for partition projection conflicts with man…
badmintoncryer Feb 3, 2026
673818a
chore: add .gitignore and project.yml for Serena configuration
badmintoncryer Feb 3, 2026
52009e4
fix(partition-projection): add validation for commas in ENUM partitio…
badmintoncryer Feb 4, 2026
9ce5326
fix(partition-projection): enhance validation for DATE format in part…
badmintoncryer Feb 4, 2026
a3067f1
Merge branch 'main' into partition-projection
badmintoncryer Feb 4, 2026
b83259d
fix(table-base): update import statements to use type imports for bet…
badmintoncryer Feb 4, 2026
67813c1
Merge remote-tracking branch 'cryer/partition-projection' into partit…
badmintoncryer Feb 4, 2026
c576d77
fix(partition-projection): standardize quote style in DATE validation…
badmintoncryer Feb 5, 2026
3c04683
Merge branch 'main' into partition-projection
mergify[bot] Feb 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 180 additions & 0 deletions packages/@aws-cdk/aws-glue-alpha/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -713,6 +713,186 @@ new glue.S3Table(this, 'MyTable', {
});
```

### Partition Projection

Partition projection allows Athena to automatically add new partitions as new data arrives, without requiring `ALTER TABLE ADD PARTITION` statements. This improves query performance and reduces management overhead by eliminating the need to manually manage partition metadata.

For more information, see the [AWS documentation on partition projection](https://docs.aws.amazon.com/athena/latest/ug/partition-projection.html).

#### INTEGER Projection

For partition keys with sequential numeric values:

```ts
declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
database: myDatabase,
columns: [{
name: 'data',
type: glue.Schema.STRING,
}],
partitionKeys: [{
name: 'year',
type: glue.Schema.INTEGER,
}],
dataFormat: glue.DataFormat.JSON,
partitionProjection: {
year: glue.PartitionProjectionConfiguration.integer({
min: 2020,
max: 2023,
interval: 1, // optional, defaults to 1
digits: 4, // optional, pads with leading zeros
}),
},
});
```

#### DATE Projection

For partition keys with date or timestamp values. Supports both fixed dates and relative dates using `NOW`:

```ts
declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
database: myDatabase,
columns: [{
name: 'data',
type: glue.Schema.STRING,
}],
partitionKeys: [{
name: 'date',
type: glue.Schema.STRING,
}],
dataFormat: glue.DataFormat.JSON,
partitionProjection: {
date: glue.PartitionProjectionConfiguration.date({
min: '2020-01-01',
max: '2023-12-31',
format: 'yyyy-MM-dd',
interval: 1, // optional, defaults to 1
intervalUnit: glue.DateIntervalUnit.DAYS, // optional: YEARS, MONTHS, WEEKS, DAYS, HOURS, MINUTES, SECONDS
}),
},
});
```

You can also use relative dates with `NOW`:

```ts
declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
database: myDatabase,
columns: [{
name: 'data',
type: glue.Schema.STRING,
}],
partitionKeys: [{
name: 'date',
type: glue.Schema.STRING,
}],
dataFormat: glue.DataFormat.JSON,
partitionProjection: {
date: glue.PartitionProjectionConfiguration.date({
min: 'NOW-3YEARS',
max: 'NOW',
format: 'yyyy-MM-dd',
}),
},
});
```

#### ENUM Projection

For partition keys with a known set of values:

```ts
declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
database: myDatabase,
columns: [{
name: 'data',
type: glue.Schema.STRING,
}],
partitionKeys: [{
name: 'region',
type: glue.Schema.STRING,
}],
dataFormat: glue.DataFormat.JSON,
partitionProjection: {
region: glue.PartitionProjectionConfiguration.enum({
values: ['us-east-1', 'us-west-2', 'eu-west-1'],
}),
},
});
```

#### INJECTED Projection

For custom partition values injected at query time:

```ts
declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
database: myDatabase,
columns: [{
name: 'data',
type: glue.Schema.STRING,
}],
partitionKeys: [{
name: 'custom',
type: glue.Schema.STRING,
}],
dataFormat: glue.DataFormat.JSON,
partitionProjection: {
custom: glue.PartitionProjectionConfiguration.injected(),
},
});
```

#### Multiple Partition Projections

You can configure partition projection for multiple partition keys:

```ts
declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
database: myDatabase,
columns: [{
name: 'data',
type: glue.Schema.STRING,
}],
partitionKeys: [
{
name: 'year',
type: glue.Schema.INTEGER,
},
{
name: 'month',
type: glue.Schema.INTEGER,
},
{
name: 'region',
type: glue.Schema.STRING,
},
],
dataFormat: glue.DataFormat.JSON,
partitionProjection: {
year: glue.PartitionProjectionConfiguration.integer({
min: 2020,
max: 2023,
}),
month: glue.PartitionProjectionConfiguration.integer({
min: 1,
max: 12,
digits: 2,
}),
region: glue.PartitionProjectionConfiguration.enum({
values: ['us-east-1', 'us-west-2'],
}),
},
});
```

### Glue Connections

Glue connections allow external data connections to third party databases and data warehouses. However, these connections can also be assigned to Glue Tables, allowing you to query external data sources using the Glue Data Catalog.
Expand Down
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-glue-alpha/lib/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ export * from './data-format';
export * from './data-quality-ruleset';
export * from './database';
export * from './external-table';
export * from './partition-projection';
export * from './s3-table';
export * from './schema';
export * from './security-configuration';
Expand Down
Loading
Loading