Missing AWS Profile Support in PyIceberg (for PyArrow backend)

[thomas-pfeiffer]

Feature Request / Improvement

Feature: Missing AWS Profile Support in PyIceberg (for PyArrow backend) / PyArrow backend should support AWS profiles (follow up issue for #2841 )

Description:
When working with multiple AWS configs / credentials in parallel, AWS profiles are a convenient way to achieve this. Ideally, PyArrow backend in PyIceberg should therefore also support AWS profiles, which currently only the Fsspec backend does support.

Current state (as of writing - pyIceberg v0.11.0):

• For fsspec backends, AWS profile support is implemented (see Missing AWS Profile Support in PyIceberg #2841)
• For PyArrow backend, the AWS profile support is not yet available, but the PyArrow team do have an enhancement ticket for it (see [Python][C++] Add Profile support to S3FileSystem arrow#47880). Once AWS profile is supported in PyArrow it can be implemented in PyIceberg as well, I assume.

(Unverified) Workaround for this feature gap:

session = Session(profile_name="your_aws_profile")
credentials = session.get_credentials()  
if credentials is None:
    raise ValueError("Could not retrieve credentials for profile")
catalog = GlueCatalog(
    name="your_glue_catalog",
    **{ 
        "client.access-key-id": credentials.access_key,
        "client.secret-access-key": credentials.secret_key,
        "client.session-token": credentials.token,
        ...
    },
)

or use Fsspec backend instead of PyArrow backend for now.

To-Be / Expected Behavior:
Once PyArrow supports AWS profile names (see apache/arrow#47880), AWS profile support should be implemented for PyArrow backend as well and client.profile-name and s3.profile-name should be fully supported.

Remark: This is a follow up issue for #2841 .
Issues possibly related to this issue: #570, #1207, #2657