[BUG] Default value of parameter shard_path_type changed in 3.x

[RamHaridas]

Describe the bug

Snapshot registration API has a parameter shard_path_type which determines how the blobs will be created in the storage service.

POST /_snapshot/<repository>/ 
PUT /_snapshot/<repository>/

parameter shard_path_type is set to HASHED_PREFIX by default

This is a major change as it causes to upload the blobs outside base_path in storage services like S3/MinIO

Related component

Storage:Snapshots

To Reproduce

1. Install OpenSearch 3.5.0
2. Register a Snapshot registry PUT /_snapshot/<repository>/ with default values
3. Take snapshot
4. Verify blobs getting created outside base path in S3/MinIO
5. Set the param shard_path_type to FIXED and register repository again.
6. Take a snapshot
7. Now verify the storage service, the blobs will be created inside base_path (which is the default expected behavior)### Expected behavior

Until OpenSearch Version 2.x, the blobs were created inside the base path but now since the default value of shard_path_type is changed to HASHED_PREFIX, we are observing this new behavior which really affects when someone is upgrading from version 2.x to 3.x.
This cause failure in incremental backup for indices from 2.x to 3.x after performing an upgrade.

Additional Details

Plugins
s3-repository-plugin

Screenshots

Host/Environment (please complete the following information):

• OS: Rocky 8 Linux
• Version: 3.5.0

Additional context
We wanted to understand the reason for change in this behavior.

[andrross]

Catch All Triage - 1

@Bukhtawar @gbbafna Can you help find someone to take a look at this and the linked PR?

[Bukhtawar]

I belive we should keep the default to FIXED and only change once we have backward compatibility supported.

[AndreVirtimo]

The PR #18163 changed this default value. For me this was a breaking change! It wasn't even mentioned in the changelog.

[ejain]

This change cost me several hours today 🙃

I'm sure there was a warning in the change log, had I bothered to read it, but I'm still puzzled how this new default behavior makes sense: Dumping files outside of the specified repository path (and possibly mixing them with files from other repositories) seems kind of dangerous?

[AndreVirtimo]

I'm sure there was a warning in the change log,

This change wasn't mentioned in any change log. The PR author (@ashking94 ) added the label skip-changelog to the PR #18163 .

We also had several hours of extra work caused by this change.