[Remote Store] BlobStore used in Remote Store doesn't use updated settings and repo metadata

[gbbafna]

Describe the bug

For RemoteStore, once we create BlobStore , we don't close it ever and reuse it for nodes' lifetime. That means the plugin level settings and repo metadata update doesn't get applied to it .

There are two issues which we need to fix here :

1. When we update repository, Repositories Service closes existing BlobRepository , BlobStore and opens a new one . However RemoteStore continues to use existing BlobStore which has no clue on new metadata .

2. We never reload the BlobStoreRepository in place. This means that cluster settings update also doesn't get applied on repo.

We need to a way to update the Remote Repositories in place, so that updated settings and repo metadata is reflected .

Expected behavior
A clear and concise description of what you expected to happen.

[BhumikaSaini-Amazon]

@gbbafna I was looking at the S3 implementation. In my understanding, the following values shouldn't be reloadable because that could possibly break things in the remote store flows:

OpenSearch/plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3Repository.java

Lines 260 to 313 in 2a5b124

	// Parse and validate the user's S3 Storage Class setting
	this.bucket = BUCKET_SETTING.get(metadata.settings());
	if (bucket == null) {
	throw new RepositoryException(metadata.name(), "No bucket defined for s3 repository");
	}
	this.bufferSize = BUFFER_SIZE_SETTING.get(metadata.settings());
	this.chunkSize = CHUNK_SIZE_SETTING.get(metadata.settings());
	// We make sure that chunkSize is bigger or equal than/to bufferSize
	if (this.chunkSize.getBytes() < bufferSize.getBytes()) {
	throw new RepositoryException(
	metadata.name(),
	CHUNK_SIZE_SETTING.getKey()
	+ " ("
	+ this.chunkSize
	+ ") can't be lower than "
	+ BUFFER_SIZE_SETTING.getKey()
	+ " ("
	+ bufferSize
	+ ")."
	);
	}
	final String basePath = BASE_PATH_SETTING.get(metadata.settings());
	if (Strings.hasLength(basePath)) {
	this.basePath = new BlobPath().add(basePath);
	} else {
	this.basePath = BlobPath.cleanPath();
	}
	this.serverSideEncryption = SERVER_SIDE_ENCRYPTION_SETTING.get(metadata.settings());
	this.storageClass = STORAGE_CLASS_SETTING.get(metadata.settings());
	this.cannedACL = CANNED_ACL_SETTING.get(metadata.settings());
	if (S3ClientSettings.checkDeprecatedCredentials(metadata.settings())) {
	// provided repository settings
	deprecationLogger.deprecate(
	"s3_repository_secret_settings",
	"Using s3 access/secret key from repository settings. Instead "
	+ "store these in named clients and the opensearch keystore for secure settings."
	);
	}
	logger.debug(
	"using bucket [{}], chunk_size [{}], server_side_encryption [{}], buffer_size [{}], cannedACL [{}], storageClass [{}]",
	bucket,
	chunkSize,
	serverSideEncryption,
	bufferSize,
	cannedACL,
	storageClass
	);

Is there a known use case where we want these too to be reloadable?

Thanks!

[gbbafna]

Hi @BhumikaSaini-Amazon , Currently these are also reloadable. However we are restricting changing these for system repositories : #9839 .

For snapshot changing the path does work - it just works like snapshot repository is cleaned up and you start from scratch .