Optimize read write lock constructs during translog upload to remote store

[ashking94]

Is your feature request related to a problem? Please describe.
While performing OpenSearch Benchmark runs, discovered a potential optimisation that obstructs the add to translog when the translog upload to remote store is ongoing.

Translog add :

OpenSearch/server/src/main/java/org/opensearch/index/translog/Translog.java

Lines 541 to 582 in 443cfca

	public Location add(final Operation operation) throws IOException {
	final ReleasableBytesStreamOutput out = new ReleasableBytesStreamOutput(bigArrays);
	try {
	final long start = out.position();
	out.skip(Integer.BYTES);
	writeOperationNoSize(new BufferedChecksumStreamOutput(out), operation);
	final long end = out.position();
	final int operationSize = (int) (end - Integer.BYTES - start);
	out.seek(start);
	out.writeInt(operationSize);
	out.seek(end);
	final BytesReference bytes = out.bytes();
	try (ReleasableLock ignored = readLock.acquire()) {
	ensureOpen();
	if (operation.primaryTerm() > current.getPrimaryTerm()) {
	assert false : "Operation term is newer than the current term; "
	+ "current term["
	+ current.getPrimaryTerm()
	+ "], operation term["
	+ operation
	+ "]";
	throw new IllegalArgumentException(
	"Operation term is newer than the current term; "
	+ "current term["
	+ current.getPrimaryTerm()
	+ "], operation term["
	+ operation
	+ "]"
	);
	}
	return current.add(bytes, operation.seqNo());
	}
	} catch (final AlreadyClosedException | IOException ex) {
	closeOnTragicEvent(ex);
	throw ex;
	} catch (final Exception ex) {
	closeOnTragicEvent(ex);
	throw new TranslogException(shardId, "Failed to write operation [" + operation + "]", ex);
	} finally {
	Releasables.close(out);
	}
	}

RemoteFsTranslog prepareForUpload :

OpenSearch/server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

Lines 239 to 251 in 443cfca

	public boolean ensureSynced(Location location) throws IOException {
	try (ReleasableLock ignored = writeLock.acquire()) {
	assert location.generation <= current.getGeneration();
	if (location.generation == current.getGeneration()) {
	ensureOpen();
	return prepareAndUpload(primaryTermSupplier.getAsLong(), location.generation);
	}
	} catch (final Exception ex) {
	closeOnTragicEvent(ex);
	throw ex;
	}
	return false;
	}

[msfroh]

While performing OpenSearch Benchmark runs, discovered a potential optimisation that obstructs the add to translog when the translog upload to remote store is ongoing.

Can you clarify what the optimization is?

[ashking94]

While performing OpenSearch Benchmark runs, discovered a potential optimisation that obstructs the add to translog when the translog upload to remote store is ongoing.

Can you clarify what the optimization is?

Currently the translog upload to remote store is happening within the write lock. I am trying to break it up and move the upload within read lock. #9636 is an attempt to do the same.