[BUG] Netty Transport test failing with large responses

[saratvemulapalli]

Coming from #1834 (comment)

REPRODUCE WITH: ./gradlew ':modules:transport-netty4:test' --tests "org.opensearch.http.netty4.Netty4HttpServerTransportTests.testLargeCompressedResponse" -Dtests.seed=49B97FF7A35F7849 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sl -Dtests.timezone=Europe/Uzhgorod -Druntime.java=15

org.opensearch.http.netty4.Netty4HttpServerTransportTests > testLargeCompressedResponse FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=273, name=Thread-5, state=RUNNABLE, group=TGRP-Netty4HttpServerTransportTests]

        Caused by:
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([49B97FF7A35F7849]:0)
            at org.opensearch.http.netty4.Netty4HttpServerTransportTests$3.dispatchRequest(Netty4HttpServerTransportTests.java:357)

    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=275, name=Thread-6, state=RUNNABLE, group=TGRP-Netty4HttpServerTransportTests]

        Caused by:
        java.lang.AssertionError
            at __randomizedtesting.SeedInfo.seed([49B97FF7A35F7849]:0)
            at org.opensearch.http.netty4.Netty4HttpServerTransportTests$3.dispatchRequest(Netty4HttpServerTransportTests.java:357)

[meghasaik]

I ran this command for reproducing this flaky test:
for i in $(seq 0 250) ; do echo "Iteration: $i" && ./gradlew clean > /dev/null 2>&1 && ./gradlew ':modules:transport-netty4:test' --tests "org.opensearch.http.netty4.Netty4HttpServerTransportTests.testLargeCompressedResponse" -Dtests.seed=49B97FF7A35F7849 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sl -Dtests.timezone=Europe/Uzhgorod -Druntime.java=15 >> /Users/kavmegha/documents/test-output-2022-01-11.txt 2>&1 ; done

I tried running it in iterations of 50, 100, 150, 250 and 350. Still, this issue was not reproducible.
The Results:

As mentioned here: running this in isolation may not yield much in terms of reproducibility.

[owaiskazi19]

Tried running the test 400 times in isolation and it passed

./gradlew ':modules:transport-netty4:test' --tests "org.opensearch.http.netty4.Netty4HttpServerTransportTests.testLargeCompressedResponse" -Dtests.seed=49B97FF7A35F7849 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sl -Dtests.timezone=Europe/Uzhgorod -Druntime.java=15 -Dtests.iters=400

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6
  OS Info               : Linux 5.11.0-1020-aws (amd64)
  Runtime JDK Version   : 15 (Oracle JDK)
  Runtime java.home     : /usr/lib/jvm/java-15/jdk-15
  Gradle JDK Version    : 17 (Oracle JDK)
  Gradle java.home      : /usr/lib/jvm/java-17/jdk-17.0.1
  Random Testing Seed   : 49B97FF7A35F7849
  In FIPS 140 mode      : false
=======================================

BUILD SUCCESSFUL in 16m 56s

[dblock]

Let's close until we see it again.