Skip to content

Made StackObjectPool's maxIdle parameter configurable in RedisClientPool#34

Merged
debasishg merged 1 commit intodebasishg:masterfrom
felipehummel:configurablePool
Sep 1, 2012
Merged

Made StackObjectPool's maxIdle parameter configurable in RedisClientPool#34
debasishg merged 1 commit intodebasishg:masterfrom
felipehummel:configurablePool

Conversation

@felipehummel
Copy link

The commit message says it all, but trying to explain better:

I was using RedisClientPool in a environment with 24 threads heavily using Redis (and doing other stuff). The problem is that, as maxIdle of StackObjectPool defaults to 8, when more than 8 objects were borrowed and then returned to the pool, this maxIdle threshold was hit. As ~24 objects are borrowed but pool only maintains up to 8 idle objects. That means that every time an object is returned and there is more than 8 idle objects in the pool, the bottom object in the stack pool is destroyed. Then causing a gigantic destroy-create-destroy-create cycle. As the number of Sockets being opened and closed to Redis increased, I guess, it also exhausted the ephemeral port range giving the error:

java.lang.RuntimeException: java.net.NoRouteToHostException: Cannot assign requested address

Changing the maxIdle parameter to 30 stopped all the problems. I've also check in the Redis log that a huge number of connections were being dropped and accepted again and again and again.

The maxIdle in StackObjectPool determines that when an object is returned
to the pool and there is more than maxIdle objects idle, then the bottom
object is destroyed and the returned object is inserted at the top of
the stack. With its default value of 8 and enough concurrent usage (more
than 8 threads) you can easily start creating and destroying a lot of
objects in sequence.

In high concurrency contexts one can increase maxIdle to the number of
threads to avoid this unnecessary destroying/creating.

The high number of creating/destroying can lead to exhausting ports
causing:
java.lang.RuntimeException: java.net.NoRouteToHostException: Cannot
assign requested address
@felipehummel
Copy link
Author

An alternative is to do something like:

class CustomRedisClientPool(host: String, port: Int, maxIdle: Int = 8) extends RedisClientPool(host, port) {
  override val pool = new StackObjectPool(new RedisClientFactory(host, port), maxIdle)
}

One problem is that RedisClientFactory is private[redis]

@ghost ghost assigned debasishg Sep 1, 2012
debasishg added a commit that referenced this pull request Sep 1, 2012
Made StackObjectPool's maxIdle parameter configurable in RedisClientPool
@debasishg debasishg merged commit 5586b89 into debasishg:master Sep 1, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants