Chat application benchmark using Kotlin with Ktor framework (Closes #428)

README.md

@@ -175,6 +175,10 @@ The following is the complete list of benchmarks, separated into groups.
   \
   Default repetitions: 12; APACHE2 license, MIT distribution; Supported JVM: 11 and later
 
+- `kotlin-ktor` - Simple Ktor chat application with multiple clients, performing various tasks.
+  \
+  Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later
+
 
 
 The suite also contains a group of benchmarks intended solely for testing

benchmarks/kotlin-ktor/README.md

@@ -0,0 +1,108 @@
+# Kotlin-Ktor benchmark
+
+The benchmark is designed to simulate a chat application where multiple clients send requests simultaneously.
+It uses the [Ktor](https://ktor.io/) framework.
+
+Namely, the benchmark runs multiple clients at the same time.
+Clients are set up with one or both of the following tasks:
+
+- Join a chat and send a randomized message to it.
+- Create a private chat with another user (if not already created) and send a randomized message to them.
+
+Clients repeat those tasks required number of times specified via `iterations_count` parameter.
+For the validation, each successful task completion is counted and compared to the expected number at the end of the
+run.
+
+It's also possible to implement validation via end-to-end comparison of the artifacts produced by the run.
+An example of such an artifact could be a log file of all sent/received messages or of completed tasks.
+However, in our case it would be tricky to implement, since every run is randomized both in content and in the order of
+operations, so a semantic comparison of logs would be required.
+Gladly, it's not needed for us, since each task auto-validates itself, so it will successfully terminate only if
+everything went as expected.
+Hence, counting the number of successfully completed tasks is enough to validate the run.
+
+## Running the benchmark
+
+To run benchmark, execute the following command:
+
+```bash
+java -jar <renaissance.jar> kotlin-ktor
+```
+
+Here are the parameters that can be passed to the benchmark:
+
+- `client_count`: Number of clients that are simultaneously sending the requests.
+  Increasing number of clients (with a proportional decrease in the number of repetitions) has little effect on the
+  overall runtime. Default value is the number of available CPUs.
+- `iterations_count`: Number of times clients should repeat their designated tasks. Default value is 2000.
+- `chat_count`: How many public chats should be set up for user interactions.
+  This simulates interactions in a few large chats.
+  Increasing the number of chats will reduce runtime, since it reduces contention between clients.
+  Default value is 10.
+- `group_message_fraction`: Fraction of clients which execute joinChatAndSendMessage. Default value
+  is 1.0.
+- `private_message_fraction`: Fraction of clients that should send private messages. Default value
+  is 0.5.
+- `random_seed`: Random seed to use for client tasks setup for reproducibility. Default value is 32.
+
+## Concurrency model
+
+The common way for concurrent programming in Kotlin is to use coroutines.
+The high-level idea of coroutines is that multiple blocking operations may run on the same thread pool.
+For more details, please see [the official coroutines guide](https://kotlinlang.org/docs/coroutines-overview.html)
+This is exactly what Ktor uses.
+
+Unfortunately, the Ktor framework does not provide a way neither to limit the number of threads used for internal
+operations nor to have separate thread pools for the client and server operations.
+Ktor, mostly uses the `Dispatchers.IO`and `Dispatchers.Default`, which operate on intersecting thread pools.
+The `Dispatchers.Default` has a thread pool of size equal to number of CPU cores, but it's at least 2.
+The `Dispatchers.IO` is a bit more complicated.
+By default, it has a thread pool of size 64 (which could be controlled via a system
+property `kotlinx.coroutines.io.parallelism`).
+However, it's possible to obtain a separate view of the `Dispatchers.IO` by using
+`Dispatchers.IO.limitedParallelism(n)`, which may create up to `n` additional threads.
+In theory the usage of the `Dispatchers.IO` should be reserved for IO-bound operations and the `Dispatchers.Default`
+should be used for CPU-bound operations.
+However, in practice, these rules are rather semantic and cannot and are not enforced by coroutines.
+So, Ktor may switch between these two dispatchers as it sees fit, with potential changes in the future versions.
+
+With this mind, we start the server using `start` method.
+To make sure that Ktor internal operations don't interfere much with us starting client jobs,
+we use a separate thread pool to start clients, though internally Ktor will switch to another thread pool.
+
+Considering there are enough available threads, both server and the client are capable of running mostly concurrently
+with a few synchronization points.
+Namely, if multiple clients are trying to perform the same operation on the same object
+(modifying the message history, chat pool, etc.), it will be processed sequentially.
+While operations such as sending a message to the server or routing the message on the server side and sending replies
+are parallel (up to contention for the system resources).
+
+It's possible to influence the amount of contention present by adjusting the parameters of the benchmark.
+Increasing the number of chats will decrease contention for the modification of the chat history (the opposite is true
+as well).
+Increasing group/private message fraction will increase contention for the chat history and the chat pool, respectively.
+Increasing the number of clients will increase contention in all synchronization points.
+
+## Core classes overview
+
+- `KtorRenaissanceBenchmark` is the root class which initialises, runs and tears down the server and the clients.
+  See [Running the benchmark](#running-the-benchmark) for more details on the available parameters for client and server
+  setup.
+- `ChatApplication` is the class which sets up the Ktor application for the `ChatServer`.
+  Namely, it installs the required plugins, such as `WebSockets`, and sets up routes from websockets calls to the server operations.
+- `ChatServer` is the class which handles the server side logic for the chat application. It is responsible for
+  maintaining the state of the chat application, such as the list of users, the list of chats, and the messages sent in
+  the chats. It also handles the logic for the different operations that the clients can perform, such as joining a
+  chat, sending a message, and sending a private message. Important detail is that the server will broadcast the message
+  which is sent to some chat. So, if the user sends a message to the chat, they will receive the message back, and
+  that's
+  how they'll now that the 'transaction' was successful.
+- `ClientManager` sets up clients according to the parameters received from the `KtorRenaissanceBenchmark`, runs them,
+  and tears them down.
+- `Client` is essentially represents a collection of tasks which are run sequentially for the required number of times.
+- `ClientTask` is the interface which represents a task that a client can perform.
+  So far, there are two types of client tasks:
+    - `GroupMessageTask` which represents the task of joining a chat and sending a message to it.
+    - `PrivateMessageTask` which represents the task of sending a private message to another user.
+- `Command` instances and `Message` are the data classes with which client and server communicate with each other. Both
+  classes are being sent as JSON objects.

benchmarks/kotlin-ktor/src/main/kotlin/org/renaissance/kotlin/ktor/KtorRenaissanceBenchmark.kt

@@ -0,0 +1,127 @@
+package org.renaissance.kotlin.ktor
+
+import io.ktor.server.engine.*
+import kotlinx.coroutines.*
+import org.renaissance.Benchmark
+import org.renaissance.Benchmark.*
+import org.renaissance.BenchmarkContext
+import org.renaissance.BenchmarkResult
+import org.renaissance.BenchmarkResult.Validators
+import org.renaissance.License
+import org.renaissance.kotlin.ktor.client.ClientManager
+import org.renaissance.kotlin.ktor.server.ChatApplication
+import kotlin.math.min
+
+@Group("web")
+@Name("kotlin-ktor")
+@Summary("Simple Ktor chat application with multiple clients, performing various tasks.")
+@Licenses(License.MIT)
+@Parameter(
+  name = "port",
+  defaultValue = "9496",
+  summary = "Port to run the server on."
+)
+@Parameter(
+  name = "client_count",
+  defaultValue = "\$cpu.count",
+  summary = "Number of clients that are simultaneously sending the requests"
+)
+@Parameter(
+  name = "iterations_count",
+  defaultValue = "2000",
+  summary = "Number of times clients should repeat their designated operations"
+)
+@Parameter(
+  name = "chat_count",
+  defaultValue = "10",
+  summary = "How many public chats should be setup for user interactions." +
+      "Reducing/increasing number of chats will increase/decrease runtime"
+)
+@Parameter(
+  name = "group_message_fraction",
+  defaultValue = "1.0",
+  summary = "Clients which execute joinChatAndSendMessage"
+)
+@Parameter(
+  name = "private_message_fraction",
+  defaultValue = "0.5",
+  summary = "How many public chats should be setup for user interactions"
+)
+@Parameter(
+  name = "random_seed",
+  defaultValue = "32",
+  summary = "Random seed to use for client tasks setup."
+)
+@Configuration(
+  name = "test",
+  settings = [
+    "iterations_count = 100",
+  ]
+)
+@Configuration(name = "jmh")
+class KtorRenaissanceBenchmark() : Benchmark {
+  private var port: Int = 0
+  private var clientCount: Int = 0
+  private var numberOfRepetitions: Int = 0
+  private var fractionOfClientsSendingPrivateMessages: Double = 0.0
+  private var fractionOfClientsSendingGroupMessages: Double = 0.0
+  private lateinit var clientPool: ExecutorCoroutineDispatcher
+  private lateinit var server: ApplicationEngine
+  private lateinit var application: ChatApplication
+  private lateinit var clientManager: ClientManager
+
+  @OptIn(DelicateCoroutinesApi::class)
+  override fun setUpBeforeAll(context: BenchmarkContext?) {
+    port = context!!.parameter("port").toPositiveInteger()
+    clientCount = context.parameter("client_count").toPositiveInteger()
+    numberOfRepetitions = context.parameter("iterations_count").toPositiveInteger()
+    val numberOfChats = context.parameter("chat_count").toPositiveInteger()
+    fractionOfClientsSendingGroupMessages = context.parameter("group_message_fraction").toDouble()
+    fractionOfClientsSendingPrivateMessages =
+      context.parameter("private_message_fraction").toDouble()
+    val randomSeed = context.parameter("random_seed").toPositiveInteger()
+
+    application = ChatApplication(numberOfChats)
+    server = embeddedServer(io.ktor.server.cio.CIO, host = "localhost", port = port) {
+      application.apply {
+        main()
+      }
+    }
+    server.start()
+
+    clientPool = newFixedThreadPoolContext(min(clientCount, Runtime.getRuntime().availableProcessors()), "clientPool")
+    clientManager = ClientManager(
+      port,
+      clientCount,
+      numberOfRepetitions,
+      fractionOfClientsSendingGroupMessages,
+      fractionOfClientsSendingPrivateMessages,
+      randomSeed,
+      CoroutineScope(clientPool)
+    )
+
+    super.setUpBeforeAll(context)
+  }
+
+  override fun setUpBeforeEach(context: BenchmarkContext?) {
+    application.setup()
+    clientManager.setupClients(application.getAvailableChatIds())
+  }
+
+  override fun run(context: BenchmarkContext?): BenchmarkResult {
+    val numberOfSuccessfulTasks = runBlocking {
+      clientManager.runClients()
+    }
+
+    return Validators.simple(
+      "Number of successful client tasks",
+      (((clientCount * fractionOfClientsSendingGroupMessages) + (clientCount * fractionOfClientsSendingPrivateMessages)) * numberOfRepetitions).toLong(),
+      numberOfSuccessfulTasks.toLong()
+    )
+  }
+
+  override fun tearDownAfterAll(context: BenchmarkContext?) {
+    server.stop()
+    clientPool.close()
+  }
+}

benchmarks/kotlin-ktor/src/main/kotlin/org/renaissance/kotlin/ktor/client/Client.kt

@@ -0,0 +1,76 @@
+package org.renaissance.kotlin.ktor.client
+
+import io.ktor.client.*
+import io.ktor.client.engine.cio.*
+import io.ktor.client.plugins.websocket.*
+import io.ktor.http.*
+import io.ktor.serialization.kotlinx.*
+import org.renaissance.kotlin.ktor.common.serializationFormat
+
+
+/**
+ * Client is an abstraction for the list of tasks to perform **sequentially**.
+ *
+ * For example:
+ * 1. Send a message to some User
+ * 2. Send picture to some chat
+ * 3. Join another chat
+ * 4. Leave the chat
+ */
+internal class Client private constructor(
+  private val httpClient: HttpClient,
+  private val port: Int,
+  private val userId: String,
+  private val operationsRepetitions: Int,
+  private val tasksToRun: List<ClientTask>,
+) {
+  /**
+   * Runs the client by establishing a WebSocket connection and performing the specified tasks.
+   *
+   * @return The number of successful tasks. Should be equal to `operationsRepetitions * tasksToRun.length`
+   */
+  suspend fun run(): Int {
+    var successfulTasks = 0
+    httpClient.webSocket(method = HttpMethod.Get, host = "localhost", port = port, path = "/ws/${userId}") {
+      tasksToRun.forEach { it.setup(this) }
+
+      for (i in 0..<operationsRepetitions) {
+        tasksToRun.forEach {
+          try {
+            it.run(this)
+            successfulTasks++
+          } catch (_: Throwable) {
+          }
+        }
+      }
+    }
+    return successfulTasks
+  }
+
+  class Builder(
+    private val port: Int,
+    val userId: String,
+    private val operationsRepetitions: Int,
+    private val httpClient: HttpClient = createDefaultClient()
+  ) {
+    private val tasksToRun: MutableList<ClientTask> = mutableListOf()
+
+    fun addTaskToRun(task: ClientTask): Builder {
+      tasksToRun.add(task)
+      return this
+    }
+
+    fun build(): Client {
+      return Client(httpClient, port, userId, operationsRepetitions, tasksToRun)
+    }
+  }
+}
+
+fun createDefaultClient(): HttpClient = HttpClient(CIO) {
+  engine {
+
+  }
+  install(WebSockets) {
+    contentConverter = KotlinxWebsocketSerializationConverter(serializationFormat)
+  }
+}

benchmarks/kotlin-ktor/src/main/kotlin/org/renaissance/kotlin/ktor/client/ClientManager.kt

@@ -0,0 +1,57 @@
+package org.renaissance.kotlin.ktor.client
+
+import io.ktor.client.*
+import io.ktor.util.*
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.CoroutineStart
+import kotlinx.coroutines.async
+import kotlinx.coroutines.awaitAll
+import kotlin.random.Random
+
+/**
+ * A **probabilistic** client manager, which sets up the appropriate number of clients, performing a specified number and kind of tasks
+ */
+class ClientManager(
+  private val port: Int,
+  numberOfClients: Int,
+  private val numberOfRequestsPerClient: Int,
+  private val fractionOfClientsSendingGroupMessages: Double,
+  private val fractionOfClientsSendingPrivateMessages: Double,
+  randomSeed: Int,
+  private val coroutineScope: CoroutineScope
+) {
+  private val random = Random(randomSeed)
+  private val userIds: Set<String> = (0..<numberOfClients).map { generateNonce() }.toSet()
+  private val predefinedHttpClients: MutableList<HttpClient> = mutableListOf()
+  private val clients: MutableList<Client> = mutableListOf()
+
+  fun setupClients(availableChatIds: List<String>) {
+    clients.clear()
+    predefinedHttpClients.forEach { it.close() }
+    predefinedHttpClients.clear()
+    predefinedHttpClients.addAll(userIds.indices.map { createDefaultClient() })
+
+    val clientBuilders = userIds.zip(predefinedHttpClients).map { (userId, httpClient) ->
+      Client.Builder(port, userId, numberOfRequestsPerClient, httpClient)
+    }.toMutableList()
+
+    clientBuilders.take((userIds.size * fractionOfClientsSendingGroupMessages).toInt()).forEach { builder ->
+      builder.addTaskToRun(JoinGroupAndSendMessageClientTask(availableChatIds.random(random), random))
+    }
+    clientBuilders.shuffle(random)
+
+    clientBuilders.take((userIds.size * fractionOfClientsSendingPrivateMessages).toInt()).forEach { builder ->
+      builder.addTaskToRun(DirectMessageClientTask(userIds.shuffled(random).first { it != builder.userId }, random))
+    }
+    clientBuilders.shuffle(random)
+
+    clients.addAll(clientBuilders.map { it.build() })
+  }
+
+  suspend fun runClients(): Int {
+    return clients.map { coroutineScope.async(start = CoroutineStart.LAZY) { it.run() } }
+      .map { it.also { it.start() } }
+      .awaitAll()
+      .sum()
+  }
+}

benchmarks/kotlin-ktor/src/main/kotlin/org/renaissance/kotlin/ktor/client/ClientTask.kt

@@ -0,0 +1,8 @@
+package org.renaissance.kotlin.ktor.client
+
+import io.ktor.client.plugins.websocket.*
+
+interface ClientTask {
+  suspend fun setup(session: DefaultClientWebSocketSession) {}
+  suspend fun run(session: DefaultClientWebSocketSession) {}
+}