added basic benchmark testing suite and corresponding notebook

notebooks/Syft - Testing - Benchmark Tests.ipynb

@@ -0,0 +1,217 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Testing: Benchmark Tests"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "One goal of the OpenMined project is to **efficiently** train Deep Learning models in a homomorphically encrypted state. Therefore it is very important to benchmark new and existing features in order to achieve better and faster implementations."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Installation\n",
+    "\n",
+    "Simpy run `pip install -r test-requirements.txt` instead of the regular `requirements.txt` to get all testing tools."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Usage"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Before using the Benchmark Testing Suite, you have to import it from `syft.test.benchmark`. After that, you can pass in the function which needs benchmark testing."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<syft.test.benchmark.Benchmark at 0x7fb2c0094b00>"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from syft.test.benchmark import Benchmark\n",
+    "\n",
+    "Benchmark(str)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "### Compute a function's execution time"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "The `exec_time` method is a very basic tool to calculate a functions' execution time. The method can be used as follows."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "EXECUTION TIME: [3.0019264221191406] SECONDS\n"
+     ]
+    }
+   ],
+   "source": [
+    "import time\n",
+    "\n",
+    "def wait_a_second(seconds=3):  # Define a function for testing or use an existing one\n",
+    "    time.sleep(seconds)\n",
+    "    \n",
+    "# Call function without params\n",
+    "exec_time = Benchmark(wait_a_second).exec_time()\n",
+    "\n",
+    "print(\"EXECUTION TIME: {} SECONDS\".format(exec_time))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "So as we see, the function returns the execution time of the function in seconds. Additional params can be added to the function call as follows."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(2) EXECUTION TIME: [1.001023292541504] SECONDS\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Call functions with params\n",
+    "exec_time = Benchmark(wait_a_second, seconds=1).exec_time()  # Pass function and params to be tested into class\n",
+    "\n",
+    "print(\"(2) EXECUTION TIME: {} SECONDS\".format(exec_time))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Profile Execution Times Per Line"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It is possible to get the execution time per line using the `profile_lines()` method."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Timer unit: 1e-06 s\n",
+      "\n",
+      "Total time: 1e-05 s\n",
+      "File: <ipython-input-4-4317e0387243>\n",
+      "Function: some_function at line 1\n",
+      "\n",
+      "Line #      Hits         Time  Per Hit   % Time  Line Contents\n",
+      "==============================================================\n",
+      "     1                                           def some_function(count):\n",
+      "     2         1            1      1.0     10.0      a = 6*8\n",
+      "     3         1            0      0.0      0.0      b = 6**3\n",
+      "     4         1            0      0.0      0.0      c = a + b\n",
+      "     5         1            0      0.0      0.0      x = [a, b, c]\n",
+      "     6         6            4      0.7     40.0      for i in range(count):\n",
+      "     7         5            5      1.0     50.0          a += x[0] * i + x[1] + x [2] \n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "def some_function(count):\n",
+    "    a = 6*8\n",
+    "    b = 6**3\n",
+    "    c = a + b\n",
+    "    x = [a, b, c]\n",
+    "    for i in range(count):\n",
+    "        a += x[0] * i + x[1] + x [2] \n",
+    "\n",
+    "Benchmark(some_function, count=5).profile_lines()"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [conda env:venvP1]",
+   "language": "python",
+   "name": "conda-env-venvP1-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.5.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

syft/test/benchmark.py

@@ -0,0 +1,36 @@
+from time import time
+from line_profiler import LineProfiler
+
+class Benchmark():
+    """
+    This is a testing class for the benchmarking of functions.
+    Input of the Benchmark class should be the function to test and the params (optional)
+    """
+
+    def __init__(self, function, **params):
+        self.function = function
+        self.params = params
+
+    def exec_time(self, reps=1):
+        """
+        Calls function x-times and returns an array of computed execution times
+        """
+        results = []
+        for rep in range(reps):
+            t0 = time()
+            self.function(**self.params)
+            t1 = time()
+            results.append(t1 - t0)
+
+        return results
+
+    def profile_lines(self):
+        """
+        A simple wrapper to call the line_profiler.
+        Prints the line_profiler output
+        """
+        lp = LineProfiler()
+        lp_wrapper = lp(self.function)
+        lp_wrapper(**self.params)
+        lp.print_stats()
+

test-requirements.txt

@@ -0,0 +1,4 @@
+-r ./requirements.txt
+
+# Everything for testing purposes
+line_profiler