Merge pull request #70 from centaurean/dev

Add MSVC support and continuous integration

Author: g1mv

.travis.yml

@@ -13,4 +13,4 @@ script:
   - tar zxvf premake.tar.gz
   - ./premake5 gmake
   - make
-  - ./bin/release/benchmark -f
+  - ./bin/Release/benchmark -f

CHANGELOG.md

@@ -1,3 +1,11 @@
+0.14.1
+------
+*January 20, 2018*
+- Added MSVC support
+- Added continuous integration on travis and appveyor
+- Premake script improvement
+- Various codebase improvements
+
 0.14.0
 ------
 *January 16, 2018*

README.md

@@ -1,4 +1,4 @@
-DENSITY
+DENSITY
 ========
 Superfast compression library
 
@@ -8,6 +8,11 @@ It is focused on high-speed compression, at the best ratio possible. **All three
 
 DENSITY features a simple API to enable quick integration in any project.
 
+Branch|Linux & OSX|Windows
+--- | --- | ---
+master|[![Build Status](https://travis-ci.org/centaurean/density.svg?branch=master)](https://travis-ci.org/centaurean/density)|[![Build status](https://ci.appveyor.com/api/projects/status/rf7x3x829il72cii/branch/master?svg=true)](https://ci.appveyor.com/project/gpnuma/density/branch/master)
+dev|[![Build Status](https://travis-ci.org/centaurean/density.svg?branch=dev)](https://travis-ci.org/centaurean/density)|[![Build status](https://ci.appveyor.com/api/projects/status/rf7x3x829il72cii/branch/dev?svg=true)](https://ci.appveyor.com/project/gpnuma/density/branch/dev)
+
 Why is it so fast ?
 -------------------
 
@@ -68,64 +73,99 @@ Build
 -----
 DENSITY can be built on a number of platforms. It uses the [premake](http://premake.github.io/) build system.
 
-It was developed and optimized against Clang/LLVM, but GCC is also supported (please use a recent version for maximum performance).
-At this time MSVC is *not* supported, but it's straightforward to install clang for windows.
-
-The following assumes you already have *git* installed.
+It was developed and optimized against Clang/LLVM which makes it the preferred compiler, but GCC and MSVC are also supported. Please use the latest compiler versions for best performance.
 
 **Mac OS X**
 
 On OS X, Clang/LLVM is the default compiler, which makes things simpler.
 
-1) Download and install [premake5](http://premake.github.io/) for OS X and make it available in your path.
+1) First things first, install the [homebrew package manager](https://brew.sh/).
+
+2) Then the prerequisites from the command line :
+
+```
+    brew install git wget
+```
+
+3) Now the source code, and premake :
+
+```
+    git clone https://github.com/centaurean/density.git
+    cd density/build
+    wget https://github.com/premake/premake-core/releases/download/v5.0.0-alpha12/premake-5.0.0-alpha12-macosx.tar.gz -O premake.tar.gz
+    tar zxvf premake.tar.gz
+```
 
-2) Run the following from the command line :
+5) Finally, build and test :
 
 ```
-    cd build
-    premake5 gmake
+    ./premake5 gmake
     make
+    bin/Release/benchmark -f
 ```
 
 **Linux**
 
-On Linux, Clang/LLVM is not always available by default.
+On Linux, Clang/LLVM is not always available by default, but can be easily added thanks to the provided package managers.
+The following example assumes a Debian or Ubuntu distribution with *apt-get*.
 
-1) Install Clang/LLVM *(optional if GCC is installed)* if you don't have it already (on Debian distributions for example, *sudo apt-get install clang* should do the trick)
+1) From the command line, install Clang/LLVM (*optional*, GCC is also supported if Clang/LLVM can't be used) and other prerequisites.
 
-2) Download and install [premake5](http://premake.github.io/) for Linux and make it available in your path.
+```
+    sudo apt-get install clang git wget
+```
 
-3) Run the following from the command line :
+2) Get the source code and premake :
 
 ```
-    cd build
-    premake5 gmake
+    git clone https://github.com/centaurean/density.git
+    cd density/build
+    wget https://github.com/premake/premake-core/releases/download/v5.0.0-alpha12/premake-5.0.0-alpha12-linux.tar.gz -O premake.tar.gz
+    tar zxvf premake.tar.gz
+```
+
+3) And finally, build and test :
+
+```
+    ./premake5 gmake
     make
+    bin/Release/benchmark -f
 ```
 
 **Windows**
 
-On Windows, things are a (little) bit more complicated.
+On Windows, things are (slightly) more complicated. We'll use the [chocolatey package manager](https://chocolatey.org/) for simplicity.
 
-1) First, check if you have git installed, if not you can [get it here](https://www.git-scm.com/download/win). Make it available in your path.
+1) Please install [chocolatey](https://chocolatey.org/install).
 
-2) We'll use *GNU make* to build the project, on Windows it's not part of the default toolset but you can [download a port here](http://gnuwin32.sourceforge.net/packages/make.htm). Make it available in your path.
+2) From the command line, get the prerequisites :
 
-3) Microsoft Visual Studio is required as we will use its linker, [get it here](https://www.visualstudio.com/en-us/products/free-developer-offers-vs.aspx).
+```
+    choco install git wget unzip
+```
 
-4) Now we can install Clang/LLVM for Windows, and thanks to the **ClangOnWin** project it's easy ! You can [download it here](http://sourceforge.net/projects/clangonwin/). Make it available in your path.
+3) Install the [Visual Studio IDE community edition](https://www.visualstudio.com/thank-you-downloading-visual-studio/?sku=Community) if you don't already have it.
 
-5) Download and install [premake5](http://premake.github.io/) for Windows and make it available in your path (edit the Path environment variable).
+4) Download and install [Clang/LLVM for Windows](http://releases.llvm.org/5.0.1/LLVM-5.0.1-win64.exe) (*optional*, MSVC compiler is also supported but resulting binaries might be slower).
 
-6) Run the following from the command line :
+5) Open a [developer command prompt](https://docs.microsoft.com/en-us/dotnet/framework/tools/developer-command-prompt-for-vs) and type :
 
 ```
-    cd build
-    premake5.exe gmake
-    make
+    git clone https://github.com/centaurean/density.git
+    cd density/build
+    wget https://github.com/premake/premake-core/releases/download/v5.0.0-alpha12/premake-5.0.0-alpha12-windows.zip -O premake.zip
+    unzip premake.zip
+```
+
+6) Build and test :
+
+```
+    premake5.exe vs2017
+    msbuild Density.sln
+    bin\Release\benchmark.exe -f
 ```
 
-And that's it ! You can now use the integrated in-memory benchmark to test your files (the *benchmark* or *benchmark.exe* binary in the build/bin/Release/ folder).
+And that's it ! If you open the Visual Studio IDE, in the solutions explorer by right-clicking on project names you can change platform toolsets, if you also want to try other compilers.
 
 Output format
 -------------

appveyor.yml

@@ -1,30 +1,22 @@
-version: 0.14.0.{build}
+version: 0.14.1.{build}
 image: Visual Studio 2015
 init:
 - cmd:
 install:
 - cmd: >-
-    set PATH=C:\Program Files\7-Zip;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\usr\bin;C:\projects\density\build\mingw64\bin\
-
-    choco install wget
+    choco install wget unzip
 
     cd build
 
     wget https://github.com/premake/premake-core/releases/download/v5.0.0-alpha12/premake-5.0.0-alpha12-windows.zip -nv -O premake5.zip
 
-    7z x premake5.zip
+    unzip premake5.zip
 
     del premake5.zip
-
-    wget https://qa.debian.org/watch/sf.php/mingw-w64/x86_64-7.2.0-release-posix-seh-rt_v5-rev1.7z -nv --no-check-certificate -O mingw.7z
-
-    7z x mingw.7z
-
-    del mingw.7z
 build_script:
 - cmd: >-
-    premake5.exe gmake
+    premake5.exe vs2015
 
-    mingw32-make.exe CC=gcc.exe
+    msbuild.exe Density.sln
 test_script:
 - cmd: bin\Release\benchmark.exe -f

benchmark/src/benchmark.c

@@ -125,7 +125,7 @@ int main(int argc, char *argv[]) {
         uint8_t value = (uint8_t) rand();
         for (unsigned int count = 0; count < uncompressed_size; count++) {
             if (!(rand() & 0xf))
-                value += rand();
+                value += (uint8_t)rand();
             in[count] = value;
         }
         out = malloc(memory_allocated * sizeof(uint8_t));
@@ -229,11 +229,11 @@ int main(int argc, char *argv[]) {
         double total_compress_time = 0.0;
         double total_decompress_time = 0.0;
         double total_time = 0.0;
-        double decompress_speed;
-        double decompress_speed_low;
-        double decompress_speed_high;
-        double compress_time_elapsed;
-        double decompress_time_elapsed;
+        double decompress_speed = 0.0;
+        double decompress_speed_low = 0.0;
+        double decompress_speed_high = 0.0;
+        double compress_time_elapsed = 0.0;
+        double decompress_time_elapsed = 0.0;
         cputime_chronometer chrono;
 
         while (total_time <= 10.0) {
@@ -285,7 +285,7 @@ int main(int argc, char *argv[]) {
                 printf("<=> ");
                 DENSITY_BENCHMARK_BLUE(printf("Decompress speed ");
                 DENSITY_BENCHMARK_BOLD(printf("%.0lf MB/s", decompress_speed)));
-                printf(" (min %.0lf MB/s, max %.0lf MB/s, best %.4lfs) ", decompress_speed_low, decompress_speed_high, decompress_time_low);
+                printf(" (min %.0lf MB/s, max %.0lf MB/s, best %.4lfs)    ", decompress_speed_low, decompress_speed_high, decompress_time_low);
             }
             fflush(stdout);
         }

benchmark/src/benchmark.h

@@ -54,13 +54,13 @@
 #define DENSITY_BENCHMARK_COMPILER_VERSION        __GNUC__, __GNUC_MINOR__, __GNUC_PATCHLEVEL__
 #elif defined(_MSC_VER)
 #define DENSITY_BENCHMARK_COMPILER                "MSVC"
-#define DENSITY_BENCHMARK_COMPILER_VERSION
+#define DENSITY_BENCHMARK_COMPILER_VERSION		  ""
 #elif defined(__INTEL_COMPILER)
 #define DENSITY_BENCHMARK_COMPILER                "ICC"
-#define DENSITY_BENCHMARK_COMPILER_VERSION
+#define DENSITY_BENCHMARK_COMPILER_VERSION		  ""
 #else
 #define DENSITY_BENCHMARK_COMPILER                "an unknown compiler"
-#define DENSITY_BENCHMARK_COMPILER_VERSION
+#define DENSITY_BENCHMARK_COMPILER_VERSION		  ""
 #endif
 
 #if defined(_WIN64) || defined(_WIN32)

build/premake5.lua

@@ -35,12 +35,19 @@
 --
 
 -- Check for tools
-if os.execute("clang -v") == true then
+if package.config:sub(1,1) == "\\" then
+	if os.execute("clang -v") == true then
+		toolset "msc-llvm-vs2014"
+	else
+		io.write("INFO : Selected compiler is MSVC. You might get better performance with Clang/LLVM.\n")
+	end
+elseif os.execute("clang -v") == true then
 	toolset "clang"
 elseif os.execute("gcc -v") == true then
 	toolset "gcc"
+	io.write("INFO : Selected compiler is GCC. You might get better performance with Clang/LLVM.\n")
 else
-	io.write("No supported compiler found on the command line. Please install Clang/LLVM or GCC.\n")
+	io.write("ERROR : No supported compiler found on the command line. Please install Clang/LLVM, GCC, or MSC.\n")
 	os.exit(0)
 end
 
@@ -53,11 +60,14 @@ end
 os.execute("git submodule update --init --recursive")
 
 solution "Density"
-	configurations { "release" }
+	configurations { "Release" }
 		flags { "NoFramePointer", "LinkTimeOptimization" }
 		optimize "Speed"
 		cdialect "C99"
 		warnings "Extra"
+		if os.is64bit() then
+			architecture "x64"
+		end
 
 	project "density"
 		kind "SharedLib"

src/algorithms/algorithms.c

@@ -34,7 +34,7 @@
 
 #include "algorithms.h"
 
-DENSITY_WINDOWS_EXPORT DENSITY_FORCE_INLINE void density_algorithms_prepare_state(density_algorithm_state *const restrict state, void *const restrict dictionary) {
+DENSITY_WINDOWS_EXPORT DENSITY_FORCE_INLINE void density_algorithms_prepare_state(density_algorithm_state *const DENSITY_RESTRICT state, void *const DENSITY_RESTRICT dictionary) {
     state->dictionary = dictionary;
     state->copy_penalty = 0;
     state->copy_penalty_start = 1;

src/algorithms/algorithms.h

@@ -66,13 +66,13 @@ typedef struct {
                 state->copy_penalty_start >>= 1;
 
 #define DENSITY_ALGORITHM_TEST_INCOMPRESSIBILITY(span, work_block_size)\
-            if (density_unlikely(span & ~(work_block_size - 1))) {\
+            if (DENSITY_UNLIKELY(span & ~(work_block_size - 1))) {\
                 if (state->previous_incompressible)\
                     state->copy_penalty = state->copy_penalty_start;\
                 state->previous_incompressible = true;\
             } else\
                 state->previous_incompressible = false;
 
-DENSITY_WINDOWS_EXPORT void density_algorithms_prepare_state(density_algorithm_state *const, void *const);
+DENSITY_WINDOWS_EXPORT void density_algorithms_prepare_state(density_algorithm_state *const DENSITY_RESTRICT_DECLARE, void *const DENSITY_RESTRICT_DECLARE);
 
 #endif

src/algorithms/chameleon/core/chameleon_decode.c

@@ -44,17 +44,17 @@
 
 #include "chameleon_decode.h"
 
-DENSITY_FORCE_INLINE void density_chameleon_decode_process_compressed(const uint16_t hash, uint8_t **restrict out, density_chameleon_dictionary *const restrict dictionary) {
+DENSITY_FORCE_INLINE void density_chameleon_decode_process_compressed(const uint16_t hash, uint8_t **DENSITY_RESTRICT out, density_chameleon_dictionary *const DENSITY_RESTRICT dictionary) {
     DENSITY_MEMCPY(*out, &dictionary->entries[hash].as_uint32_t, sizeof(uint32_t));
 }
 
-DENSITY_FORCE_INLINE void density_chameleon_decode_process_uncompressed(const uint32_t chunk, uint8_t **restrict out, density_chameleon_dictionary *const restrict dictionary) {
+DENSITY_FORCE_INLINE void density_chameleon_decode_process_uncompressed(const uint32_t chunk, uint8_t **DENSITY_RESTRICT out, density_chameleon_dictionary *const DENSITY_RESTRICT dictionary) {
     const uint16_t hash = DENSITY_CHAMELEON_HASH_ALGORITHM(chunk);
     (&dictionary->entries[hash])->as_uint32_t = chunk;
     DENSITY_MEMCPY(*out, &chunk, sizeof(uint32_t));
 }
 
-DENSITY_FORCE_INLINE void density_chameleon_decode_kernel(const uint8_t **restrict in, uint8_t **restrict out, const density_bool compressed, density_chameleon_dictionary *const restrict dictionary) {
+DENSITY_FORCE_INLINE void density_chameleon_decode_kernel(const uint8_t **DENSITY_RESTRICT in, uint8_t **DENSITY_RESTRICT out, const density_bool compressed, density_chameleon_dictionary *const DENSITY_RESTRICT dictionary) {
     if (compressed) {
         uint16_t hash;
         DENSITY_MEMCPY(&hash, *in, sizeof(uint16_t));
@@ -73,11 +73,11 @@ DENSITY_FORCE_INLINE bool density_chameleon_decode_test_compressed(const uint_fa
     return (density_bool const) ((signature >> shift) & DENSITY_CHAMELEON_SIGNATURE_FLAG_MAP);
 }
 
-DENSITY_FORCE_INLINE void density_chameleon_decode_4(const uint8_t **restrict in, uint8_t **restrict out, const uint_fast64_t signature, const uint_fast8_t shift, density_chameleon_dictionary *const restrict dictionary) {
+DENSITY_FORCE_INLINE void density_chameleon_decode_4(const uint8_t **DENSITY_RESTRICT in, uint8_t **DENSITY_RESTRICT out, const uint_fast64_t signature, const uint_fast8_t shift, density_chameleon_dictionary *const DENSITY_RESTRICT dictionary) {
     density_chameleon_decode_kernel(in, out, density_chameleon_decode_test_compressed(signature, shift), dictionary);
 }
 
-DENSITY_FORCE_INLINE void density_chameleon_decode_256(const uint8_t **restrict in, uint8_t **restrict out, const uint_fast64_t signature, density_chameleon_dictionary *const restrict dictionary) {
+DENSITY_FORCE_INLINE void density_chameleon_decode_256(const uint8_t **DENSITY_RESTRICT in, uint8_t **DENSITY_RESTRICT out, const uint_fast64_t signature, density_chameleon_dictionary *const DENSITY_RESTRICT dictionary) {
     uint_fast8_t count = 0;
 
 #ifdef __clang__
@@ -91,12 +91,12 @@ DENSITY_FORCE_INLINE void density_chameleon_decode_256(const uint8_t **restrict
 #endif
 }
 
-DENSITY_FORCE_INLINE void density_chameleon_decode_read_signature(const uint8_t **restrict in, density_chameleon_signature *restrict signature) {
+DENSITY_FORCE_INLINE void density_chameleon_decode_read_signature(const uint8_t **DENSITY_RESTRICT in, density_chameleon_signature *DENSITY_RESTRICT signature) {
     DENSITY_MEMCPY(signature, *in, sizeof(density_chameleon_signature));
     *in += sizeof(density_chameleon_signature);
 }
 
-DENSITY_WINDOWS_EXPORT DENSITY_FORCE_INLINE density_algorithm_exit_status density_chameleon_decode(density_algorithm_state *const restrict state, const uint8_t **restrict in, const uint_fast64_t in_size, uint8_t **restrict out, const uint_fast64_t out_size) {
+DENSITY_WINDOWS_EXPORT DENSITY_FORCE_INLINE density_algorithm_exit_status density_chameleon_decode(density_algorithm_state *const DENSITY_RESTRICT state, const uint8_t **DENSITY_RESTRICT in, const uint_fast64_t in_size, uint8_t **DENSITY_RESTRICT out, const uint_fast64_t out_size) {
     if (out_size < DENSITY_CHAMELEON_DECOMPRESSED_UNIT_SIZE)
         return DENSITY_ALGORITHMS_EXIT_STATUS_OUTPUT_STALL;
 
@@ -113,12 +113,12 @@ DENSITY_WINDOWS_EXPORT DENSITY_FORCE_INLINE density_algorithm_exit_status densit
     const uint8_t *in_limit = *in + in_size - DENSITY_CHAMELEON_MAXIMUM_COMPRESSED_UNIT_SIZE;
     uint8_t *out_limit = *out + out_size - DENSITY_CHAMELEON_DECOMPRESSED_UNIT_SIZE;
 
-    while (density_likely(*in <= in_limit && *out <= out_limit)) {
-        if (density_unlikely(!(state->counter & 0xf))) {
+    while (DENSITY_LIKELY(*in <= in_limit && *out <= out_limit)) {
+        if (DENSITY_UNLIKELY(!(state->counter & 0xf))) {
             DENSITY_ALGORITHM_REDUCE_COPY_PENALTY_START;
         }
         state->counter++;
-        if (density_unlikely(state->copy_penalty)) {
+        if (DENSITY_UNLIKELY(state->copy_penalty)) {
             DENSITY_ALGORITHM_COPY(DENSITY_CHAMELEON_WORK_BLOCK_SIZE);
             DENSITY_ALGORITHM_INCREASE_COPY_PENALTY_START;
         } else {
@@ -157,7 +157,7 @@ DENSITY_WINDOWS_EXPORT DENSITY_FORCE_INLINE density_algorithm_exit_status densit
             break;
     }
 
-    if (density_unlikely(shift == density_bitsizeof(density_chameleon_signature)))
+    if (DENSITY_UNLIKELY(shift == density_bitsizeof(density_chameleon_signature)))
         goto read_signature;
     else
         goto read_and_decode_4;