Basic dibspacks

This repository contains some basic dibspacks that can be used with dibs. To use them, you can…​

• …​ put a git reference to the main repository, e.g.:

packs:
  foo:
    type: git
    origin: https://github.com/polettix/dibspack-basic

• …​ use it from a stroke:

actions:
    some-stroke:
        pack: foo
        path: perl/build
        user: ubuntu

But there’s more: some parts in this repository are actually aimed at being included inside the target containers, e.g. to smooth some sharp edges.

Table of Contents

• git/fetch
• install/plain-copy
• install/with-dibsignore
• perl/build
• prereqs
• procfile/add
• shellrun
• wrapexec/install
• wrapexec/profilexec
• wrapexec/suexec

git/fetch

This dibspack helps checking out a git repository. It is called like this:

git/fetch [git_uri]

where the optional parameter is a URI where the code can be found. In case this parameters is absent, environment variable DIBSPACK_GIT_URI is used instead (or an exception is thrown if this is not defined properly too).

The git URI can include a fragment section, interpreted as whatever comes after the first dash character #. When present, the fragment represents the ref to check out; by default, the master branch is considered. This can help pinning a specific revision and avoid surprises due to updates.

The checkout is performed in the src_dir directory, i.e. the directory passed via the DIBS_DIR_SRC envile. If this directory already contains a .git sub-directory and environment variable DIBSPACK_GIT_REFRESH is set to 1 (exactly), then the current contents will be used for a fetch from the current origin remote. Otherwise, the src_dir is wiped out completely and the remote is cloned to get a fresh copy.

It is possible to set variable DIBSPACK_GIT_DEBUG to 1 (exactly) and activate the shell’s capabilities for debugging (options -x and '-v').

Example usage in a dibs.yml file:

packs:
  foo:
    type:   git
    origin: https://github.com/polettix/dibspack-basic
actions:
  bar:
    pack: foo
    path: git/fetch
    args: ['https://github.com/polettix/sample-mojo#844f256']

This dibspack requires git to be available in the container; it contains some scripts to assist in the installation of git depending on the operating system tools inside the container (e.g. Alpine, Debian, …​).

install/plain-copy

This buildpack copies files from a source to a destination. It is invoked like this:

install/plain-copy source destination

If source is missing or empty, src_dir is used instead (according to the contents of envile DIBS_DIR_SRC). If destination is empty, path /app is used instead.

This dibspack tries hard to get rid of destination before starting the copy.

The copy is done using cp -pPR to keep the same permissions as in the source.

Example usage in a dibs.yml file:

packs:
  foo:
    type:   git
    origin: https://github.com/polettix/dibspack-basic
actions:
  bar:
    pack: foo
    path: install/plain-copy
    args: [{path_cache: '/app'}, '/app']

install/with-dibsignore

This buildpack copies files from a source to a destination, pruning those that match patterns defined inside a file. This allows selecting only part of the files in a wider source, leaving behind those that possibly help building/testing artifacts but are then unneeded in the normal runtime phase.

It is invoked like this:

install/with-dibsignore [options...] [step]

This program is a Perl program and supports flexible GNU-style parameters:

• --debug/-D: boolean option, turn on debug mode (defaults to value of environment variable DIBSPACK_INSTALL_DEBUG and to off as a fallback).

• --dibsignore/-f: string option, sets the name of the file where to take patterns for exclusions. It defaults to environment variable DIBSPACK_INSTALL_DIBSIGNORE and, as a fallback. to the string .dibsignore. There can be one such file inside each directory where you want to do pruning.

• --dst/-d: string setting the destination of the copy. It defaults to the environment variable DIBSPACK_INSTALL_DST and, depending on the step parameter, to something else as a fallback (see below).

• --preserve/-p: boolean option, when set it does not wipe the previous content of the destination directory. It defaults to the environment variable DIBSPACK_INSTALL_PRESERVE or to a false value as fallback.

• --print/-P: boolean option, when set it does not copy anything but prints on standard output what will be copied. Defaults to environment variable DIBSPACK_INSTALL_PRINT or to a false value as fallback.

• --src/-s: string setting the source of the copy. It defaults to the environment variable DIBSPACK_INSTALL_SRC and, depending to the step parameter, to something else as a fallback (see below).

The optional parameter step is a shorthand to set fallback defaults for the source and destination directories, namely:

• if it takes value build, then the source eventually defaults to src_dir (i.e. the content of envile DIBS_DIR_SRC) and the destination to sub-directory app inside the cache directory (i.e. the second parameter).

• if it takes value bundle, then the source eventually defaults to sub-directory app inside the cache directory (i.e. the content of envile DIBS_DIR_CACHE) and the destination defaults to /app.

• any other value throws an exception.

The step can also be set via environment variable DIBSPACK_STEP.

The dibsignore file has the same format and follows the same rules as the more popular .gitignore file used by Git.

This program requires to run Perl inside the container. This should be a no-problem in the build steps, but might be trickier in the bundle steps. If this is actually the case, the suggestion is to prepare the copy with install/with-dibsignore during the build step, then use install/simple-copy (which only relies on POSIX compliant /bin/sh) to place the artifacts in the right place during the bundle step.

perl/build

This dibspack aims at compiling Perl code. As a matter of fact, it only makes sure that prerequisites modules are properly installed, e.g. via cpanm or carton. It is invoked like this:

perl/build [work_dir]

When set, work_dir indicates that installations should be done "from within" the specific directory. To do this, work_dir is created as a symlink to src_dir (i.e. the content of envile DIBS_DIR_SRC) and then the rest of operations performed from there. If not set, it defaults to the environment variable DIBSPACK_PERL_APP or, as a fallback, the string /app.

The dibspack saves some configurations inside the target directory (work_dir/src_dir) in file .profile/10.perl-env.sh (directory .profile is the profile_dir). This is mainly aimed at setting the right paths for executing the shipped Perl programs.

Other environment variables can influence the dibspack execution:

• DIBSPACK_VERBOSE, when set to 1 (exactly) turns on verbose mode.

• DIBSPACK_SAVE_ENV can be set to a path where the environment is saved (both env and set). If the variable is defined but it does not start with a slash, then the environment is saved inside directory /.build_env.

• DIBSPACK_SET_VERSION, when set to a non-empty string, triggers its saving inside the profile_dir inside file 20.version-env.sh.

• CPANM_OPTS options passed to cpanm.

• DIBSPACK_CPANM_VERBOSE sets verbose mode when running cpanm. Defaults to --quiet.

• DIBSPACK_CPANM_TEST sets or disable testing of modules. Defaults to --notest.

The outcome of compilation is saved in the cache (i.e. the directory saved in envile DIBS_DIR_CACHE), inside sub-directory perl/local.

prereqs

This dibspack supports in the installation of OS-specific packages/prerequisites. It is invoked like this:

prereqs [--os OS] [-w|--workdir|--work-dir DIR] [step]

The source project is supposed to have a prereqs sub-directory, and have executable files like this inside:

prereqs/
    alpine.build
    alpine.bundle
    debian.build
    debian.bundle
    debian.some-other-step...

It works like this:

• loads all enviles as environment variables;

• it establishes the platform’s os based on command-line option --os, environment variable DIBS_OS, or looking at the ID inside /etc/os-release;

• it establishes the work dir base on command-line option --workdir (or its aliases), on environment variable DIBS_WORK_DIR or, as a fallback, on the contents of envile DIBS_DIR_SRC;

• it establishes a step name from the command line or from environment variable DIBS_PREREQS (leaving it blank by default);

• it runs file $work_dir/prereqs/$os.$step if the step is defined, otherwise it run $work_dir/prereqs/$os.

procfile/add

This adds a simple handler for Procfile-like configurations support. This means that it’s possible to put a Procfile file inside the application directory, and it will hopefully honored (it also requires to set the associated program as the entry point of the generated container image).

This dibspack is controlled by environment variables (or enviles, all of them are loaded) with sensible defaults:

• DIBSPACK_PROCFILE_DEFAULT: sets the default process to run, defaults to web.

• DIBSPACK_PROCFILE_RUNNER: sets the name of the runner inside the container, defaults to /procfilerun.

• DIBSPACK_PROCFILE_SPEC: sets the position of the Procfile file, defaults to /app/Procfile.

As anticipated, to use this dibspack effectively it is necessary to ensure that the dibs.yml configuration file sets the right entry point and command while saving the image, like this:

packs:
  foo:
    type:   git
    origin: https://github.com/polettix/dibspack-basic
actions:
  bundle:
    - from: 'some-image:latest'
    - name: add procfile
      pack: foo
      path: procfile/add
      commit:
        entrypoint: ['/procfilerun']
        cmd: []
    - tags: 'new-image:1.0'
    # ...

shellrun

This dibspack is a swiss-army knife that allows running multiple shell commands. It is invoked like this:

shellrun [command1 [command2 [...]]]

Each argument is a shell command that is run "plainly". For example, if the argument is echo ciao a tutti, then the following is executed:

echo ciao a tutti

and so on.

Example usage:

packs:
  foo:
    type:   git
    origin: https://github.com/polettix/dibspack-basic
actions:
  foobar:
    path:   shellrun
    args:
      - "printf '%s\n' 'whatever you want'"
      - 'ls -l /'
      # ...

For simplicity, all standard output is redirected to standard error, so that execution of command appears in the run log of dibs.

wrapexec/install

Within wrapexec, this is actually the only dibspack. Its goal is to put other programs in wrapexec inside the root directory of the target container image, so that it can be later used.

Just pass the list of the other programs to install in the args.

Example usage:

packs:
  foo:
    type:   git
    origin: https://github.com/polettix/dibspack-basic
actions:
  foobar:
    path: wrapexec/install
    args:
      - procexec
      - suexec

wrapexec/profilexec

This is a wrapper intended for inclusion inside the target container image. It’s easier to read what it does directly:

if [ -r "$HOME/.profile" ] ; then
   . "$HOME/.profile"
elif [ -d "$HOME/.profile.d" ] ; then
   for file in "$HOME"/.profile.d/*.sh ; do
      . "$file"
   done
fi
exec "$@"

It ensures that either $HOME/.profile or whatever is in $HOME/.profile.d is sourced before running the real command that is passed on the command line. This allows saving pre-condition stuff (like environment variables) in a file or a bunch of files, then set the program as the entrypoint.

Example:

packs:
  foo:
    type:   git
    origin: https://github.com/polettix/dibspack-basic
actions:
  install-procexec:
    path: wrapexec/install
    args:
      - procexec
  some-sketch:
    # ...
    - name: some final stroke
      # ...
      commit:
        entrypoint: ['/procexec']
        cmd: ['/bin/sh', '-l']

Now, whatever you pass as the command when running this container, will be actually executed through procexec.

wrapexec/suexec

When building container images to distribute software for the command line, many times there’s a mismatch between the user that the software runs as within the container and the user that runs the container from the host.

wrapexec/suexec is meant to address the mismatch. It’s intended to be included inside the container image (e.g. through wrapexec/install) and then invoked by default setting it as the entrypoint, like this:

packs:
  foo:
    type:   git
    origin: https://github.com/polettix/dibspack-basic
actions:
  install-procexec:
    path: wrapexec/install
    args:
      - procexec
  some-sketch:
    # ...
    - name: some final stroke
      # ...
      commit:
        entrypoint: ['/suexec', '--reference', '/mnt', '--']
        cmd: ['/bin/sh', '-l']

In this example, suexec sets the user and group to match the owner of /mnt, so that when the container is run like this:

$ docker run -v "$PWD:/mnt" ...

the user that the command will be run as is the same as the owner of $PWD in the host.

The generic invocation of wrapexec/suexec is as follows:

/suexec [options] -- command [arguments]

Options are:

--also|-a group

add group to the list of additional groups for the target user

--also-for|-A path-in-container

add the group associated to file at path-in-container to the list of additional groups for the target user

--gid|-G group-id

set the main group identifier for the target user (defaults to environment variable GROUP_ID)

--group|-g group-name

set the main group name for the target user (defaults to environment variable GROUP_NAME)

--home|-h

set the home directory path (defaults to environment variable HOME_DIR);

--os os-name

set the name of the Linux distribution. This is used to figure out which tools are available for manipulating /etc/passwd and /etc/group (defaults to environment variable OS or is auto-detected)

--reference|-r path-in-container

set the user identifier and group identifier from the owner and group of the file at path-in-container. When you bind-mount a directory in the container, this is probably the most straightforward way to set the proper uid/gid pair for the user that should run the command inside the container.

--source|-s path-in-container

source (via .) the file at path-in-container, which might e.g. contain a few environment variables to set different defaults

--uid|-U user-id

set the user identifier for the target user (defaults to environment variable USER_ID)

--user|-u user-name

set the user name for the target user (defaults to environment variable USER_NAME)

After analyzing command-line options, the program tries its best to figure out the missing parts, e.g. a username or a main group name.

After setting up the container with the options above, wrapexec/suexec runs su-exec with the provided command and arguments. This allows executing the command with the right user, while at the same time doing a proper exec instead of putting any process in the middle (thus also guaranteeing that the exit code of command is correctly propagated as the return value from the container).

You have to "independently" ensure that su-exec is present in the container image. As an example, {Alpine Linux}[alpine] includes the su-exec package for it.