Skip to content

Latest commit

 

History

History
131 lines (85 loc) · 4.5 KB

README.md

File metadata and controls

131 lines (85 loc) · 4.5 KB

Development Stack Overview

The docker-compose.yml file for this project is a docker-compose configuration to be used only for development purpose. There is almost zero security in the stack configuration.

It is composed of the Zimit frontend UI and API (of course), but also a local Zimfarm DB, API, worker and UI, so that you can test the whole integration locally.

Zimit frontend UI has two containers, one identical to production and one allowing hot reload of local developments.

Zimit frontend API has one container, slightly modified to allow hot reload of most modifications done to the source code.

Zimfarm UI, API worker, and DB are deployed with official production Docker images.

List of containers

zimit-ui-prod

This container is Zimit frontend UI as served in production (already compiled as a static website, so not possible to live-edit)

zimit-ui-dev

This container is Zimit frontend UI served in development mode (possible to live-edit)

zimit-api

This container is Zimit frontend API server (only slightly modified to enable live reload of edits)

zimfarm-db

This container is a local Zimfarm database

zimfarm-api

This container is a local Zimfarm API

zimfarm-ui

This container is a local Zimfarm UI

zimfarm-worker-manager

This container is a local Zimfarm worker manager. It pulls the Zimfarm task worker image to execute tasks

zimfarm-receiver

This container stores the uploaded files/logs for each task.

Instructions

Starting the Compose Stack

  • To start the compose services (without the worker):

    cd dev
docker compose up --build

  • To start the compose services (with a registered worker):

    cd dev
docker compose --profile worker up --build

  • If you are running with worker profile, you will need to create warehouse paths to upload the logs and files for each task.

    docker exec -it zimfarm-receiver bash
/contrib/create-warehouse-paths.sh

If it is your first execution of the dev stack, you need to create offliners and a "virtual" worker in Zimfarm DB. Thus, you need to start the services without the worker profile till you register a worker.

  • To create offliners:

    cd dev
./create_offliners.sh

    This pulls the latest offliner definitions from the online Zimfarm API and registers them with the dev Zimfarm API. Set ZIMIT_DEFINITION_VERSION defined in dev/docker-compose.yml to the specific version of the zimit scraper you want.

  • To register a worker

    cd dev
./create_worker.sh

NOTE: These shell scripts have been configured withs some reasonable defaults like:

  • admin username is admin with password admin. This must be the same as teh INIT_USERNAME AND INIT_PASSWORD of the zimfarm-api service and _ZIMFARM_USERNAME and _ZIMFARM_PASSWORD of the zimit-api service. See the Compose file for the definition of these environment variables.
  • a worker test_worker with 1Gb memory, 1Gb disk and 1 CPU. These are specified in the environment section of the zimfarm-worker-manager too.

If you have requested a task via Zimit UI and want to simulate a worker starting this task to observe the consequence in Zimit UI, you might use the dev/start_first_req_task.sh.

Restart the backend

Should the API process fail, you might restart it with:

docker restart zimit-zimit_api-1

Browse the web UIs

You might open following URLs in your favorite browser:

You can login into Zimfarm UI with username admin and password admin.

Contributing

The api folder is a Python project that contains the code for the zimit-api described in the Overview while the ui folder is a Vue 3 project. If you want to contribute code to this project, you would need to install Hatch for managing dependencies of the Python project and Yarn for the Vue 3 project.

This project is configured to use git pre-commit hooks. If you have set up the Python project, pre-commit would be installed in the virtual environment. To install the hooks, use:

pre-commit install

Additionally, you would need to install Prettier globally on your machine. This is because the pre-commit hook uses your machine's Prettier to lint files in the ui folder.