Investigate running REF recipes with sbatch

[alistairsellar]

REF solve runs esmvaltool recipes, and has an HPC executor that uses sbatch (which is ideal for CMEW). Until now we have only tried the default "local" executor.

Can we make this work on our system? Does the executor run the recipes in parallel? If so, how does it monitor completion?

Time box: 0.5 day

[zmaalick]

We need to add following in configuration file.

[executor.config]
scheduler = "slurm"
account = "your_account"
username = "your_username"
partition = "compute"           # or "debug" for testing
qos = "debug"                   # quality of service
req_nodes = 1                   # number of nodes
walltime = "00:30:00"           # time limit per job
cores_per_worker = 1            # cores per worker
max_workers_per_node = 64       # workers per node
scheduler_options = ""          # extra #SBATCH directives

[zmaalick]

Still getting error but some progress:

[executor.config]
scheduler = "slurm"
account = "normal"
username = "cdi_user_name"
partition = "cpu"           # or "debug" for testing
#qos = "debug"                   # quality of service
req_nodes = 1                   # number of nodes
walltime = "00:30:00"           # time limit per job
cores_per_worker = 1            # cores per worker
max_workers_per_node = 64       # workers per node
scheduler_options = ""          # extra #SBATCH directives

But still getting error in picking up the slurm.
following workaround didn't work:

srun -p cpu -A normal --time=01:00:00 --nodes=1 --ntasks=1 --cpus-per-task=1 \
  --cpu-bind=none \
  bash -lc 'export PARSL_HTEX_ADVERTISED_ADDRESS=$(hostname -I | awk "{print \$1}");
            export PARSL_HTEX_ADDRESS=$PARSL_HTEX_ADVERTISED_ADDRESS;
            export PARSL_WORKER_PORT_RANGE="55000,56000";
            ref solve --provider esmvaltool'

[zmaalick]

Error:

parsl.jobs.errors.TooManyJobFailuresError: Error 1:
	Job is marked as MISSING since the workers failed to register to the executor. Check the stdout/stderr logs in the submit_scripts directory for more debug information
	STDOUT: Found cores : 120
Found nodes : 1

	STDERR: 0: Traceback (most recent call last):
0:   File "/home/users/zubair.maalick/.conda/envs/climate-ref/bin/process_worker_pool.py", line 1000, in <module>
0:     manager = Manager(port=args.port,
0:               ^^^^^^^^^^^^^^^^^^^^^^^
0:   File "/home/users/zubair.maalick/.conda/envs/climate-ref/bin/process_worker_pool.py", line 163, in __init__
0:     self._ix_url = probe_addresses(
0:                    ^^^^^^^^^^^^^^^^
0:   File "/home/users/zubair.maalick/.conda/envs/climate-ref/lib/python3.11/site-packages/parsl/executors/high_throughput/probe.py", line 67, in probe_addresses
0:     raise ConnectionError(f"No viable ZMQ url from: {addys}")
0: ConnectionError: No viable ZMQ url from: tcp://127.0.0.1:54201, tcp://10.138.80.53:54201, tcp://151.170.240.197:54201
srun: error: sla-cpu-r-03: task 0: Exited with exit code 1

So, it is port connection error

added following in config (but useless):

address = "eth0"
worker_init = "export PARSL_HTEX_ADVERTISED_ADDRESS=10.138.80.53; export PARSL_HTEX_ADDRESS=10.138.80.53;"

Also installed:

conda install -c conda-forge -y zeromq pyzmq