Killing a `run` service doesn't cause svcinit to fail

[samschlegel]

This isn't as much of a clearcut issue as #58 but figured I'd throw up an issue here as well to start a discussion

When running an itest_service with bazel run, killing the service doesn't cause svcinit to fail. I would have expected either svcinit would fail and start to shutdown other services, or I'd expect it to try and restart the service.

The motivation here is that we're trying to use itest_service_groups as a replacement for an existing dev/test flow that uses docker-compose, and would likely want to just start that up as a background daemon and have it auto-restart things if they fail in dev mode. I believe that if things fail when running as a test (4a5ebef), which I think matches what I expect, but in dev I think I'd like them to restart ideally. Curious on your take here.

Example Output

SIGTERM

Running with bazel run --enable_runfiles @@//shutdown:service_graceful:

19:46:51.656917 Assigning port 40799 to @@//shutdown:service_graceful
19:46:51.667615 Starting @@//shutdown:service_graceful [40799]
19:46:51.668005 HTTP Healthchecking @@//shutdown:service_graceful (pid 3899681) : http://localhost:40799
19:46:51.668436 healthcheck for @@//shutdown:service_graceful failed: Get "http://localhost:40799": dial tcp 127.0.0.1:40799: connect: connection refused
19:46:51.717880 @@//shutdown:service_graceful> Server is running
19:46:51.869230 HTTP Healthchecking @@//shutdown:service_graceful (pid 3899681) : http://localhost:40799
19:46:51.873080 @@//shutdown:service_graceful healthy!

Target                               Critical Path Contribution
@@//shutdown:service_graceful        205.397078ms

After kill 3899681 from another terminal

19:47:06.346630 @@//shutdown:service_graceful> SIGTERM received. Waiting 1 seconds before shutting down...
19:47:07.348493 @@//shutdown:service_graceful> Server closed.
^C19:49:56.656421 Shutdown requested, exiting gracefully. Press Ctrl-C again to force exit
19:49:56.656471 Shutting down services.
19:49:56.656521 Stopping @@//shutdown:service_graceful
ERRNO no such process
19:49:56.656578 Cleaning up.

SIGKILL

This has a similar behavior, but oddly it takes around 5 seconds for the exit log line to get printed and it also triggered the critical path info to get printed again.

20:00:16.821620 @@//shutdown:service_graceful healthy!

Target                               Critical Path Contribution
@@//shutdown:service_graceful        205.764427ms

20:00:35.053362 @@//shutdown:service_graceful exited with error: signal: killed
Target                               Startup Time
@@//shutdown:service_graceful        205.764427ms


Target                               Critical Path Contribution
@@//shutdown:service_graceful        205.764427ms

^C20:11:32.215874 Shutdown requested, exiting gracefully. Press Ctrl-C again to force exit
20:11:32.215945 Shutting down services.
20:11:32.216020 Stopping @@//shutdown:service_graceful
20:11:32.216053 Sent SIGTERM to @@//shutdown:service_graceful
20:11:32.216062 Sent SIGTERM to @@//shutdown:service_graceful, waiting for 10s to stop gracefully
20:11:32.216091 Cleaning up.