Stops watching when pods restart

[bryanlarsen]

If a pod crashes and gets restarted kubetail stops tailing the pod, and any logs from the newly restarted pod are not seen. And since monitoring these types of crashes and the clusters response to them is my main purpose for trying to use kubetail...

[SebastianKristof]

+1 to this issue. It is not only when a pod is killed that it becomes a problem. A very common use case for me is to watch the pods as they dynamically scale up and down. Kubetail only grabs a "snapshot" as it were of the pods that are running at the time the command is ran. This makes it pretty useless for monitoring the logs from pods as they come up and down - which is a shame!

'Stern' looks like it supports this feature, although I haven't tried it:

In addition, if a pod is killed and recreated during a deployment Stern will stop listening to the old pod and automatically hook into the new one. There’s no more need to figure out what the id of that newly created pod is.

Let's add this functionality to Kubetail, it's totally worth it.

[johanhaleby]

Yes this is something that I've been trying to achieve several times in the past, but I've failed so far. Have a look at the comment here. See this issue as well.

[lknite]

Yes please, my use case is a development environment with pods getting restarted all the time. I've written a script to pick back up, but have to manually restart kubetail. Copying here for folks to use as a workaround for a single pod anyway:

$ cat bin/logs
#!/bin/bash

# Watch the specified application log, restart if pod disappears
while(true) do kubectl logs -f -l app=$1; sleep 1; done

[johanhaleby]

@lknite Maybe this should be integrated into kubetail if it works well. Looks simple enough :)

[lknite]

Above, generate temp file used to restart tail if a watched pod restarts

TMPFILE_RESTART=`mktemp`
rm $TMPFILE_RESTART

To raise the event that a pod log has exited:

logs_commands+=("${kubectl_cmd} ${timestamps} | ${colorify_lines_cmd} && touch $TMPFILE_RESTART");

Start the end tail command in the background with '&' (but still write to this shell's stdout):

while(true); do

  # Prep for tail
  # \<something here\>
  
  # Start tail command in a separate process and track PID so we can restart it if a pod exits
  tail ${tail_follow_command} -n +1 <( eval "${command_to_tail}" ) $line_buffered > /proc/$$/fd/1 &
  PID_TAIL=$!

  # Watch for pod exit by checking TMPFILE_RESTART
  while (true); do if [ -f $TMPFILE_RESTART ]; then break; fi; sleep 1; done
  
  # Tail has been interrupted
  rm $TMPFILE_RESTART; disown $PID_TAIL; kill $PID_TAIL; 
done

Now that we can catch and restart, just need to implement an algorithm to restart things...

[johanhaleby]

@lknite Could you turn this into a PR? It would be the PR of year!!! (and possibly the entire project :))

[lknite]

Pull request submitted: #147

The PR adds an idle processing loop where decisions can be made run time. Someone will still need to write an algorithm to respond to a pod being restarted, but seems like a good step in the right direction. For now if it detects that a pod was deleted it exits.