Prevent race condition between shutdown and worker or extension launch by Smjert...

Smjert commented on Jul 19, 2021

Due to a race condition between the shutdown procedure part
handled by the WatcherRunner::stop and the execution of a new worker
or extension, it's possible that a worker or extension is left running
after the watcher process has exited.

This happens because when requested to shutdown
the WatcherRunner::stop function might not find any extensions or workers
to stop, but they are about to be launched.
This means that the stop function will return not having stopped any
process and the watcher will still start a worker or an extension,
and then finally realize that it has to interrupt and exit,
leaving the child processes running.

This is especially problematic with workers because if starting and
stopping osquery quickly, due to the issue above sometimes at the next
osquery service start, a worker process will still be running briefly,
and this causes the new osquery instance to fail to run,
causing a failure in the service start.

NOTE: This is a best effort attempt to fix this there might be (and there are) other issues related to this.

Namely something that's partially related is: #5233
And I've also noticed that we have two different handlers of the extensions lifetime/lifecycle, namely the WatcherRunner and the ExtensionManagerWatcher, which both try to shutdown an extension in different ways and the ExtensionManagerWatcher is always failing (I'll open an issue about this).