Ensure check-mirrors.rb does not run forever
Originally created by @intrigeri on #17702 (Redmine)
Today on misc.lizard
there were 3 instances of that cronjob running: 1
current + 2 that started respectively on April 22 and 23. The 2 oldest
ones were all stuck waiting for a wget
command to complete.
This might explain some of the leftover temporary directories we’ve seen
recently (tails#17679 (closed)): one hypothesis is that these directories were not
leftovers of “old instances of check-mirrors.rb
was killed and did not
clean up after itself”, but rather a consequence of “weeks-old
check-mirrors.rb
instances were still running”.
Either way, I think we need 2 things:
-
sysadmin: a lock on the cronjob (like we have for most similar cronjobs already) and possibly a timeout (see e.g. timeout(1)
which seems to fit the bill), in the cronjob. In case the timeout expires or the lock can’t be taken by a new cronjob instance, I think the mirrors team should be made aware. -
check-mirrors.rb
itself: tails#17736
Related issues
- Related to tails#17679 (closed)
Edited by Zen Fu