Ensure check-mirrors.rb does not run forever
misc.lizard there were 3 instances of that cronjob running: 1
current + 2 that started respectively on April 22 and 23. The 2 oldest
ones were all stuck waiting for a
wget command to complete.
This might explain some of the leftover temporary directories we’ve seen
recently (tails#17679 (closed)): one hypothesis is that these directories were not
leftovers of “old instances of
check-mirrors.rb was killed and did not
clean up after itself”, but rather a consequence of “weeks-old
check-mirrors.rb instances were still running”.
Either way, I think we need 2 things:
sysadmin: a lock on the cronjob (like we have for most similar
cronjobs already) and possibly a timeout (see e.g.
timeout(1)which seems to fit the bill), in the cronjob. In case the timeout expires or the lock can’t be taken by a new cronjob instance, I think the mirrors team should be made aware.
- Related to tails#17679 (closed)