Some test jobs are getting stucked in Jenkins
Originally created by @bertagaz on #11546 (Redmine)
We’ve noticed that some test jobs were taking too much time to run in Jenkins. We had to abort them in order to have more free isotesters. Symptom is that no scenarios go further than waiting for the Tails Greeter to show up, but the ISO never boot up to it, so every scenario reaches the 30 minute timeout. With every scenario taking 30 minutes to get aborted, the entire test job is taking a lot of time to run.
Booting by hand one of such ISO, it appears that the live-config boot script is stuck without any limit, so it’s blocking the boot process, and the Tails greeter never shows up.
We’ve seen that on this particular jobs:
https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-10779-start-haveged-earlier/
https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-10298-linux-4.x-aufs-kaslr/
(started to fail on job
16)
https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-10298-linux-4.x-aufs/
(started to fail on job
22)
https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-stretch/
https://jenkins.tails.boum.org/job/test_Tails_ISO_feature-8230-greeter-revamp-phase1/
The later is expected, as the test suite can’t handle the new greeter
yet. The feature/stretch
is WIP and always had this problem too.
Regarding the linux-4.x
feature branches, this problem seems to have
appeared on both of them due to a new kernel version.
Comparing the packages list for each branch of a working ISO and a
faulty one, the diff shows
only:
$ diff -Naur tails-i386-feature_10298-linux-4.x-aufs-2.6-20160619T2233Z-bbb7c37+devel@435cf17.iso.packages tails-i386-feature_10298-linux-4.x-aufs-2.6-20160620T2233Z-bbb7c37+devel@435cf17.iso.packages
--- tails-i386-feature_10298-linux-4.x-aufs-2.6-20160619T2233Z-bbb7c37+devel@435cf17.iso.packages 2016-06-20 01:30:23.000000000 +0200
+++ tails-i386-feature_10298-linux-4.x-aufs-2.6-20160620T2233Z-bbb7c37+devel@435cf17.iso.packages 2016-06-21 01:29:04.000000000 +0200
@@ -1477,8 +1477,8 @@
liferea-data 1.10.12-1
linux-base 4.3
linux-image-4.5.0-0.bpo.2-686 4.5.4-1~bpo8+1
-linux-image-4.5.0-0.bpo.2-amd64 4.5.4-1~bpo8+1
-linux-image-amd64 4.5+73~bpo8+1
+linux-image-4.6.0-0.bpo.1-amd64 4.6.1-1~bpo8+1
+linux-image-amd64 4.6+74~bpo8+1
live-boot 4.0.2-1
live-boot-initramfs-tools 4.0.2-1
live-config 4.0.4-1
That’s the only visible difference, other than that their ISO are built from the same commit. Building at home an ISO with the 4.5.4-1 kernel and another same but with the only the newer 4.6.1-1 kernel resulted in the same problem.
It seems that the test_Tails_ISO_feature-10779-start-haveged-earlier
job always exhibited this problem (and doesn’t use a 4.x kernel), for
this one the root cause is probably different. This would need some more
debugging though.
Related issues
- Is duplicate of #11449 (closed)