As part of this ticket, once the bug is fixed, in the test suite, stop accepting:
- "Tails - Trying a testing version of Tails": that was introduced in b69a4dc6 (but keep the rest of the commit)
- "Tails - Welcome to Tails!": revert e5a61417
This scenario has failed 38 times, over the last 2 months, at the “I can watch a WebM video over HTTPs” step, so I’m going to mark it as fragile.
Interestingly, in the failure I’ve looked into, I see a NoScript
click-to-play placeholder, while we’re supposed to let WebM play
automatically these days.
As per #11592 (comment 58078) and following comments:
- This bug affects Tails 3.x as well as Buster-based branches ⇒ not a regression in Buster.
- This bug already occurred between Aug 7 and Aug 19, with Tor Browser 8.5.4 ⇒ not a regression in Tor Browser 8.5.5.
Nevertheless, as per #17053 (comment 2063), the first time sajolida saw this bug was a couple days ago, while running 4.0~beta2. Under the hypothesis that this bug is caused by a race condition, we can infer that:
- Either something changed recently, that makes us lose this race condition more often in general.
- Or something changed recently, that makes us lose this race condition more often with sajolida’s config and hardware.
On https://jenkins.tails.boum.org/view/RM/job/test_Tails_ISO_devel/ I’ve been annotating failed test suite runs with the reasons for failures. But I’ve been doing this only since August 31 so that’s not enough to bisect what “recently” means exactly here. One would need to analyze a bunch of test suite runs between Aug 9 (when we started running the full test suite on the devel branch) and Aug 31. I’ve sampled 6 jobs in this timespan and I’ve seen that JS was disabled in 2 of them:
For comparison, in the 20 runs since the beginning of September, I’ve attributed test failures to this bug 11 times. So yeah, it looks like the bug happens more often. Note, however, that Jenkins has been particularly busy in late Aug & early September, compared to the usual; workload variations can affect the occurrence rate of such race conditions.