Stop testing Firefox Beta and Chrome Beta
With the verification on the page, we're adding support for many more browser (Safari, Edge, Opera, etc.). We won't test them all anyway. Firefox Beta and Chrome Beta were the most time-consuming tests and the ones that prevented me from running the test suite comfortably from Tails.
@sajolida, I hear and understand the laudable goal to streamline the QA process.
The commit message lets me believe you did not have in mind the actual reason for the tests you've removed, so I'd like to ensure you are conscious of what we're losing here. I totally trust you to make the final decision with this extra info in mind :)
IIRC the 2 reasons why we wanted to test on these beta versions was not to be exhaustive (I agree this would be mostly futile). Instead, it was meant to:
Allow us to learn in advance if changes in these browsers will break compatibility with our stuff once these beta versions become stable. And then in theory, we would have some months to adjust. If we drop these tests, fine, but let's be conscious this is what we lose.
Give us a good guestimate of how our stuff works on non-ESR Firefox "release" channel, i.e. the one most Firefox users use: testing on ESR + beta implicitly gave us some confidence that the Firefox release channel, that sits right between those 2, would work as well.
Sorry my commit message was badly written and I didn't put forward my strongest argument. Thanks for digging further!
I think that testing the extension in beta versions of Firefox and Chrome made more sense because we were relying on a lot more technology that is browser specific, namely WebExtension, its manifest, permissions, etc.
readAsBinaryStringthat doesn't work in Internet Explorer (discontinued in 2016). For me, this is a sign that the current code only relies on very basic APIs that I don't expect to break on any browser any time soon. But you know that I'm far from being an expert on this topic.
I'm not sure how much of this is true for Forge but:
- I couldn't find any browser compatibility information on their GitHub.
- I couldn't find any trace of fixed or broken browser compatibility in their changelog.
So my uninformed guess is that it's a good signa and that is also relies on very basic APIs only.
But I understand your concern and it's still valid: even the best supported API could break in future versions. I think that the risk is very low and not worth the amount of extra work that I used to do before this commit (not so often, though: 1-2 times a year).
Maybe another trade-off could be for me to have Firefox Beta (and maybe Chrome Beta) installed on the MacBook and test only a successful verification after the new version is released. It would be much quicker than downloading both beta browsers in a Tails and testing all 3 result cases each time while still allow us to detect serious regression in advance.
What do you think?
Testing Firefox and Chrome beta once a year or so, and not as a blocking step to update the code, looks plenty good enough to address my concern!