Commit ca293946 authored by sajolida's avatar sajolida
Browse files

Stop testing Firefox Beta and Chrome Beta

With the verification on the page, we're adding support for many more
browser (Safari, Edge, Opera, etc.). We won't test them all anyway.

Firefox Beta and Chrome Beta were the most time-consuming tests and the
ones that prevented me from running the test suite comfortably from
parent 922d4beb
......@@ -93,28 +93,11 @@ Perform the following steps for each of:
sudo apt install firefox-esr
- The latest beta version of Firefox:
wget -cO firefox.tar.bz2 ""
tar jxvf firefox.tar.bz2
firefox/firefox --new-instance
- The version of Chromium available in Debian stable:
sudo apt install chromium
- The latest beta version of Chrome:
wget -cO google-chrome-beta_current_amd64.deb
sudo dpkg -i google-chrome-beta_current_amd64.deb
sudo apt-get -f install
......@@ -159,19 +142,9 @@ Checklist
- [ ] Good
- [ ] Truncated
- [ ] Rogue
- [ ] Firefox Beta
- [ ] IMG
- [ ] Good
- [ ] Truncated
- [ ] Rogue
- [ ] Chromium
- [ ] IMG
- [ ] Good
- [ ] Truncated
- [ ] Rogue
- [ ] Chrome Beta
- [ ] IMG
- [ ] Good
- [ ] Truncated
- [ ] Rogue
- [ ] Backward incompatibility
  • @sajolida, I hear and understand the laudable goal to streamline the QA process.

    The commit message lets me believe you did not have in mind the actual reason for the tests you've removed, so I'd like to ensure you are conscious of what we're losing here. I totally trust you to make the final decision with this extra info in mind :)

    IIRC the 2 reasons why we wanted to test on these beta versions was not to be exhaustive (I agree this would be mostly futile). Instead, it was meant to:

    • Allow us to learn in advance if changes in these browsers will break compatibility with our stuff once these beta versions become stable. And then in theory, we would have some months to adjust. If we drop these tests, fine, but let's be conscious this is what we lose.

    • Give us a good guestimate of how our stuff works on non-ESR Firefox "release" channel, i.e. the one most Firefox users use: testing on ESR + beta implicitly gave us some confidence that the Firefox release channel, that sits right between those 2, would work as well.


  • Sorry my commit message was badly written and I didn't put forward my strongest argument. Thanks for digging further!

    I think that testing the extension in beta versions of Firefox and Chrome made more sense because we were relying on a lot more technology that is browser specific, namely WebExtension, its manifest, permissions, etc.

    The only API used by the JavaScript on the page and that is not universally supported in readAsBinaryString that doesn't work in Internet Explorer (discontinued in 2016). For me, this is a sign that the current code only relies on very basic APIs that I don't expect to break on any browser any time soon. But you know that I'm far from being an expert on this topic.

    I'm not sure how much of this is true for Forge but:

    • I couldn't find any browser compatibility information on their GitHub.
    • I couldn't find any trace of fixed or broken browser compatibility in their changelog.

    So my uninformed guess is that it's a good signa and that is also relies on very basic APIs only.

    But I understand your concern and it's still valid: even the best supported API could break in future versions. I think that the risk is very low and not worth the amount of extra work that I used to do before this commit (not so often, though: 1-2 times a year).

    Maybe another trade-off could be for me to have Firefox Beta (and maybe Chrome Beta) installed on the MacBook and test only a successful verification after the new version is released. It would be much quicker than downloading both beta browsers in a Tails and testing all 3 result cases each time while still allow us to detect serious regression in advance.

    What do you think?

  • Testing Firefox and Chrome beta once a year or so, and not as a blocking step to update the code, looks plenty good enough to address my concern!

  • Done in 6902e672!

  • Well done!

Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment