[[_TOC_]] # Rationale Compared to our current crazy-size-optimized XZ settings, in theory zstd should give us: * faster compression ⇒ improves dev & RM experience * vastly faster decompression ⇒ improves performance for users * only slightly larger output ⇒ makes install/upgrade downloads take a little longer Regarding tooling, zstd is supported by: * `squashfs-tools-ng` * Linux kernel * `squashfs-tools` 4.4+ # Benchmarks ## Image size These tests are with `defaultcomp` * xz (`feature/trixie` at b1bd804b9d3a5b1d17f98755b0559803a3e07d32, i.e. without pre-compiled AppArmor policy): * 1768435712 bytes * zstd (`feature/trixie` at b1bd804b9d3a5b1d17f98755b0559803a3e07d32, i.e. without pre-compiled AppArmor policy + 855e32bdd57bb09f38455579e77f4acd4a72d682): * 1932941312 bytes i.e. **+164MB** i.e. **+9%** ## Boot time ### Rationale for the process * As baseline we use 7.0\~rc1 + bringing back the cached AppArmor policy, because this is a more realistic baseline than 7.0\~rc1. * All tests are done with `defaultcomp` (release), which is what this issue is now about, since we've already switched the `fastcomp` builds to zstd. * We use the same way to measure boot time as what we do during manual QA for releases: we're familiar with it, it's fair, and it avoids bias due to first-boot repartitioning. ### Process #### Download 1. Download the _baseline_ USB image (built from !2361) from https://nightly.tails.net/build_Tails_ISO_21105-bring-back-precompiled-apparmor-policy/lastSuccessful/archive/build-artifacts/ 2. Download the _zstd22_ USB image (built from !2360) from https://nightly.tails.net/build_Tails_ISO_18655-zstd-squashfs-simple/lastSuccessful/archive/build-artifacts/ #### Test baseline Prepare the _baseline_ USB stick: 1. Install the _baseline_ USB image to a USB stick. 2. Boot the _baseline_ USB stick on a bare-metal computer a first time to trigger re-partitioning. 3. Wait until you see the Welcome Screen. 4. Shutdown Tails Then, on every spare computer that you can have access to: 1. Boot this USB stick a second time, add the `login` option to GRUB, and measure the boot time (from the GRUB menu until the GNOME desktop is ready). 2. Take note of the boot time you measured in the table below, in the _baseline_ column. #### Test zstd22 Prepare the _zstd22_ USB stick: 1. Install the _zstd22_ USB image to the exact same USB stick that you used to test the _baseline_ image. 2. Boot the _zstd22_ USB stick on a bare-metal computer a first time to trigger re-partitioning. 3. Wait until you see the Welcome Screen. 4. Shutdown Tails Then, on every spare computer that you can have access to: 1. Boot this USB stick a second time, add the `login` option to GRUB, and measure the boot time (from the GRUB menu until the GNOME desktop is ready). 2. Take note of the boot time you measured in the table below, in the _zstd22_ column. #### Repeat with a different USB stick Repeat the **Test baseline** and **Test zstd22** sections with another USB stick. And another one. And maybe that'll be enough. Please stay hydrated. ### Results <table> <tr> <td align="center"> **Computer** </td> <td align="center"> **USB Stick** </td> <td align="center"> **Baseline** </td> <td align="center"> **zstd22** </td> <td> **Seconds saved** </td> <td> **Percent saved** </td> </tr> <tr> <td>ThinkPad X1C6</td> <td>Kingston</td> <td align="right">72</td> <td align="right">57</td> <td align="right">15</td> <td align="right">21%</td> </tr> <tr> <td>ThinkPad X1C6</td> <td>Toshiba</td> <td align="right">78</td> <td align="right">63</td> <td align="right">15</td> <td align="right">19%</td> </tr> <tr> <td>ThinkPad X1C6</td> <td>Generic</td> <td align="right">92</td> <td align="right">81</td> <td align="right">11</td> <td align="right">12%</td> </tr> <tr> <td>ThinkPad X250</td> <td>Kingston</td> <td align="right">90</td> <td align="right">76</td> <td align="right">14</td> <td align="right">16%</td> </tr> <tr> <td>ThinkPad X250</td> <td>Toshiba</td> <td align="right">91</td> <td align="right">77</td> <td align="right">14</td> <td align="right">15%</td> </tr> <tr> <td>ThinkPad X250</td> <td>Generic</td> <td align="right">103</td> <td align="right">93</td> <td align="right">10</td> <td align="right">10%</td> </tr> <tr> <td>ThinkPad X201</td> <td>Kingston</td> <td align="right">102</td> <td align="right">75</td> <td align="right">27</td> <td align="right">26%</td> </tr> <tr> <td>ThinkPad X201</td> <td>Toshiba</td> <td align="right">103</td> <td align="right">76</td> <td align="right">27</td> <td align="right">26%</td> </tr> <tr> <td>ThinkPad X201</td> <td>Generic</td> <td align="right">111</td> <td align="right">87</td> <td align="right">24</td> <td align="right">22%</td> </tr> <tr> <td>ThinkPad X200</td> <td>Kingston DataTraveler 3.0 (PMAP)</td> <td align="right">110</td> <td align="right">82</td> <td align="right">28</td> <td align="right">25%</td> </tr> <tr> <td>ThinkPad X200</td> <td>ADATA USB Flash Drive (1.00)</td> <td align="right">111</td> <td align="right">82</td> <td align="right">29</td> <td align="right">26%</td> </tr> <tr> <td>ThinkPad X200</td> <td>TOSHIBA TransMemory (1.00)</td> <td align="right">113</td> <td align="right">83</td> <td align="right">30</td> <td align="right">27%</td> </tr> <tr> <td>HP EliteBook 840G1</td> <td>NVMe in USB enclosure</td> <td align="right">82</td> <td align="right">60</td> <td align="right">22</td> <td align="right">27%</td> </tr> <tr> <td>HP EliteBook 840G1</td> <td>Kingston DataTraveler 3.0 (PMAP)</td> <td align="right">78</td> <td align="right">59</td> <td align="right">19</td> <td align="right">24%</td> </tr> <tr> <td>HP EliteBook 840G1</td> <td>ADATA USB Flash Drive (1.00)</td> <td align="right">76</td> <td align="right">57</td> <td align="right">19</td> <td align="right">25%</td> </tr> <tr> <td>HP EliteBook 840G1</td> <td>TOSHIBA TransMemory (1.00)</td> <td align="right">78</td> <td align="right">60</td> <td align="right">18</td> <td align="right">23%</td> </tr> </table> ## Compression speed ### New tests Builds take a few minutes less on Jenkins (dragon, iguana). Not a game changer. ### Old tests **Note**: these are results from the 4.x era. | system | compression | time to compress SquashFS | build time saved | |--------|-------------|---------------------------|------------------| | sib | xz (release) | 537s | n/a | | sib | zstd (release) | 299s | 238s | | ant01 | xz (release) | 404s | n/a | | ant01 | zstd (release) | 220s | 184s | | lizard | xz (release) | 910s | n/a | | lizard | zstd (release) | 519s | 391s | | iguana | xz (release) | 393s | n/a | | iguana | zstd (release) | 263s | 130s | | intrigeri's laptop | xz (fast) | 265s | n/a | | intrigeri's laptop | zstd (fast) | 170s | 95s | | boyska's desktop | xz (fast) | 62s | n/a | | boyska's desktop | zstd (fast) | 33s | 29s | # Conclusions ## release (defaultcomp) We cannot switch to zstd as-is: the image grows too much. (**UPDATE**: actually, we can, see discussion on tails/tails#21100. So what's below is moot.) We have 2 options: ### With squashfs-tools Append, to our SquashFS sort file, the files that are _not_ added to it by `boot-profile`, grouped by filename extension. This can [save about 5%](https://github.com/AgentD/squashfs-tools-ng/issues/94#issuecomment-958762651) on the image size. I tried this and saw no impact at all on the image size: - https://gitlab.tails.boum.org/tails/tails/-/tree/wip/18655-zstd-squashfs - https://gitlab.tails.boum.org/tails/live-build/-/commits/tails/update-squashfs-sort-file ### With squashfs-tools-ng Branches: - https://gitlab.tails.boum.org/tails/tails/-/tree/wip/18655-squashfs-tools-ng - https://gitlab.tails.boum.org/tails/live-build/-/tree/tails/squashfs-tools-ng If the former is not sufficient, then we could switch to `squashfs-tools-ng`, if upstream can add advanced compression options (bcj filters, dictionary) to the zstd compressor, that would give us ISO/USB image not much greater than our current ones, with better performance on users' systems (and hopefully not slower to build on CI and RM's machines). For this to happen: - [x] Base this work on `feature/bullseye` or backport `squashfs-tools-ng` to Buster - [x] Build a version of our `live-build` fork that uses `gensquashfs` - [ ] Advanced zstd compression option: request this from upstream - [x] SquashFS file ordering - SquashFS file ordering is very useful even with USB sticks (https://gitlab.tails.boum.org/tails/tails/-/issues/15915#note_21450). - Support is being added upstream (https://github.com/AgentD/squashfs-tools-ng/issues/94, pushed to `master` on 2021-11-01). - Adjust our code and SquashFS sort file to `gensquashfs`'s sort file format - [x] Replace `mksquashfs-excludes` - [x] First, narrow down the problem a little bit (!649) - Update `mksquashfs-excludes`: some of these excludes are obsolete. - Delete as many of these files as possible via `config/chroot_local-hooks/` - [x] Remove as much as possible via `config/binary_rootfs/excludes`: this covered all the remaining excluded files ## dev (fastcomp) It seems switching has only benefits → !643.