First boot partitioning sometimes yields broken backup GPT, non-resized filesystem, and tpsd fails to create Persistent Storage

I've seen a bunch of reports in the last 6 months that looked like this was happening, but finally we have all the logging in place to confirm, and we've received a complete example with enough info for me to create an issue, so here we go.

Failure modes

Backup GPT corrupted after sgdisk --new

(wb:8116057893b0684dcc1f7b5c3eef55eb and wb:7478368e3d7b6ef53c3bb1251ab5f1bb)

I understand this report was sent the first time Tails was started, after freshly installing.

• During initial repartitioning:
  • sgdisk reports success so presumably the partition was resized. Indeed, tails-block-device-info sees the system partition is 8GB.
  • partprobe complains: "Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used."
  • fatresize complains: "Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.", and I see that df sees a 1.3GB filesystem so it's not been resized
  • the partitioning hook aborts after fatresize (presumably due to set -e) so all the stuff we would otherwise do after fatresize is skipped
• The kernel is unhappy too (after repartitioning, I believe): "kernel: Alternate GPT is invalid, using primary GPT."

Finally, tpsd fails to create a Persistent Storage partition:

Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: ERROR:window.py:218: failed to create Persistent Storage: GDBus.Error:gi.repository.GLib.Error: udisks-error-quark: GDBus.Error:org.freedesktop.UDisks2.Error.Failed: Error creating partition on
/dev/sdb: Process reported exit code 2: Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: on the recovery & transformation menu to examine the two tables.
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: Warning! One or more CRCs don't match. You should repair the disk!
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: Main header: OK
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: Backup header: OK
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: Main partition table: OK
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: Backup partition table: ERROR
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]: Invalid partition data!
Jan 21 15:07:35 amnesia tails-persistent-storage[8681]:  (0)
Jan 21 15:07:35 amnesia python3[8681]: Failed to set text '<span insert-hyphens='no'>gi.repository.GLib.Error: udisks-error-quark: GDBus.Error:org.freedesktop.UDisks2.Error.Failed: Error creating partition on /dev/sdb: Process reported exit code 2:
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
				       on the recovery & transformation menu to examine the two tables.

				       Warning! One or more CRCs don't match. You should repair the disk!
				       Main header: OK
				       Backup header: OK
				       Main partition table: OK
				       Backup partition table: ERROR

				       Invalid partition data!
					(0)

This error message comes from gdisk (the package that includes e.g. sgdisk): https://sources.debian.org/src/gdisk/1.0.9-2.1/gpt.cc/?hl=1157#L1157.

To find relevant bug reports in a archive of WhisperBack reports, search for Bug-specific details: Resizing system partition failed.

Backup GPT corrupted before sgdisk --new

E.g. wb:e0ea3297ed5d11833bbe36a440d9ec8b

+ sgdisk --move-second-header --randomize-guids '--delete=1' '--new=1:0:+8192M'
'--typecode=1:C12A7328-F81F-11D2-BA4B-00A0C93EC93B' '--change-name=1:Tails' /dev/sdb
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: ERROR
Main partition table: OK
Backup partition table: ERROR

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
The operation has completed successfully.
+ settle
+ sync
+ udevadm settle
+ sync
+ log_end_msg
+ _log_msg 'done.\n'
+ '[' y '=' y ]
+ return
+ partprobe /dev/sdb
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
+ settle
+ sync
+ udevadm settle
+ sync
+ dd 'if=/dev/sdb1' 'of=/tmp/vbr' 'bs=512' 'count=1'
1+0 records in
1+0 records out
+ settle
+ sync
+ udevadm settle
+ sync
+ FS_SIZE=8190Mi
+ fatresize '--size=8190Mi' /dev/sdb1
fatresize 1.1.0 (20221231)
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.

And the filesystem was not resized: /dev/sdb1 vfat 1.4G 1.4G 13M 100% /usr/lib/live/mount/medium.

Timed out for waiting the udev queue being empty

See you on #20451 (closed).

wrong usb stick identified

#20656

Benefit and cost

Benefit: this breaks automatic upgrades and the ability to create a Persistent Storage. It's been spotted in the wild a bunch of times so it's definitely affecting a number of users.

Cost: unknown.

Worth spending up to 3 hours researching and PoC'ing solutions, but probably not diving into a deeper rabbit hole.