In the Linux kernel, the following vulnerability has been resolved:
btrfs: reject invalid reloc tree root keys with stack dump
[BUG] Syzbot reported a crash that an ASSERT() got triggered inside prepare_to_merge().
That ASSERT() makes sure the reloc tree is properly pointed back by its subvolume tree.
[CAUSE] After more debugging output, it turns out we had an invalid reloc tree:
BTRFS error (device loop1): reloc tree mismatch, root 8 has no reloc root, expect reloc root key (-8, 132, 8) gen 17
Note the above root key is (TREE_RELOC_OBJECTID, ROOT_ITEM, QUOTA_TREE_OBJECTID), meaning it's a reloc tree for quota tree.
But reloc trees can only exist for subvolumes, as for non-subvolume trees, we just COW the involved tree block, no need to create a reloc tree since those tree blocks won't be shared with other trees.
Only subvolumes tree can share tree blocks with other trees (thus they have BTRFS_ROOT_SHAREABLE flag).
Thus this new debug output proves my previous assumption that corrupted on-disk data can trigger that ASSERT().
[FIX] Besides the dedicated fix and the graceful exit, also let tree-checker to check such root keys, to make sure reloc trees can only exist for subvolumes.