In the Linux kernel, the following vulnerability has been resolved:
btrfs: fix deadlock in wait_current_trans() due to ignored transaction type
When wait_current_trans() is called during start_transaction(), it currently waits for a blocked transaction without considering whether the given transaction type actually needs to wait for that particular transaction state. The btrfs_blocked_trans_types[] array already defines which transaction types should wait for which transaction states, but this check was missing in wait_current_trans().
This can lead to a deadlock scenario involving two transactions and pending ordered extents:
Transaction A is in TRANS_STATE_COMMIT_DOING state
A worker processing an ordered extent calls start_transaction() with TRANS_JOIN
join_transaction() returns -EBUSY because Transaction A is in TRANS_STATE_COMMIT_DOING
Transaction A moves to TRANS_STATE_UNBLOCKED and completes
A new Transaction B is created (TRANS_STATE_RUNNING)
The ordered extent from step 2 is added to Transaction B's pending ordered extents
Transaction B immediately starts commit by another task and enters TRANS_STATE_COMMIT_START
The worker finally reaches wait_current_trans(), sees Transaction B in TRANS_STATE_COMMIT_START (a blocked state), and waits unconditionally
However, TRANS_JOIN should NOT wait for TRANS_STATE_COMMIT_START according to btrfs_blocked_trans_types[]
Transaction B is waiting for pending ordered extents to complete
Deadlock: Transaction B waits for ordered extent, ordered extent waits for Transaction B
This can be illustrated by the following call stacks: CPU0 CPU1 btrfs_finish_ordered_io() start_transaction(TRANS_JOIN) join_transaction() # -EBUSY...