55 lines
2.2 KiB
Diff
55 lines
2.2 KiB
Diff
From a2ef8773db38d0c3a41761dbed6fc57afa440161 Mon Sep 17 00:00:00 2001
|
|
From: Jens Axboe <axboe@kernel.dk>
|
|
Date: Fri, 13 Jun 2025 13:37:41 -0600
|
|
Subject: nvme: always punt polled uring_cmd end_io work to task_work
|
|
|
|
Currently NVMe uring_cmd completions will complete locally, if they are
|
|
polled. This is done because those completions are always invoked from
|
|
task context. And while that is true, there's no guarantee that it's
|
|
invoked under the right ring context, or even task. If someone does
|
|
NVMe passthrough via multiple threads and with a limited number of
|
|
poll queues, then ringA may find completions from ringB. For that case,
|
|
completing the request may not be sound.
|
|
|
|
Always just punt the passthrough completions via task_work, which will
|
|
redirect the completion, if needed.
|
|
|
|
Cc: stable@vger.kernel.org
|
|
Fixes: 585079b6e425 ("nvme: wire up async polling for io passthrough commands")
|
|
Signed-off-by: Jens Axboe <axboe@kernel.dk>
|
|
---
|
|
drivers/nvme/host/ioctl.c | 21 +++++++--------------
|
|
1 file changed, 7 insertions(+), 14 deletions(-)
|
|
|
|
--- a/drivers/nvme/host/ioctl.c
|
|
+++ b/drivers/nvme/host/ioctl.c
|
|
@@ -429,21 +429,14 @@ static enum rq_end_io_ret nvme_uring_cmd
|
|
pdu->result = le64_to_cpu(nvme_req(req)->result.u64);
|
|
|
|
/*
|
|
- * For iopoll, complete it directly. Note that using the uring_cmd
|
|
- * helper for this is safe only because we check blk_rq_is_poll().
|
|
- * As that returns false if we're NOT on a polled queue, then it's
|
|
- * safe to use the polled completion helper.
|
|
- *
|
|
- * Otherwise, move the completion to task work.
|
|
+ * IOPOLL could potentially complete this request directly, but
|
|
+ * if multiple rings are polling on the same queue, then it's possible
|
|
+ * for one ring to find completions for another ring. Punting the
|
|
+ * completion via task_work will always direct it to the right
|
|
+ * location, rather than potentially complete requests for ringA
|
|
+ * under iopoll invocations from ringB.
|
|
*/
|
|
- if (blk_rq_is_poll(req)) {
|
|
- if (pdu->bio)
|
|
- blk_rq_unmap_user(pdu->bio);
|
|
- io_uring_cmd_iopoll_done(ioucmd, pdu->result, pdu->status);
|
|
- } else {
|
|
- io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_cb);
|
|
- }
|
|
-
|
|
+ io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_cb);
|
|
return RQ_END_IO_FREE;
|
|
}
|
|
|