91 lines
3.1 KiB
Diff
91 lines
3.1 KiB
Diff
From 7c9d5350d8acfe1b876a8acabdf247b44a803d58 Mon Sep 17 00:00:00 2001
|
|
From: Ryan Roberts <ryan.roberts@arm.com>
|
|
Date: Fri, 6 Jun 2025 10:28:07 +0100
|
|
Subject: mm: close theoretical race where stale TLB entries could linger
|
|
|
|
Commit 3ea277194daa ("mm, mprotect: flush TLB if potentially racing with a
|
|
parallel reclaim leaving stale TLB entries") described a theoretical race
|
|
as such:
|
|
|
|
|
|
"""
|
|
Nadav Amit identified a theoretical race between page reclaim and mprotect
|
|
due to TLB flushes being batched outside of the PTL being held.
|
|
|
|
He described the race as follows:
|
|
|
|
CPU0 CPU1
|
|
---- ----
|
|
user accesses memory using RW PTE
|
|
[PTE now cached in TLB]
|
|
try_to_unmap_one()
|
|
==> ptep_get_and_clear()
|
|
==> set_tlb_ubc_flush_pending()
|
|
mprotect(addr, PROT_READ)
|
|
==> change_pte_range()
|
|
==> [ PTE non-present - no flush ]
|
|
|
|
user writes using cached RW PTE
|
|
...
|
|
|
|
try_to_unmap_flush()
|
|
|
|
The same type of race exists for reads when protecting for PROT_NONE and
|
|
also exists for operations that can leave an old TLB entry behind such as
|
|
munmap, mremap and madvise.
|
|
"""
|
|
|
|
The solution was to introduce flush_tlb_batched_pending() and call it
|
|
under the PTL from mprotect/madvise/munmap/mremap to complete any pending
|
|
tlb flushes.
|
|
|
|
However, while madvise_free_pte_range() and
|
|
madvise_cold_or_pageout_pte_range() were both retro-fitted to call
|
|
flush_tlb_batched_pending() immediately after initially acquiring the PTL,
|
|
they both temporarily release the PTL to split a large folio if they
|
|
stumble upon one. In this case, where re-acquiring the PTL
|
|
flush_tlb_batched_pending() must be called again, but it previously was
|
|
not. Let's fix that.
|
|
|
|
There are 2 Fixes: tags here: the first is the commit that fixed
|
|
madvise_free_pte_range(). The second is the commit that added
|
|
madvise_cold_or_pageout_pte_range(), which looks like it copy/pasted the
|
|
faulty pattern from madvise_free_pte_range().
|
|
|
|
This is a theoretical bug discovered during code review.
|
|
|
|
Link: https://lkml.kernel.org/r/20250606092809.4194056-1-ryan.roberts@arm.com
|
|
Fixes: 3ea277194daa ("mm, mprotect: flush TLB if potentially racing with a parallel reclaim leaving stale TLB entries")
|
|
Fixes: 9c276cc65a58 ("mm: introduce MADV_COLD")
|
|
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
|
|
Reviewed-by: Jann Horn <jannh@google.com>
|
|
Acked-by: David Hildenbrand <david@redhat.com>
|
|
Cc: Liam Howlett <liam.howlett@oracle.com>
|
|
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
|
|
Cc: Mel Gorman <mgorman <mgorman@suse.de>
|
|
Cc: Vlastimil Babka <vbabka@suse.cz>
|
|
Cc: <stable@vger.kernel.org>
|
|
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
---
|
|
mm/madvise.c | 2 ++
|
|
1 file changed, 2 insertions(+)
|
|
|
|
--- a/mm/madvise.c
|
|
+++ b/mm/madvise.c
|
|
@@ -503,6 +503,7 @@ restart:
|
|
pte_offset_map_lock(mm, pmd, addr, &ptl);
|
|
if (!start_pte)
|
|
break;
|
|
+ flush_tlb_batched_pending(mm);
|
|
arch_enter_lazy_mmu_mode();
|
|
if (!err)
|
|
nr = 0;
|
|
@@ -736,6 +737,7 @@ static int madvise_free_pte_range(pmd_t
|
|
start_pte = pte;
|
|
if (!start_pte)
|
|
break;
|
|
+ flush_tlb_batched_pending(mm);
|
|
arch_enter_lazy_mmu_mode();
|
|
if (!err)
|
|
nr = 0;
|