When THP always Undoes the Go Scavenger's Work

A fleet of Go services kept getting OOM-killed inside containers whose memory limits were generous — two to three times the live heap the runtime reported. We had GODEBUG=madvdontneed=1 set from years back, so I expected RSS to track the heap closely. Instead RSS climbed in a sawtooth that never came back down, while runtime.MemStats.HeapReleased insisted the memory had been handed back to the kernel.

Both were telling the truth. The host had /sys/kernel/mm/transparent_hugepage/enabled set to always.

What the scavenger does

The runtime’s background scavenger walks free heap pages and returns them to the OS with madvise(2), in 64 KiB chunks, paced to spend about 1% of a CPU. Virtual address space is untouched; only the physical backing goes away.

Which flavor of madvise matters. MADV_DONTNEED unmaps the pages on the spot: RSS drops immediately, and the next touch page-faults a fresh zero page. MADV_FREE marks them lazily reclaimable: the kernel takes them only under memory pressure, so RSS stays high and monitoring lies.

Go 1.12 switched Linux to MADV_FREE and added GODEBUG=madvdontneed=1 as the escape hatch. The misleading RSS was enough of a problem that Go 1.16 flipped the default back to MADV_DONTNEED (golang/go#42330). Our flag was redundant on a modern toolchain, but the behavior it asks for is exactly the default, so it is the behavior that matters below.

Keep the MADV_FREE picture in mind, because it is also the reason the lazy variant does not suffer from what follows: pages that were never unmapped leave nothing for the kernel to fill back in.

Enter transparent huge pages

With THP in madvise mode the kernel only builds 2 MiB huge pages for ranges a process explicitly asks for. In always mode every anonymous mapping is eligible: page faults may be served with a whole huge page, and the khugepaged daemon scans in the background collapsing groups of 512 small pages into one huge page.

How aggressive that collapse is comes down to one knob, khugepaged/max_ptes_none. It says how many of the 512 entries may be empty and still be collapsed. The kernel default is 511. One live 4 KiB page in an otherwise empty 2 MiB region is enough: khugepaged allocates a fresh huge page, copies the 4 KiB, and zero-fills the other 2044 KiB.

The fight

Put the two together and the scavenger and khugepaged are working against each other on the same memory.

1. A region of the heap goes mostly free after a GC cycle.
2. The scavenger calls madvise(MADV_DONTNEED) on the free 64 KiB chunks. That splits the huge page and punches holes in the page table: PTEs go from present to none. RSS drops.
3. khugepaged comes around, sees one present PTE and 511 none, which is within max_ptes_none, and collapses the region again.
4. The kernel hands back a full 2 MiB, zero-filled. RSS is back where it was, plus whatever the application grew in the meantime.

The runtime has no way to see step 4. From its point of view the pages were released; HeapReleased still counts them, and GOMEMLIMIT is tuned against heap accounting that no longer matches what the cgroup is charging. Every cycle burns CPU on split, zap, allocate, copy, zero-fill — and the RSS floor ratchets up until the OOM killer ends it.

MADV_FREE would not have raced the same way: lazily freed pages stay mapped until the kernel actually reclaims them, so there are no holes for khugepaged to fill — and no RSS drop either. MADV_DONTNEED is the right call, but it is what creates the sparse regions always is so eager to re-inflate. The kernel considers that a feature; the comment above the check in mm/khugepaged.c reads “default collapse hugepages if there is at least one pte mapped like it would have happened if the vma was large enough during page fault.”

This is a known interaction, and Go used to fight it. Before Go 1.21.1 the runtime worked around the kernel default by marking scavenged memory MADV_NOHUGEPAGE and flipping it back with MADV_HUGEPAGE, unevenly and at a CPU cost. 1.21.1 dropped that workaround, 1.21.4 removed the last of the runtime’s huge page hints, and the memory growth people saw on upgrade became golang/go#64332. The maintainers’ position there is that this belongs to whoever owns the host, not the language runtime, and the GC guide was updated to say so.

The fix

The kernel side is a one-liner either way. Take THP away from anything that does not ask for it:

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

Or keep always and follow the GC guide, which asks for two settings whenever THP is on for Go programs:

echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag

We went with the second. Other workloads on those hosts wanted huge pages, and so did our bigger Go heaps: since 1.21.4 the runtime never asks for huge pages on the heap, so madvise mode means no huge pages for Go at all. With max_ptes_none at zero, khugepaged only collapses regions that are already fully populated, which is the one case where a huge page is pure win. RSS stopped ratcheting the same day.

Same region, same scavenger, same khugepaged scan. The only difference is step 3 now says “not eligible”, so the release in step 2 is the last thing that happens to those pages.

If you cannot touch the host, two process-level escapes exist. Any Go version can call unix.Prctl(unix.PR_SET_THP_DISABLE, 1, 0, 0, 0) at startup to opt the whole process out of THP. Go 1.21.6 and later also ship GODEBUG=disablethp=1, which marks every heap arena MADV_NOHUGEPAGE as it is mapped. The runtime documents that one as a compatibility setting that may be removed, so treat it as a stopgap.

Takeaway

madvdontneed=1 was never the problem, and GOMEMLIMIT was never going to be the fix. When RSS and HeapReleased disagree, the kernel is doing something with memory the runtime believes it gave away; check /sys/kernel/mm/transparent_hugepage/ before you check the heap.