Tachyon frame cache can write into freed thread-state memory #160267

Tachyon reads PyInterpreterState and PyThreadState objects asynchronously. Its frame cache also writes last_profiled_frame and last_profiled_frame_seq into the target thread state.

A thread can exit after Tachyon reads its thread-state address and before the cache write. free_threadstate() then returns the block to the raw allocator. The allocator can reuse that address for unrelated data, and Tachyon can overwrite the new allocation. The reads have the same lifetime problem.

This reproducer just churns short-lived threads with deep stacks:

import os
import threading
import time

DURATION = float(os.environ.get("DURATION", "30"))
WORKERS = int(os.environ.get("WORKERS", "8"))


def deep(n):
    if n:
        return deep(n - 1)
    time.sleep(0.0005)


def churn(deadline):
    while time.monotonic() < deadline:
        t = threading.Thread(target=deep, args=(200,))
        t.start()
        t.join()


deadline = time.monotonic() + DURATION
workers = [threading.Thread(target=churn, args=(deadline,)) for _ in range(WORKERS)]
for w in workers:
    w.start()
for w in workers:
    w.join()
print("done")

Attach the sampling profiler to it:

DURATION=20 PYTHONMALLOC=malloc ./python thread_churn.py &
./python -m profiling.sampling attach -a -r 1khz -d 15 --collapsed -o /tmp/s.txt $!

On main at 39d7c0ff0d (GIL build) the target segfaulted in 6 of 11 runs. The crashes land in different places in the eval loop (_PyEval_EvalFrameDefault, PyObject_Size) on bad object pointers, which looks like heap corruption. Without the profiler the same program always finishes.

The profiler must not access a state after CPython can free or repurpose its storage.

Linked PRs

• gh-160267: Keep thread and interpreter state memory alive for remote profilers #161674
• [3.15] gh-160267: Keep thread and interpreter state memory alive for remote profilers (GH-161674) #161687