Repository navigation
gh-160267: Keep thread and interpreter state memory alive for remote profilers - #161674
Conversation
|
CC @ivonastojanovic can you make a review when you have time? |
dfd69e1 to
026cd39
Compare
ivonastojanovic
left a comment
There was a problem hiding this comment.
Looks good! I left one nit and one question just for my understanding.
| } | ||
|
|
||
| static void | ||
| recycle_state(void **freelist, void *state, size_t size) |
There was a problem hiding this comment.
A question for my understanding: if we keep the allocated states around now, could that interfere with ASan's detection?
There was a problem hiding this comment.
A little: a stale access to a deleted state from inside CPython now reads a zeroed state instead of being reported. None of our other free lists poison their entries for ASan either, so I don't think it matters. If we want it, we could poison entries while they sit on the list and unpoison them on reuse. The profiler writes through process_vm_writev, which ASan doesn't see.
…e for remote profilers
|
Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15. |
|
GH-161687 is a backport of this pull request to the 3.15 branch. |
We discussed this at the sprint.
Tachyon reads thread and interpreter states of a running process without synchronizing with their lifetime, and its frame cache writes
last_profiled_frameinto thread states. Once a state is freed, a late write corrupts whatever malloc puts there next.Instead of freeing them, deleted thread and interpreter states go onto per-type free lists and are reused for new states, so a stale access can only reach a state of the same type. The lists are freed in
_PyRuntimeState_Fini(). An interpreter whose embedded initial thread state is still owned (for example by a daemon thread) is not recycled.With the reproducer from the issue (
PYTHONMALLOC=malloc, GIL build), main segfaults in 14/16 thread-churn runs and 13/15 subinterpreter-churn runs. With this change, 0/36 (and 0/49 with an earlier revision of it). Thread and subinterpreter creation benchmarks show no slowdown. The cost is that memory for the peak number of live states is kept until finalization.