Repository navigation
profiling.sampling CPU mode may attribute on-CPU samples to detached tstates when sharing native_thread_id #159312
Copy link
Copy link
Open
Labels
extension-modulesC modules in the Modules dirC modules in the Modules dirtopic-profilingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
My reproducer uses one worker OS thread. The worker enters funA -> funB using tstate0, detaches tstate0, creates and attaches tstate1 in the same interpreter, then runs funC -> funD. Only funD performs CPU work. Despite that, CPU-mode output attributes about half of the direct samples to the suspended funB/tstate0 stack and half to the executing funD/tstate1 stack.
import os
import sys
import threading
import _native
def funD():
print(
f"HOT LOOP funD pid={os.getpid()} tid={threading.get_native_id()}",
file=sys.stderr,
flush=True,
)
value = 1
while True:
value = (value * 1_103_515_245 + 12_345) & 0x7FFF_FFFF
value ^= value >> 7
def funC():
_native.print_tstate("tstate1 in funC")
funD()
def funB():
_native.switch_tstate_and_call(funC)
def funA():
funB()
def work0():
print(
f"WORKER tid={threading.get_native_id()}",
file=sys.stderr,
flush=True,
)
_native.print_tstate("initial tstate0")
funA()
def main():
print(
f"MAIN tid={threading.get_native_id()}",
file=sys.stderr,
flush=True,
)
worker = threading.Thread(target=work0, name="work0")
worker.start()
worker.join()
if __name__ == "__main__":
main()#define _GNU_SOURCE
#include <Python.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>
static long
gettid_linux(void)
{
return (long)syscall(SYS_gettid);
}
static void
print_tstate_info(const char *where, PyThreadState *tstate)
{
PyThreadState *current = PyThreadState_GetUnchecked();
unsigned long native_tid = tstate ? tstate->native_thread_id : 0;
long tid = gettid_linux();
fprintf(stderr,
"TSTATE %-24s tstate=%p current=%p native_thread_id=%lu "
"gettid=%ld match=%s\n",
where, (void *)tstate, (void *)current, native_tid, tid,
tstate && native_tid == (unsigned long)tid ? "yes" : "no");
fflush(stderr);
}
static PyObject *
print_tstate(PyObject *self, PyObject *args)
{
const char *where;
if (!PyArg_ParseTuple(args, "s", &where)) {
return NULL;
}
print_tstate_info(where, PyThreadState_Get());
Py_RETURN_NONE;
}
static PyObject *
switch_tstate_and_call(PyObject *self, PyObject *callback)
{
if (!PyCallable_Check(callback)) {
PyErr_SetString(PyExc_TypeError, "callback must be callable");
return NULL;
}
PyThreadState *tstate0 = PyThreadState_Get();
PyInterpreterState *interp = PyThreadState_GetInterpreter(tstate0);
print_tstate_info("tstate0 in funB", tstate0);
PyThreadState *saved0 = PyEval_SaveThread();
print_tstate_info("tstate0 detached", saved0);
PyThreadState *tstate1 = PyThreadState_New(interp);
if (tstate1 == NULL) {
PyEval_RestoreThread(saved0);
return PyErr_NoMemory();
}
print_tstate_info("tstate1 created", tstate1);
PyEval_RestoreThread(tstate1);
print_tstate_info("tstate1 attached", tstate1);
/* The callback enters funC -> funD, and funD intentionally does not return. */
PyObject *result = PyObject_CallNoArgs(callback);
/* Cleanup remains correct if funC is changed to return. */
PyObject *raised = result ? NULL : PyErr_GetRaisedException();
Py_XDECREF(result);
PyThreadState *saved1 = PyEval_SaveThread();
PyEval_RestoreThread(saved0);
PyThreadState_Clear(saved1);
PyThreadState_Delete(saved1);
if (raised) {
PyErr_SetRaisedException(raised);
return NULL;
}
Py_RETURN_NONE;
}
static PyMethodDef methods[] = {
{"print_tstate", print_tstate, METH_VARARGS, NULL},
{"switch_tstate_and_call", switch_tstate_and_call, METH_O, NULL},
{NULL, NULL, 0, NULL}
};
static PyModuleDef_Slot slots[] = {
{Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
{Py_mod_gil, Py_MOD_GIL_NOT_USED},
{0, NULL}
};
static struct PyModuleDef module = {
PyModuleDef_HEAD_INIT, "_native", NULL, 0, methods, slots
};
PyMODINIT_FUNC
PyInit__native(void)
{
return PyModuleDef_Init(&module);
}and the sample result is:
cpython315-tstate-tachyon-demo.zip
CPython versions tested on:
3.15
Operating systems tested on:
Linux
Linked PRs
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
extension-modulesC modules in the Modules dirC modules in the Modules dirtopic-profilingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
maurycy commentedon Oct 13, 2026
@pablogsal Maybe we should check
tstate_status.bound_gilstatebefore settingTHREAD_STATUS_ON_CPU?pablogsal commentedon Oct 13, 2026
We should probably fix this, but what's the use case for creating a second thread state on the same OS thread in the same interpreter? Did you run into this in an actual application, or is this something you found while experimenting?
pythongh-159312: Confine CPU attribution checks to unbound states
superajun-wsj commentedon Oct 14, 2026
Hi @pablogsal,Since Python 3.15 was released only recently, we haven’t yet used Tachyon with 3.15 in a production workload. This report is based on the targeted reproducer above. The association between a PyThreadState and a native thread can change in CPython.
pablogsal commentedon Oct 14, 2026
Yeah but how you arrived to this reproducer?
superajun-wsj commentedon Oct 14, 2026
Since Tachyon was introduced, we’ve been interested in using it. We’d like to find a way to extend it to support mixed Python/native stack traces, which it does not currently provide.
While studying Tachyon’s implementation, we noticed that its on-CPU mode reconstructs Python call stacks starting from a PyThreadState. In CPython, the association between a thread state and an OS thread can change over time. That led us to suspect that samples might be attributed to the wrong thread state. We constructed the reproducer in this issue to test that hypothesis, and it reproduced the problem. @liuzhijie-0614