Skip to content

profiling.sampling CPU mode may attribute on-CPU samples to detached tstates when sharing native_thread_id #159312

Open

Description

@superajun-wsj

Bug report

My reproducer uses one worker OS thread. The worker enters funA -> funB using tstate0, detaches tstate0, creates and attaches tstate1 in the same interpreter, then runs funC -> funD. Only funD performs CPU work. Despite that, CPU-mode output attributes about half of the direct samples to the suspended funB/tstate0 stack and half to the executing funD/tstate1 stack.

import os
import sys
import threading

import _native


def funD():
    print(
        f"HOT LOOP funD pid={os.getpid()} tid={threading.get_native_id()}",
        file=sys.stderr,
        flush=True,
    )
    value = 1
    while True:
        value = (value * 1_103_515_245 + 12_345) & 0x7FFF_FFFF
        value ^= value >> 7


def funC():
    _native.print_tstate("tstate1 in funC")
    funD()


def funB():
    _native.switch_tstate_and_call(funC)


def funA():
    funB()


def work0():
    print(
        f"WORKER tid={threading.get_native_id()}",
        file=sys.stderr,
        flush=True,
    )
    _native.print_tstate("initial tstate0")
    funA()


def main():
    print(
        f"MAIN tid={threading.get_native_id()}",
        file=sys.stderr,
        flush=True,
    )
    worker = threading.Thread(target=work0, name="work0")
    worker.start()
    worker.join()


if __name__ == "__main__":
    main()
#define _GNU_SOURCE
#include <Python.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>

static long
gettid_linux(void)
{
    return (long)syscall(SYS_gettid);
}

static void
print_tstate_info(const char *where, PyThreadState *tstate)
{
    PyThreadState *current = PyThreadState_GetUnchecked();
    unsigned long native_tid = tstate ? tstate->native_thread_id : 0;
    long tid = gettid_linux();

    fprintf(stderr,
            "TSTATE %-24s tstate=%p current=%p native_thread_id=%lu "
            "gettid=%ld match=%s\n",
            where, (void *)tstate, (void *)current, native_tid, tid,
            tstate && native_tid == (unsigned long)tid ? "yes" : "no");
    fflush(stderr);
}

static PyObject *
print_tstate(PyObject *self, PyObject *args)
{
    const char *where;
    if (!PyArg_ParseTuple(args, "s", &where)) {
        return NULL;
    }
    print_tstate_info(where, PyThreadState_Get());
    Py_RETURN_NONE;
}

static PyObject *
switch_tstate_and_call(PyObject *self, PyObject *callback)
{
    if (!PyCallable_Check(callback)) {
        PyErr_SetString(PyExc_TypeError, "callback must be callable");
        return NULL;
    }

    PyThreadState *tstate0 = PyThreadState_Get();
    PyInterpreterState *interp = PyThreadState_GetInterpreter(tstate0);
    print_tstate_info("tstate0 in funB", tstate0);

    PyThreadState *saved0 = PyEval_SaveThread();
    print_tstate_info("tstate0 detached", saved0);

    PyThreadState *tstate1 = PyThreadState_New(interp);
    if (tstate1 == NULL) {
        PyEval_RestoreThread(saved0);
        return PyErr_NoMemory();
    }

    print_tstate_info("tstate1 created", tstate1);
    PyEval_RestoreThread(tstate1);
    print_tstate_info("tstate1 attached", tstate1);

    /* The callback enters funC -> funD, and funD intentionally does not return. */
    PyObject *result = PyObject_CallNoArgs(callback);

    /* Cleanup remains correct if funC is changed to return. */
    PyObject *raised = result ? NULL : PyErr_GetRaisedException();
    Py_XDECREF(result);
    PyThreadState *saved1 = PyEval_SaveThread();
    PyEval_RestoreThread(saved0);
    PyThreadState_Clear(saved1);
    PyThreadState_Delete(saved1);

    if (raised) {
        PyErr_SetRaisedException(raised);
        return NULL;
    }
    Py_RETURN_NONE;
}

static PyMethodDef methods[] = {
    {"print_tstate", print_tstate, METH_VARARGS, NULL},
    {"switch_tstate_and_call", switch_tstate_and_call, METH_O, NULL},
    {NULL, NULL, 0, NULL}
};

static PyModuleDef_Slot slots[] = {
    {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
    {Py_mod_gil, Py_MOD_GIL_NOT_USED},
    {0, NULL}
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT, "_native", NULL, 0, methods, slots
};

PyMODINIT_FUNC
PyInit__native(void)
{
    return PyModuleDef_Init(&module);
}

and the sample result is:

Image

cpython315-tstate-tachyon-demo.zip

tachyon-flamegraph.html

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

Activity

  1. maurycy commented on Oct 13, 2026

    @maurycy
    Contributor

    @pablogsal Maybe we should check tstate_status.bound_gilstate before setting THREAD_STATUS_ON_CPU?

  2. pablogsal commented on Oct 13, 2026

    @pablogsal
    Member

    We should probably fix this, but what's the use case for creating a second thread state on the same OS thread in the same interpreter? Did you run into this in an actual application, or is this something you found while experimenting?

  3. added a commit that references this issue on Oct 13, 2026
  4. superajun-wsj commented on Oct 14, 2026

    @superajun-wsj
    Author

    Hi @pablogsal,Since Python 3.15 was released only recently, we haven’t yet used Tachyon with 3.15 in a production workload. This report is based on the targeted reproducer above. The association between a PyThreadState and a native thread can change in CPython.

  5. pablogsal commented on Oct 14, 2026

    @pablogsal
    Member

    This report is based on the targeted reproducer above

    Yeah but how you arrived to this reproducer?

  6. superajun-wsj commented on Oct 14, 2026

    @superajun-wsj
    Author

    Since Tachyon was introduced, we’ve been interested in using it. We’d like to find a way to extend it to support mixed Python/native stack traces, which it does not currently provide.
    While studying Tachyon’s implementation, we noticed that its on-CPU mode reconstructs Python call stacks starting from a PyThreadState. In CPython, the association between a thread state and an OS thread can change over time. That led us to suspect that samples might be attributed to the wrong thread state. We constructed the reproducer in this issue to test that hypothesis, and it reproduced the problem. @liuzhijie-0614

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions