[Python-Dev] Why is the GIL not in PyInterpreterState?

[Tim Peters]

[Tobias Oberstein]
> Is there a reason why PyInterpreterState is not defined like so:
>
> typedef struct _is {
>    ..
>    PyThread_type_lock interpreter_lock;
>    PyThreadState *_PyThreadState_Current;
>    ..
> } PyInterpreterState;
>
> which could be the basis for support of multiple, separated
> interpreters within a single process.

The GIL is a process-level lock now, and must be now -- pretending each
thread had its own GIL wouldn't work (unless they all happened to be the
same GIL).  The docs for Py_NewInterpreter explain some of the problems:

    http://www.python.org/doc/current/api/initialization.html

but there are many others.  As a simple example of another, integer objects
are allocated out of a special free list, and code manipulating that list
implicitly assumes it has exclusive access to the list.  This is so given
the process-level GIL.

> ...
> Would it be so hard to proceed like indicated below?

Nobody knows -- you'll have to try it and see.  But I see Mark already tried
to depress you, so I won't belabor it <wink>.

OTOH, if your goal is N completely independent interpreters, why not fire up
N processes?  Then independence comes for free and you don't have to change
a thing.  If the answer is that you don't really want them to be
*completely* independent, then maintaining the refcounts on the objects you
want them to share will be a nightmare:  every Py_INCREF and Py_DECREF in
the codebase also relies on the process-level GIL for proper operation.