🌐 Mailman 3 October 2020 - Python-list - python.org

Re: Python extension module with callbacks into Python
by Paul Grinberg Oct. 29, 2020

Oct. 29, 2020

> Try calling PyEval_InitThreads() to force the python threading to be all setup. Can you please clarify where/when I should call PyEval_InitThreads()? Is this in the main python thread before any pthread callbacks are generated? If so, should this be done only once?

2 1

Re: Python extension module with callbacks into Python
by Paul Grinberg Oct. 28, 2020

Oct. 28, 2020

> > I am running into unpredictable behavior with my Python extension module that wraps around a C++ library that starts a new pthread and, after doing some work, generates callbacks back into the caller. I've greatly simplified this to a simplistic example which still demonstrates this problem. The following will sometimes generate a Fatal Python error: PyEval_SaveThread: NULL tstate, usually rather quickly. Sometimes it SIGSEGV on tupledealoc. Occasionally this deadlocks. I am at a loss why. Does anyone have any ideas? > You did not say what OS and python version you are working with. I apologize. I am running this on Ubuntu 18.04 under Python 3.6 (default for this Ubuntu's Bionic release) fully updated to the latest provided by Ubuntu's standard repos, which is 3.6.9-1~18.04ubuntu1.3. After posting this question here I continued investigating this problem and found that when I repeat the same test under Python 3.7.3 or 3.8.5 (installed via PyEnv), there are no issues. Then, I went back to the most recent 3.6 release which is 3.6.12 (also installed via PyEnv) and again saw the issue. This seems to suggest that the problem is in 3.6.x Python. > Why do you need to call PyEval_ThreadsInitialized()? its deprecated. > > PyInitialize sets up threads unconditionally since 3.7. I did not realize that PyEval_ThreadsInitialized() is deprecated. However, I was also running under 3.6. I will move away from that once I move to a newer version of Python

2 1

Re: Symlinks already present
by unknown＠example.com Oct. 28, 2020

Oct. 28, 2020

In article <mailman.879.1598986729.9580.python-list(a)python.org>, Grant Edwards <grant.b.edwards(a)gmail.com> wrote: >On 2020-09-01, Richard Damon <Richard(a)Damon-Family.org> wrote: > >> Remember, we are talking about a hypothetical OS that handles hardlinks >> to directories, and defines that .. will point to the parent used to >> come to it, NOT just having current *NIX allowing hardlinks to >> directories with no remediation of the issues cause. > >I can testify from personal experience that SunOS 3/4 was in the >latter category. After creating a hard-link to a directory, things >like fsck and the filesystem dump backup utility got very upset and >confused. IIRC the only way to recover was to nuke the involved >inodes then let fsck try to pick up the pieces and put them in the >lost+found. IIRC, I managed to recover without losing any files, but >it wasn't a fun day. It was a defect ("bug") in the SUNOS that this was possible. So reread this thread. "Let us imagine the situation that a severe, known defect was reintroduced in linux, just for the fun of it." > >-- >Grant Groetjes Albert -- This is the first day of the end of your life. It may not kill you, but it does make your weaker. If you can't beat them, too bad. albert(a)spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

1 0

Re: Symlinks already present
by unknown＠example.com Oct. 28, 2020

Oct. 28, 2020

In article <mailman.856.1598858817.9580.python-list(a)python.org>, Cameron Simpson <cs(a)cskk.id.au> wrote: > >Yeah, makes me ill. That's because these days "pwd" is usually a shell >builtin with funny semantics and a cache/sanity=check against $PWD >(which gets computed as you cd around, typically). And if has a -P >option and friends explicitly because of this hideous stuff. Would you be a fan of nash , the Never Again so complicated SHell ? The base idea that it is a Forth interpreter that has a few shell like features tucked onto it, like executing programs. And if you type pwd, sure as hell it would look through $PATH and execute the exact program you want. (You could defined `pwd' in the interpreter yourself to do something different. If you do that, probably you want it.) > >Cheers, >Cameron Simpson <cs(a)cskk.id.au> Groetjes Albert -- This is the first day of the end of your life. It may not kill you, but it does make your weaker. If you can't beat them, too bad. albert(a)spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

1 0

Python extension module with callbacks into Python
by Paul Grinberg Oct. 28, 2020

Oct. 28, 2020

As full disclosure, I posted this question on StackOverflow as well, but it looks like questions with [Python] [Extension-Module] tags are not frequently answered. The link to my question there is https://stackoverflow.com/questions/64559322/python-extension-module-with-c… I am running into unpredictable behavior with my Python extension module that wraps around a C++ library that starts a new pthread and, after doing some work, generates callbacks back into the caller. I've greatly simplified this to a simplistic example which still demonstrates this problem. The following will sometimes generate a Fatal Python error: PyEval_SaveThread: NULL tstate, usually rather quickly. Sometimes it SIGSEGV on tupledealoc. Occasionally this deadlocks. I am at a loss why. Does anyone have any ideas? The relevant code for this is below, between the ==== sections ==== === python program === import mymod from time import sleep from random import randrange def my_cb1(s): print("Python cb %s" % (s)); for x in range(1,1000): num_cb = randrange(5) + 1 print("Starting %d" % mymod.doit(my_cb1, "myid" + str(x), num_cb)) while True: sleep(1) === Extension Module === #include <pthread.h> #define PY_SSIZE_T_CLEAN #include <Python.h> #include <stddef.h> #include <iostream> #include <map> #include <deque> #include <mutex> #include <functional> #include <thread> static std::map<std::string, PyObject *> cb_map; static std::mutex map_mtx; struct fake_cb_info { fake_cb_info() = delete; fake_cb_info(const unsigned long &num_cb, const std::string &id) : num_cb(num_cb), id(id) { } const unsigned long num_cb; const std::string id; }; static std::deque<struct fake_cb_info> deq; static std::mutex deq_mtx; static bool is_worker_thread_running = false; static std::thread worker_thread; typedef std::function<void(const std::string &id, const std::string &s)> doit_cb_t; static void internal_cb(const std::string &id, const std::string &s) { std::scoped_lock<std::mutex> lk(map_mtx); if (0 != cb_map.count(id)) { PyGILState_STATE gstate; gstate = PyGILState_Ensure(); PyObject *arglist = Py_BuildValue("(s)", s.c_str()); PyObject *result = PyObject_CallObject(cb_map.at(id), arglist); Py_DECREF(arglist); if (NULL == result) { if (NULL == PyErr_Occurred()) { std::cerr << "Unknown error occurred in C callback" << std::endl; } else { PyErr_Print(); } } else { Py_DECREF(result); } PyGILState_Release(gstate); } else { std::cerr << "Unknown callback id " << id << std::endl; } } void static worker() { size_t x = 0; while(true) { std::scoped_lock<std::mutex> lk(deq_mtx); if (deq.size() == 0) { usleep(1000); continue; } auto info = deq.front(); deq.pop_front(); for (unsigned long i=0; i<info.num_cb; i++) { internal_cb(info.id, std::to_string(x++)); } } } PyObject * _wrap_doit(void *self, PyObject *args, PyObject *kwargs) { PyObject *py_retval; PyThreadState *py_thread_state = NULL; PyObject *cb; const char *id = NULL; Py_ssize_t id_len; std::string id_std; unsigned long num_callbacks; const char *keywords[] = {"cb_func", "id", "num_cb", NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwargs, (char *) "Os#k", (char **) keywords, &cb, &id, &id_len, &num_callbacks)) { abort(); } if (!PyCallable_Check(cb)) { abort(); } id_std = std::string(id, id_len); { std::scoped_lock<std::mutex> lk(map_mtx); if (0 == cb_map.count(id_std)) { Py_INCREF(cb); cb_map.insert(std::make_pair(id_std, cb)); // N.B. The corresponding Py_DECREF for the callback function PyObject // is intentionally not here. It is in another extension module method // that is not listed here (just trying to keep this example as small // and lean as possible) } else { std::cerr << "Only one callback for ID!" << std::endl; abort(); } } if (PyEval_ThreadsInitialized ()) { std::cout << "Saving thread" << std::endl; py_thread_state = PyEval_SaveThread(); } { // Stash away the info so that we will know how many callbacks to // generate and sleep a bit. This is to simulate a real external library // doing work which will, in turn, generate callbacks struct fake_cb_info info(num_callbacks, id_std); std::scoped_lock<std::mutex> lk(deq_mtx); deq.push_back(info); if (!is_worker_thread_running) { std::cout << "@@@@ Creating a new thread\n"; worker_thread = std::thread(&worker); pthread_setname_np(worker_thread.native_handle(), "worker_thread"); worker_thread.detach(); is_worker_thread_running = true; } usleep(10000); } if (py_thread_state) { std::cout << "Restoring thread" << std::endl; PyEval_RestoreThread(py_thread_state); } py_retval = Py_BuildValue((char *) "k", num_callbacks); return py_retval; } static PyMethodDef mymod_functions[] = { { (char *) "doit", (PyCFunction) _wrap_doit, METH_KEYWORDS | METH_VARARGS, "Generate requested number of multi-threaded callbacks.\n doit(callback_fn, id, num_callbacks)" }, {NULL, NULL, 0, NULL} }; static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "mymod", "pthread test module", -1, mymod_functions, }; #define MOD_ERROR NULL #define MOD_INIT(name) PyObject* PyInit_##name(void) #define MOD_RETURN(val) val #if defined(__cplusplus) extern "C" #endif #if defined(__GNUC__) && __GNUC__ >= 4 __attribute__ ((visibility("default"))) #endif MOD_INIT(mymod) { PyObject *m = PyModule_Create(&moduledef); if (m == NULL) { return MOD_ERROR; } return MOD_RETURN(m); }

2 1

GUI (tkinter) popularity and job prospects for
by Lammie Jonson Oct. 27, 2020

Oct. 27, 2020

I have been a rails developer as well as JS/react. I had wanted to look at python a bit due to it's popularity. I looked at tkinter which seems to have quite a few examples out there, but when I searched indeed.com for tkinter and wxpython it appeared that there was hardly any job listings mentioning those. Why is that ? It's a bit of a demotivating factor to get very serious with tk etc. I was going to look at something like tensorflow perhaps, though I am not sure if machine learning is that easy to pickup or not

11 13

Re: Question on ABC classes
by Julio Di Egidio Oct. 26, 2020

Oct. 26, 2020

On Thursday, 22 October 2020 23:04:25 UTC+2, Ethan Furman wrote: > On 10/22/20 9:25 AM, Julio Di Egidio wrote: > > > Now, I do read in the docs that that is as intended, > > but I am not understanding the rationale of it: why > > only if there are abstract methods defined in an ABC > > class is instantiation disallowed? IOW, why isn't > > subclassing from ABC enough? > > Let's say you subclass from ABC: > > class Abstract(ABC): > pass > > Then you subclass from that: > > class Concrete(Abstract): > pass > > Then subclass from that: > > class MoreConcrete(Concrete): > pass > > If you do a > > issubclass(<any of the above classes>, ABC) > > you'll get > > True Ah, right, that's the point I was missing: how to tell the compiler when a more derived class is *not* abstract... I was indeed making the mistake of inheriting from ABC also in the derived classes, and omitting it in the classes that are eventually concrete, not realising that ABC isn't just a keywork or a decorator, so it gets inherited all the way. > The idea behind abstract classes is the prevention of creating > non-functional instances Well, in Python, not in any other OO language, where abstract is just synonym with must-override (hence cannot instantiate), no other constraints. I am now thinking whether I could achieve the "standard" behaviour via another approach, say with decorators, somehow intercepting calls to __new__... maybe. Anyway, abstract classes are the gist of most library code, and I remain a bit puzzled by the behaviour implemented in Python: but I am not complaining, I know it will take me some time before I actually understand the language... For now, thank you and Marco very much for your feedback, Julio

4 4

Re: GUI (tkinter) popularity and job prospects for
by Edmondo Giovannozzi Oct. 26, 2020

Oct. 26, 2020

Il giorno venerdì 23 ottobre 2020 alle 18:55:53 UTC+2 john... ha scritto: > On 23/10/2020 05:47, Grant Edwards wrote: > > > >> I think that commercial desktop applications with a python > >> compatible GUI would likely use QT or a Python binding thereof. > > Agreed. If you want to improve you "hirability" for GUI application > > development, I would probably put Qt first. Then gobject or > > wx. Tkinter would probably be last. > I've used tkinter and wxPython occasionally in the past for 1 off test > tasks (and interest). What's the advantage of Qt? > > John I use PyQt in research application, there is a library pyqtgraph (http://www.pyqtgraph.org/) based on Qt that is very fast, you can zoom and pan even complicated plots, it is much faster than matplotlib. For me, the graphic makes the difference.

1 0

Regarding the issue I had faced
by Ayush Mishra Oct. 26, 2020

Oct. 26, 2020

Hi I would to like to say that Python is a very good language with a vast area of application and an easy way to start for the beginner. I am also a beginner and I am very much comfortable with syntax of Python. Now coming to my problem. Say for example if I have created a n HTML file and want to edit it in Brackets , I will right on the file and select the option “ Edit with Bracket” . We also have a similar thing in IDLE too , when we right click on .py file we get the option “ Edit with IDLE” comes but I don’t get that option on right clicking .py file . One more hint regarding the problem I mentioned is that when I select the repair option in Python Setup window a dialog box appears stating “ Ensure you have access to <location_of _python folder> “ . I am using Windows 10 Home 64-bit (Specification of my laptop) , I hope it will help. Please contact me on the same email id to help me resolve the issue . I would be very much thankful to developer team for the help from your side. Sent from Mail for Windows 10

1 0

_pydoc.css
by Ian Gay Oct. 26, 2020

Oct. 26, 2020

The default colors of pydoc are truly horrendous! Has anyone written a _pydoc.css file to produce something reasonable? (running 3.6 on OpenSuse 15.1) Ian -- *********** To reply by e-mail, make w single in address **************

1 0