🌐 Mailman 3 July 2018 - Python-list - python.org

`import somemodule` does not check all paths in `sys.path`
by stefand986＠gmail.com July 30, 2018

July 30, 2018

Hi everyone, I am currently investigating an interesting bug in homeassistant which is a bit hard to reproduce. I already dug so deep that I ask here, because I think the root cause has to do with how Python locates and loads modules/packages. The "story" is: In homeassistant (short HA) some modules are installed at runtime during first start ("bootstrapping"), and for some reason loading the modules fails directly after installing the modules. Subsequent starts work fine. The modules that are installed during bootstrapping are installed to `~/.homeassistant/deps/lib/python3.7/site-packages`. Installing the modules happens in a different thread than where the modules are imported. (However as far as I can tell the importing thread waits for the install to finish.) Now the interesting thing is that when I start HA in verbose mode, I can see that Python is trying not all of the paths that are in `sys.path` (in the following snippet I cut out some stuff for brevity. Link to the full log at the end). As one can see the `~/.homeassistant/deps/lib/python3.7/site-packages` location is not tried and hence the import fails: ``` # trying /home/pyuser/.homeassistant/sqlalchemy.cpython-37m-x86_64-linux-gnu.so # trying /home/pyuser/.homeassistant/sqlalchemy.abi3.so # trying /home/pyuser/.homeassistant/sqlalchemy.so # trying /home/pyuser/.homeassistant/sqlalchemy.py # trying /home/pyuser/.homeassistant/sqlalchemy.pyc # trying /home/pyuser/sqlalchemy.cpython-37m-x86_64-linux-gnu.so [...] # trying /usr/local/lib/python3.7/sqlalchemy.cpython-37m-x86_64-linux-gnu.so [...] # trying /usr/local/lib/python3.7/lib-dynload/sqlalchemy.cpython-37m-x86_64-linux-gnu.so [...] # trying /home/pyuser/.local/lib/python3.7/site-packages/sqlalchemy.cpython-37m-x86_64-linux-gnu.so [...] # trying /usr/local/lib/python3.7/site-packages/sqlalchemy.cpython-37m-x86_64-linux-gnu.so [...] Exception in thread Recorder: Traceback (most recent call last): File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/home/pyuser/.local/lib/python3.7/site-packages/homeassistant/components/recorder/__init__.py", line 187, in run from .models import States, Events File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/pyuser/.local/lib/python3.7/site-packages/homeassistant/components/recorder/models.py", line 8, in <module> from sqlalchemy import ( File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked ModuleNotFoundError: No module named 'sqlalchemy' ``` I also checked that this path is also really in `sys.path` at the time the import happens. This is `sys.path` printed out directly before the `import sqlalchemy statement`: ``` ['/home/pyuser/.homeassistant', '/home/pyuser/.homeassistant/deps/lib/python3.7/site-packages', '/home/pyuser', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/pyuser/.local/lib/python3.7/site-packages', '/usr/local/lib/python3.7/site-packages'] ``` What I already tried is to do a importlib.invalidate_caches() after installing the modules. However this does not fix it. So what is happening here? Why does the import skip `~/.homeassistant/deps/lib/python3.7/site-packages`? Any tips on how I can investigate further? Link to the full log: https://gist.github.com/StefanD986/fbf08426ceaad657929b973cd5add25a#file-ho…

4 3

Re: Python bug in ArcGIS - Urban Network analysis tool
by Станимира Николова July 30, 2018

July 30, 2018

неделя, 29 юли 2018 г., 23:41:01 UTC+3, MRAB написа: > On 2018-07-29 18:56, stanimira.s.93(a)gmail.com wrote: > > Hi there, > > > > I'm trying make reach analyze with Urban network analysis (UNA) tool in ArcGIS. It's based on points for buildings and pedestrian network of acceptability steets. The aim is to show how many people from each building can reach different building in radius of 150 meters (so i can find the density of the pedestrian routes). > > > > I run Urban network analysis but It shows similar mistake several times. The UNA tool is free plugin that i downloaded, it's not from the default intalled in ArcGIS packed. It ask for integer data. > > I checked the type of the attributes, it's all integer. PLus it's all in geo data based file. > > > > Unfortunately I don't understand Python, I'm one of those who use ArcGIS as sample customer. > > > > This is the mistake: > > > > Start Time: Fri Jul 27 14:48:32 2018 > > Running script Centrality... > > [started] Copying input buildings > > [finished] > > [1 started] Computing adjacency list > > [1 failed] > > Not successful > > Completed script Centrality... > > Succeeded at Fri Jul 27 14:48:36 2018 (Elapsed Time: 4,56 seconds) > > > > Any suggestions? How it's calling these adjaency list? What could be wrong? I even don't know how to get debugger, so it could give me more information. > > > > Thank you previously, > > Mira > > > All you've shown is that says that it was unsuccessful, and it doesn't > say why. > > We'd need a lot more info than that! > > The best way to debug it is to try it with the minimal amount of test > data that should give a result. I'm sorry. Could someone help me by telling me how to make these test data? (so i could send it to You) I realize the review of the bug is just the customer view. Any chance to help me out with the test of the process? Thank you previously. If there is another forum that i have to write to, i will switch. The fact is that in the Themes for UNA tool or GIS software, almost no one respond.

2 1

Using Python on a fork-less POSIX-like OS
by Barath Aron July 29, 2018

July 29, 2018

Hello Python list, I intend to cross-compile Python v3.6.6 to Threos ( https://threos.io ) operating system. Threos is supports a quite large set from POSIX and C89/C99. Unfortunately, Threos lacks fork(2), but provides posix_spawn(3) instead. I already made some local changes in posixmodule.c to compile due to some features are detected as present but actually not supported, like HAVE_FORK -- I blame autotools for this :-). I don't know, however, whether the Python shall cross-compile without issues. My question is that the _posixsubprocess.c can be prepared to use posix_spawn(3) instead of fork(2)? Maybe the UNIX/Linux version can also benefit from it, see: https://salsa.debian.org/ruby-team/ruby-posix-spawn Best regards, Aron

3 2

Python bug in ArcGIS - Urban Network analysis tool
by stanimira.s.93＠gmail.com July 29, 2018

July 29, 2018

Hi there, I'm trying make reach analyze with Urban network analysis (UNA) tool in ArcGIS. It's based on points for buildings and pedestrian network of acceptability steets. The aim is to show how many people from each building can reach different building in radius of 150 meters (so i can find the density of the pedestrian routes). I run Urban network analysis but It shows similar mistake several times. The UNA tool is free plugin that i downloaded, it's not from the default intalled in ArcGIS packed. It ask for integer data. I checked the type of the attributes, it's all integer. PLus it's all in geo data based file. Unfortunately I don't understand Python, I'm one of those who use ArcGIS as sample customer. This is the mistake: Start Time: Fri Jul 27 14:48:32 2018 Running script Centrality... [started] Copying input buildings [finished] [1 started] Computing adjacency list [1 failed] Not successful Completed script Centrality... Succeeded at Fri Jul 27 14:48:36 2018 (Elapsed Time: 4,56 seconds) Any suggestions? How it's calling these adjaency list? What could be wrong? I even don't know how to get debugger, so it could give me more information. Thank you previously, Mira

2 1

Demo-Mode in IPython broken?
by Albert Brandl July 29, 2018

July 29, 2018

Hi! The IPython documentation mentions a demo mode that can be used to execute chunks of a Python file that are separated by special comments (e.g. "# <demo> stop"). The annotated Python file can be executed by importing `Demo` from `IPython.lib.demo`, instantiating it using the file name and calling the `Demo` instance: from IPython.lib.demo import Demo d = Demo("some_file.py") d() According to the documentation, all names that are defined in such a chunk are transferred to the global IPython namespace. This would allow me to interactively inspect the names etc. It looks like this feature does not work in the version 6.4.0 of IPython: the global namespace does not change after executing any chunk of a Python file. The Demo instance has a dictionary called `user_ns` that _is_ updated when a chunk is executed, but using it to access the names is quite clumsy ;-). Is this a known bug? Do you have a suggestion how I could emulate the behavior of the demo mode by other means (not necessarily with IPython)? TIA and best regards, Albert

1 0

Why is multiprocessing.Queue creating a thread ?
by Léo El Amri July 29, 2018

July 29, 2018

Hello list, This is a simple question: I wonder what is the reason behind multiprocessing.Queue creating a thread to send objects through a multiprocessing.connection.Connection. I plan to implement an asyncio "aware" Connection class. And while reading the source code of the multiprocessing module, I found that (As outlined in the documentation) Queue is indeed creating a thread. I want to know if I'm missing something. -- Leo

2 1

Re: `import somemodule` does not check all paths in `sys.path`
by stefand986＠gmail.com July 29, 2018

July 29, 2018

Am Sonntag, 29. Juli 2018 01:32:07 UTC+2 schrieb MRAB: That was also my first suspicion, but I checked that the path exists by printing the directory contents out, right before the ìmport ` statement. The "please check your config" message are a result of another package (aiohttp_cors) that is also dynamically installed and then not found when it is imported. > The log contains entries that say "Please check your config". To me it > looks like it hasn't been installed. > > Does the path > /home/pyuser/.homeassistant/deps/lib/python3.7/site-packages actually exist?

1 0

Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS
by Bartosz Golaszewski July 28, 2018

July 28, 2018

Hi! A user recently reported a memory leak in python bindings (C extension module) to a C library[1] I wrote. I've been trying to fix it since but so far without success. Since I'm probably dealing with a space leak rather than actual memory leak, valgrind didn't help much even when using malloc as allocator. I'm now trying to use PYTHONMALLOCSTATS but need some help on how to interpret the output emitted it's enabled. I'm setting PYTHONMALLOCSTATS=1 & PYTHONMALLOC=pymalloc_debug and then running the script that triggers the leak. The last debug message is as follows: class size num pools blocks in use avail blocks ----- ---- --------- ------------- ------------ 3 32 1 2 124 4 40 2 9 193 5 48 1 3 81 6 56 1 4 68 7 64 11 295 398 8 72 9 260 244 9 80 36 831 969 10 88 79 1542 2092 11 96 131 3262 2240 12 104 70 1903 757 13 112 19 289 395 14 120 11 139 224 15 128 7 88 129 16 136 6 70 104 17 144 5 44 96 18 152 4 47 57 19 160 24 342 258 20 168 4 17 79 21 176 24 360 192 22 184 2 8 36 23 192 2 11 31 24 200 22 227 213 25 208 3 13 44 26 216 3 7 47 27 224 2 13 23 28 232 2 6 28 29 240 3 8 40 30 248 2 10 22 31 256 3 10 35 32 264 2 9 21 33 272 3 11 31 34 280 2 10 18 35 288 1 3 11 36 296 2 9 17 37 304 2 9 17 38 312 2 5 19 39 320 2 5 19 40 328 14 105 63 41 336 2 3 21 42 344 1 3 8 43 352 1 3 8 44 360 2 3 19 45 368 1 3 8 46 376 1 1 9 47 384 2 4 16 48 392 2 6 14 49 400 2 3 17 50 408 1 1 8 51 416 1 3 6 52 424 2 4 14 53 432 50967 458680 23 54 440 3 9 18 55 448 4 15 21 56 456 4 12 20 57 464 3 8 16 58 472 2 5 11 59 480 1 4 4 60 488 1 3 5 61 496 4 11 21 62 504 4 13 19 63 512 2 7 7 # times object malloc called = 2,811,245 # arenas allocated total = 810 # arenas reclaimed = 0 # arenas highwater mark = 810 # arenas allocated current = 810 810 arenas * 262144 bytes/arena = 212,336,640 # bytes in allocated blocks = 199,277,432 # bytes in available blocks = 1,138,472 308 unused pools * 4096 bytes = 1,261,568 # bytes lost to pool headers = 2,473,536 # bytes lost to quantization = 8,185,632 # bytes lost to arena alignment = 0 Total = 212,336,640 The number of pools in arena 53 continuously grows. Its size column says: 432. I couldn't find any documentation on what it means but I assume it's an allocation of 432 bytes. I launched gdb and set the following breakpoint: 'b PyMem_Malloc if size == 432' but the breakpoint was never triggered. I tried the same for PyObject_Malloc and still nothing. How do I use the info produced by PYTHONMALLOCSTATS do get to the culprit of the leak? Is there anything wrong in my reasoning here? Best regards, Bartosz Golaszewski [1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/

3 5

Re: Checking whether type is None
by Steven D'Aprano July 28, 2018

July 28, 2018

On Sat, 28 Jul 2018 09:47:07 +0000, Gilmeh Serda wrote: > On Tue, 24 Jul 2018 12:33:27 -0700, Tobiah wrote: > >> I'm trying to get away from things like: >> >> >>> type(thing) is type(None) > > How about: > > >>> some_thing = None > >>> type(some_thing).__str__(some_thing) > 'None' > > Equally weird, I'd say, but what the heck... class Foo: def __str__(self): return 'None' -- Steven D'Aprano "Ever since I learned about confirmation bias, I've been seeing it everywhere." -- Jon Ronson

2 1

Guilty as charged
by Schachner, Joseph July 27, 2018

July 27, 2018

Re: "...while thing == None would work perfectly in almost all cases in practice, it's unidiomatic and suggests the writer isn't quite comfortable with the workings of the language" I admit, I've been a C++ programmer for many years and a Python programmer for only about 5 years. But, I did learn something here. I've read books on Python and somehow still did not know that None is guaranteed to be a singleton in all implementations, and that not only does "thing is None" therefore work (guaranteed), but that that is the Pythonic thing to do. ... Maybe I missed it, but I'll remember it now. --- Joseph S.

2 1