🌐 Mailman 3 September 2019 - Python-list - python.org

Announcing colour-text and colour-print
by Barry Scott Oct. 1, 2019

Oct. 1, 2019

See https://pypi.org/project/colour-text/ for documentation with colourful examples. Install using pip: python -m pip install colour-text Many command line tools now colour there output to make it easier to pick out important details. I want to do the same for my commands that I tend to write in Python, bash and Windows CMD. The solution needs to work for Unix, macOS and Windows. Embedding escape sequences I found to be too low level. And the existing python libraries a bit too verbose. My solution is to use a simple markup for adding colour to text. The markup delimited defaults to "<>". Names of colours include semantic names like, info, error and em. Here is an example: from colour_text import ColourText ct = ColourText() ct.initTerminal() print( ct( "The next section is in green: <>green example<>." ) ) I also want to be able to add colour output to shell scripts and Windows CMD scripts. When pip installs colour-text it also installs the colour-print command for use from scripts. For example: $ colour-print "<>info Info:<> this is an <>em informational<> message" $ colour-print "<>error Error: This is an error message<>" And you can also pass in arguments in a printf or python % style, only %s is supported. $ colour-print "<>info Info:<> Home folder is %s" "$HOME" Barry

4 3

Re: pathlib
by Barry Scott Oct. 1, 2019

Oct. 1, 2019

> On 30 Sep 2019, at 14:20, Dan Sommers <2QdxY4RzWzUUiLuE(a)potatochowder.com> wrote: > > That's the wording I read. I inferred that "path-handling operations > which don't actually access a filesystem" meant an object that didn't > necessarily represent an actual file, and that "provide methods to do > system calls on path objects" did indicate an actual file. From the > existence of Path.read_bytes, I inferred that at least some Path objects > represent (and operate on) actual existing files. I've been doing this > for a long time, and I may have read my expecations into those words. pathlib.Path() adds the semantics of operating system file rules to strings. The addition of open(), read_bytes() etc is a convenience to the programmer. Some people consider this a mistake other a great feature. You can write this: pathlib.Path("name").open() Which is the same as: open(pathlib.Path("name")) Which is the same as: open("name") You would not expect str to track the file, why expect Path() to? Barry

4 7

Re: Delay in python startup.
by Tobiah Oct. 1, 2019

Oct. 1, 2019

On 9/30/19 9:54 AM, Chris Angelico wrote: > On Tue, Oct 1, 2019 at 1:56 AM Tobiah <toby(a)tobiah.org> wrote: >> >> I don't have a lot of information, so here goes a shot in >> the dark. One day I started experiencing a delay when >> starting python. I'm on Ubuntu 16.04. It takes three >> seconds to get a prompt when I type 'python' on the command >> line (Python 2.7.12). When I run a script that imports >> packages, it takes longer, up to 17 seconds just to do >> the imports. Python3 is not affected, and is snappy as >> expected. >> >> That's all I know. I'm hoping someone else has seen this. >> I'm about ready to wipe the drive and upgrade to 18.04. >> > > Python 2 and Python 3 have completely independent installations, so it > could be a lot of things. First question: Does it take three seconds > *every* time you type 'python', or only the first time? If it's slow > the first time but then fast, it's probably just a matter of disk > caching; running Python 3 doesn't pre-cache the files for Python 2, so > you have to pay the load-time cost anew. > > If it's slow every time, though, you may have something messing with > your startup. Try "python -S" and "python -E" see if they're just as > slow. That would be a way to delve into things a bit. > > ChrisA > It was much faster with -S and instantaneous with -E. I had a directory in my PYTHONPATH that I mount with sshfs from a server. For some reason that mount is very slow. Thanks for helping me figure this out. Tobiah

3 2

Delay in python startup.
by Tobiah Sept. 30, 2019

Sept. 30, 2019

I don't have a lot of information, so here goes a shot in the dark. One day I started experiencing a delay when starting python. I'm on Ubuntu 16.04. It takes three seconds to get a prompt when I type 'python' on the command line (Python 2.7.12). When I run a script that imports packages, it takes longer, up to 17 seconds just to do the imports. Python3 is not affected, and is snappy as expected. That's all I know. I'm hoping someone else has seen this. I'm about ready to wipe the drive and upgrade to 18.04.

2 1

Hi how do I import files inside a txt file?
by Spencer Du Sept. 30, 2019

Sept. 30, 2019

Hi How do i import files inside a txt file if they exist in the current directory? Here is the current code but I dont know how to do it correctly. import paho.mqtt.client as mqtt from mqtt import * import importlib import os import os.path # from stateMachine import * with open("list_of_devices.txt", "r") as reader: for item in reader: try: os.getcwd() print("hi") except: print("error") This is "list_of_devices.txt": test1,test2 Each name refers to a python file. Thanks Spencer

4 5

Announcing config-path an OS independent configuration file paths
by Barry Scott Sept. 30, 2019

Sept. 30, 2019

See https://pypi.org/project/config-path/ for documentation. Install using pip: python -m pip install config-path I write tools that run on macOS, Unix and Windows systems. Each operating system has its own conventions on where config files should be stored. Following the conventions is not always straight forward. After a question raised here on Python users about config file locations I was inspired to write this module to help with that problem. config_path library works out which path to use for configuration folders and files in an operating system independent way. Each operating system has particular conventions for where an application is expected to stores it configuration. The information provided to ConfigPath is used to figure out an appropriate file path or folder path for the application's configuration data. Supports Windows, macOS and most unix systems using the 'XDG Base Directory Specification'. Example for the "widget" app from "example.com" that uses JSON for its config: from config_path import ConfigPath conf_path = ConfigPath( 'example.com', 'widget', '.json' ) path = conf_path.saveFilePath( mkdir=True ) with path.open() as f: f.write( config_data ) And to read the config: path = conf_path.readFilePath() if path is not None: # path exists and config can be read with path.open() as f: config = json.loads( f ) else: config = default_config Barry

2 3

Re: Synchronous and Asynchronous callbacks
by Eko palypse Sept. 30, 2019

Sept. 30, 2019

Am Montag, 30. September 2019 11:46:43 UTC+2 schrieb Barry Scott: > > On 29 Sep 2019, at 21:41, Eko palypse <ekopalypse(a)gmail.com> wrote: > > > > Am Sonntag, 29. September 2019 19:18:32 UTC+2 schrieb Barry Scott: > >>> On 29 Sep 2019, at 14:14, Eko palypse <ekopalypse(a)gmail.com> wrote: > >>> > >>> Unfortunately, I can't make all callbacks synchronous or asynchronous because this has negative effects on the application in one way or another. > >> > >> Surely you can make all callbacks async? > >> You do not have to have them wait, they can complete their work in one call. > >> > >> sync == async-that-does-not-await > >> > >> Barry > > > > Thank you for your answer but I'm sorry I don't get it. > > What do you mean by I don't have to have them wait? > > > > Let me explain my understanding, with some pseudo code, how async callbacks > > works and please keep in mind that I'm a novice when it comes to OO designs > > and basic software design strategies/techniques, so I might be totally > > wrong about what I'm doing. > > When you said async I thought you meant the specific programming model in Python > that uses "async def", "await" etc. You are using threads. > > The problem you are facing is that the UI must not become unresponsive. > Further a background thread cannot call any UI framework functions as the UI > framework can only be called on the main thread. > > You can run fast event handers on the UI thread but slow running handlers > must be run on another thread to allow the UI to be responsive. > > What I do is have the event handler for a long running task take the responsibility > to queue the task onto a background thread. While the task is running it sends status > to the UI thread so that UI elements are updated (progress bars, etc). > > If you are using PyQt5 you might find the code I used for SCM Workbench interesting. > > I can write event handlers that look like this: > > @thread_switcher > def eventHandler( self, ... ): > # starts in the foreground (UI thread) > self.status.update('Working...') > > yield self.switchToBackground > # now on the background thread > self.doSlowTask() > > yield self.switchToForeground > # now in the foreground > self.status.update('Done') > > The code is in: > > https://github.com/barry-scott/scm-workbench/blob/master/Source/Common/wb_b… > > @thread_switcher adds an attribute to the function so the rest of the > code knows this is an event handler that needs to use a background thread. > > When I add an event handler I call through a function (wrapWithThreadSwitcher) > that wraps only functions that have been marked with @thread_switcher. > > The rest of the code is the machinery to handle moving between threads via the generator. > > Barry > > > > > An SYNCHRONOUS callback is like this. Event/notification gets fired and > > notification handler calls all registered methods one after the other. > > Something like > > > > if notification.list_of_methods: > > for method in list_of_methods: > > method() > > > > whereas an ASYNCHRONOUS callback is more like this. Event/notification gets > > fired and notification handler informs another thread that the event has > > been fired and all registered async callback methods should get executed. > > So something like this > > > > class EventThread(threading.Thread): > > def __init__(...): > > super()... > > self.event = threading.Event() > > self.kill = False > > ... > > def run(self): > > while True: > > self.event.wait() > > self.event.clear() > > if not self.kill: > > for method in self.list_of_methods: > > method() > > > > et = EventThread() > > if notification.list_of_methods: > > et.event.set() # run all async methods > > for method in list_of_methods: # run sync methods > > method()) > > > > > > So if there is no synchronous callback for that notification then the > > notification handler would just set the event and return. > > The EventThread would then call one registered callback after the other. > > Right? > > > > Using this approach does sound fine from UI point of view as there is > > minimal impact and UI keeps responsive but from code execution point of view > > it sound much more complicated. What if one async callback is slow or buggy? > > This could lead to unpredictable behavior where as in a synchronous execution you immediately know what is causing it. > > That was the reason I decided not to offer async callbacks until I > > discovered that other parts, like UI modifications aren't working properly. > > And there are notification which must be called in a synchronous way, like > > the modification event. If you want to intercept a modification safely then it can only be done within a synchronous callback. > > > > Does this makes sense to you? Or do I have wrong understanding or, again, an > > wrong assumption how this works? > > > > Thank you > > Eren > > -- > > https://mail.python.org/mailman/listinfo/python-list > > Thank you very much, this sounds like what I'm looking for and I'm sure I find tons of other useful information/techniques which I'm not aware of yet. Thank you Eren

1 0

Re: Synchronous and Asynchronous callbacks
by Eko palypse Sept. 30, 2019

Sept. 30, 2019

Am Sonntag, 29. September 2019 19:18:32 UTC+2 schrieb Barry Scott: > > On 29 Sep 2019, at 14:14, Eko palypse <ekopalypse(a)gmail.com> wrote: > > > > Unfortunately, I can't make all callbacks synchronous or asynchronous because this has negative effects on the application in one way or another. > > Surely you can make all callbacks async? > You do not have to have them wait, they can complete their work in one call. > > sync == async-that-does-not-await > > Barry Thank you for your answer but I'm sorry I don't get it. What do you mean by I don't have to have them wait? Let me explain my understanding, with some pseudo code, how async callbacks works and please keep in mind that I'm a novice when it comes to OO designs and basic software design strategies/techniques, so I might be totally wrong about what I'm doing. An SYNCHRONOUS callback is like this. Event/notification gets fired and notification handler calls all registered methods one after the other. Something like if notification.list_of_methods: for method in list_of_methods: method() whereas an ASYNCHRONOUS callback is more like this. Event/notification gets fired and notification handler informs another thread that the event has been fired and all registered async callback methods should get executed. So something like this class EventThread(threading.Thread): def __init__(...): super()... self.event = threading.Event() self.kill = False ... def run(self): while True: self.event.wait() self.event.clear() if not self.kill: for method in self.list_of_methods: method() et = EventThread() if notification.list_of_methods: et.event.set() # run all async methods for method in list_of_methods: # run sync methods method() So if there is no synchronous callback for that notification then the notification handler would just set the event and return. The EventThread would then call one registered callback after the other. Right? Using this approach does sound fine from UI point of view as there is minimal impact and UI keeps responsive but from code execution point of view it sound much more complicated. What if one async callback is slow or buggy? This could lead to unpredictable behavior where as in a synchronous execution you immediately know what is causing it. That was the reason I decided not to offer async callbacks until I discovered that other parts, like UI modifications aren't working properly. And there are notification which must be called in a synchronous way, like the modification event. If you want to intercept a modification safely then it can only be done within a synchronous callback. Does this makes sense to you? Or do I have wrong understanding or, again, an wrong assumption how this works? Thank you Eren

2 1

Interesting performance question
by Anthony Flury Sept. 29, 2019

Sept. 29, 2019

I have just noticed an oddity : Using python 3.6 building a tuple like this : my_tuple = tuple([x*x for x in range(1,1000)]) is about 1/3 quicker than my_tuple = tuple(x*x for x in range(1,1000)) Measurements : $ python3 -m timeit 'my_tuple = tuple([x*x for x in range(1,1000)])' 10000 loops, best of 3: 33.5 usec per loop $ python3 -m timeit 'my_tuple = tuple(x*x for x in range(1,1000))' 10000 loops, best of 3: 44.1 usec per loop My first assumption was that on the first example (with the list comprehension) the tuple constructor is able allocate the right amount of memory immediately, and therefore doesn't have the cost overhead of having to resize compared to the version having to resize every so often as new items are fetched from the generator. One question though - wouldn't the list generator have a similar resizing issue, since the list comprehension wont predict the final size of the list ahead of time. Is there a good explanation of the performance difference - I don't think that the memory resizing overhead is the whole answer. -- Tony Flury

3 2

Re: Synchronous and Asynchronous callbacks
by Eko palypse Sept. 29, 2019

Sept. 29, 2019

Am Sonntag, 29. September 2019 01:02:48 UTC+2 schrieb Paul Rubin: > Eko palypse <ekopalypse(a)gmail.com> writes: > > Two dicts, one for sync and the other for async registrations? > > Why not make all the callbacks asynchronous? Clients that don't want to > handle a callback immediately can keep it pending until they are ready, > perhaps in a queue. > > Whether this can work of course depends on how the application is put > together, but the pattern of a bunch of asynchronous tasks communicating > through fifos tends to work pretty well. I'm used to doing it with > threads, though those can be expensive if you use a lot of them. I > don't know whether async/await unavoidably complicates things. Thank you very much for your answer. Unfortunately, I can't make all callbacks synchronous or asynchronous because this has negative effects on the application in one way or another. I also came across the asyncio methods yesterday, Whoa, what can I say, I haven't really understood yet. My first impression is that either the whole library works asynchronously or not. Still too many Bohemian villages (not sure if this is also common in English - it should say there are too many things I don't understand). But as I said, thank you very much for the time and your thoughts you have given about my problem, very much appreciated. Eren

2 1