🌐 Mailman 3 February 2017 - Python-list - python.org

user+sys time bigger than real time (with threads)
by Marco Buttu Feb. 2, 2017

Feb. 2, 2017

Let's have the following example: $ cat foo.py from threading import Thread, Lock result = 0 lock = Lock() def task(): global result for i in range(10**6): lock.acquire() result += 1 lock.release() if __name__ == '__main__': t1, t2 = Thread(target=task), Thread(target=task) t1.start() t2.start() t1.join() t2.join() print('result:', result) When I execute it (Python 3.6), I get a sys+user time bigger than the real time: $ time python foo.py result: 2000000 real 0m7.088s user 0m6.597s sys 0m5.043s I can expect this result when I run some processes in parallel on different CPUs, but this code uses threads, so the GIL prevents the two task() functions to be executed in parallel. What am I missing? -- Marco Buttu INAF-Osservatorio Astronomico di Cagliari Via della Scienza n. 5, 09047 Selargius (CA) Phone: 070 711 80 217 Email: mbuttu(a)oa-cagliari.inaf.it

1 0

how i can write command to run...........This is code to convert .json file to .csv......Input file name is sample.json
by muetced124＠gmail.com Feb. 2, 2017

Feb. 2, 2017

import sys import json import csv ## # Convert to string keeping encoding in mind... ## def to_string(s): try: return str(s) except: #Change the encoding type if needed return s.encode('utf-8') ## # This function converts an item like # { # "item_1":"value_11", # "item_2":"value_12", # "item_3":"value_13", # "item_4":["sub_value_14", "sub_value_15"], # "item_5":{ # "sub_item_1":"sub_item_value_11", # "sub_item_2":["sub_item_value_12", "sub_item_value_13"] # } # } # To # { # "node_item_1":"value_11", # "node_item_2":"value_12", # "node_item_3":"value_13", # "node_item_4_0":"sub_value_14", # "node_item_4_1":"sub_value_15", # "node_item_5_sub_item_1":"sub_item_value_11", # "node_item_5_sub_item_2_0":"sub_item_value_12", # "node_item_5_sub_item_2_0":"sub_item_value_13" # } ## def reduce_item(key, value): global reduced_item #Reduction Condition 1 if type(value) is list: i=0 for sub_item in value: reduce_item(key+'_'+to_string(i), sub_item) i=i+1 #Reduction Condition 2 elif type(value) is dict: sub_keys = value.keys() for sub_key in sub_keys: reduce_item(key+'_'+to_string(sub_key), value[sub_key]) #Base Condition else: reduced_item[to_string(key)] = to_string(value) if __name__ == "__main__": if len(sys.argv) != 4: print "\nUsage: python json_to_csv.py <node_name> <json_in_file_path> <csv_out_file_path>\n" else: #Reading arguments node = sys.argv[1] json_file_path = sys.argv[2] csv_file_path = sys.argv[3] fp = open(json_file_path, 'r') json_value = fp.read() raw_data = json.loads(json_value) try: data_to_be_processed = raw_data[node] except: data_to_be_processed = raw_data processed_data = [] header = [] for item in data_to_be_processed: reduced_item = {} reduce_item(node, item) header += reduced_item.keys() processed_data.append(reduced_item) header = list(set(header)) header.sort() with open(csv_file_path, 'wb+') as f: writer = csv.DictWriter(f, header, quoting=csv.QUOTE_ALL) writer.writeheader() for row in processed_data: writer.writerow(row) print "Just completed writing csv file with %d columns" % len(header)

1 0

Why doesn't module finalization delete names as expected?
by Philippe Proulx Feb. 2, 2017

Feb. 2, 2017

I'm developing a C module with the help of SWIG. My library manages objects with reference counting, much like Python, except that it's deterministic: there's no GC. I'm using Python 3.5.2. I create two Python objects like this: bakery = Bakery() bread = bakery.create_bread() Behind the scenes, the situation looks like this: +--------------------+ | UserBread obj (Py) | +----------^---+-----+ | : | : | : +------------------+ +---------+---V---+ | Bakery obj (lib) <----------------+ Bread obj (lib) | +--------^---+-----+ +--------^--------+ | : | | : | +--------+---V----+ +--------+-------+ | Bakery obj (Py) | | Bread obj (Py) | +---------^-------+ +------^---------+ | | | | + + bakery bread A pipe link means one "strong" reference and a colon link means one borrowed/weak reference. I have some ownership inversion magic for the Bakery lib. and Python objects to always coexist. So here it's pretty clear what can happen. I don't know which reference gets deleted first, but let's assume it's `bakery`. Then the situation looks like this: +--------------------+ | UserBread obj (Py) | +----------^---+-----+ | : | : | : +------------------+ +---------+---V---+ | Bakery obj (lib) <----------------+ Bread obj (lib) | +--------^---+-----+ +--------^--------+ : | | : | | +--------+---V----+ +--------+-------+ | Bakery obj (Py) | | Bread obj (Py) | +-----------------+ +------^---------+ | | + bread The Bakery Python object's __del__() drops the reference to its library object, but first its reference count is incremented during this call (so it's not really destroyed) and it's marked as only owned by the library object from now on. When `bread` gets deleted: 1. The Bread Python object's __del__() method gets called: the reference to its library object is dropped. 2. The Bread library object's destroy function drops its reference to the Bakery library object. 3. The Bakery library object's destroy function drops its reference to the Bakery Python object. 4. The Bakery Python object's __del__() method does nothing this time, since the object is marked as only owned by its library object (inverted ownership). 5. The Bread library object's destroy function then drops its reference to the UserBread Python object. In the end, everything is correctly destroyed and released. This also works if `bread` is deleted before `bakery`. My problem is that this works as expected when used like this: def func(): bakery = Bakery() bread = bakery.create_bread() if __name__ == '__main__': func() but NOTHING is destroyed when used like this: bakery = Bakery() bread = bakery.create_bread() That is, directly during the module's initialization. It works, however, if I delete `bread` manually: bakery = Bakery() bread = bakery.create_bread() del bread It also works with `bakery` only: bakery = Bakery() My question is: what could explain this? My guess is that my logic is correct since it works fine in the function call situation. It feels like `bread` is never deleted in the module initialization situation, but I don't know why: the only reference to the Bread Python object is this `bread` name in the module... what could prevent this object's __del__() method to be called? It works when I call `del bread` manually: I would expect that the module finalization does the exact same thing? Am I missing anything? Thanks, Phil

2 1

Is there a lint tool that can spot unused classes/methods/attributes/functions/etc?
by Mark Summerfield Feb. 2, 2017

Feb. 2, 2017

Suppose I have a project with an application .py file and various module .py files all in the same directory, and after lots of refactoring and other changes the modules contain lots of unused stuff. Is there a lint tool that will spot the unused things so that I can get rid of them? I've tried flake and pylint but neither seem to help. Here's the smallest example I can think of: # app.py import Mod a = Mod.A() print(a.quack()) # Mod.py class A: # Used in app.py def __init__(self): self.x = "hello" # Used in quack() which is used in app.py self.y = 7 # Unused @property def z(self): # Unused return 4.5 def quack(self): # Used in app.py return self.x class B: # Unused pass CONST = 999 # Unused def func(): # Unused pass Thanks.

2 1

Re: How to know what to install (Ubuntu/Debian) for a given import?
by Chris Green Feb. 2, 2017

Feb. 2, 2017

Michael Torrie <torriem(a)gmail.com> wrote: > On 02/01/2017 01:03 PM, Wildman via Python-list wrote: > > > > It is the proper way. This page helps explain it. > > > > http://askubuntu.com/questions/784068/what-is-gi-repository-in-python > > > >> ... and doesn't it need an internet connection? > > > > No. > > However the gi module provides access to GTK+3, and it's quite likely > Chris's project requires GTK+2, and would probably take some work to > port from GTK+2 to GTK+3, though in the long run it's worth the effort. > > Anyway the GTK+2 module for python is usually called pygtk in the repos. > It would appear that there isn't a GTK+2 for Python 3 in Ubuntu, or at least I can't find one. I've reverted to staying with Python 2 fr the moment. I can't manage changing the code to work with a database instead of LDAP *and* migrating from GTK+2 to GTK+3. :-) When I have it working as I want in python 2 I'll consider the GTK+2 to GTK+3 migration and moving to python 3. -- Chris Green ·

1 0

Re: How to know what to install (Ubuntu/Debian) for a given import?
by Chris Green Feb. 2, 2017

Feb. 2, 2017

Michael Torrie <torriem(a)gmail.com> wrote: > On 02/01/2017 02:29 PM, Chris Green wrote: > > OK, thank you, what a strange way to do it. > > Why is it strange? Essentially, python bindings for any GObject-based > library are now fully automatic via this gi module. No longer do we Which is thus different from every other python library! So we have to somehow 'know' that the components of GObject are imported by this special mechanism. Everything else works the old/normal way. > > > OK, no problem, but isn't it very non-portable? > > Not inherently, no. Should work on OS X and Windows. > OK, good, I thought from the original reply that it was Ubuntu specific, but it's not. -- Chris Green ·

1 0

Re: command line micro wiki written in Python
by paul.wolf＠yewleaf.com Feb. 2, 2017

Feb. 2, 2017

On Tuesday, 31 January 2017 23:39:41 UTC, Ben Finney wrote: > The Python community has a stronger (?) preference for reStructuredText > format. Can that be the default? > > That is, I want my text files to be named ‘foo’ (no suffix) or ‘foo.txt’ > (because they're primarily text), and have the default be to parse them > as reStructuredText. Good point. It should at least be possible to provide an arbitrary default so you can have rst as default for new files. And perhaps as you say also start with rst as the main default. Regarding no suffix, whatever the default is gets used for newly created files, but not for the 'take' command which I use mainly to link to (with --symlink) config files, like: yd take ~/.emacs for instance.

1 0

Re: How to know what to install (Ubuntu/Debian) for a given import?
by Rustom Mody Feb. 1, 2017

Feb. 1, 2017

On Thursday, February 2, 2017 at 5:44:51 AM UTC+5:30, Erik wrote: > On 01/02/17 23:20, Wildman via Python-list wrote: > > On Wed, 01 Feb 2017 21:29:00 +0000, Chris Green wrote: > > > >> Wildman wrote: > >>> On Wed, 01 Feb 2017 19:15:13 +0000, Chris Green wrote: > >> OK, no problem, but isn't it very non-portable? > > > > I don't see why not. It should work on any system > > that has Python3 installed, at least that is my > > understanding. I'm sure someone will correct me > > if I'm wrong. > > > > OTOH if you want in insure 100% portability with any > > script, you can use pyinstaller. > > > > To install for Python2: > > pip install pyinstaller > > > > For Python3: > > > > pip3 install pyinstaller > > Out of interest (as someone who grew up on the great 1.5.7 ;)) - is > there a definitive resource that explains all of the various packaging > and installation options that exist for Python modules these days (both > for an author and a user)? > > A lot of Linux distributions have Python-related packages (other than > the language itself) which can be installed using the system installer. > > Then there's "pip", which is an installer which is installed using the > system installer. > > Now, apparently, there's "pyinstaller" which can be installed using the > "pip" installer! > > I'd like to understand the differences and how this all fits together. +10 From me for this question My impression is that this question is signally badly addressed because it falls between OS-language stools: ie in the larger Linux-Python ecosystem some things are naturally addressed as Linux docs, some as python docs. This (and such issues) seems to be a buck that loves to be passed around

1 0

Re: How coding in Python is bad for you
by Steve D'Aprano Feb. 1, 2017

Feb. 1, 2017

On Mon, 30 Jan 2017 10:52 am, Erik wrote: > On 29/01/17 14:42, Steve D'Aprano wrote: >> 1. for...else is misspelled, and should be for...then; >> >> 2. Same for while...else; > > I don't think I'll ever agree with you on this one. > > "then", to me, implies the code following it is always executed. > "else" implies it's conditional. > > In those constructs it's conditional and therefore, to me, "else" is a > better reminder of that. But it isn't conditional. When the code reaches the end of the for/while block, the else block is ALWAYS executed. Unconditionally. The only way to skip the else block is to avoid reaching the end of the for/while block: raise, break or return out of the body of the block. > It would be even better if it was "else if not break:" to make the > meaning clearer. break is not the only way to exit the for loop. > I would agree that it would be even better than that if > it was "then if not break:" (apart from needing the new keyword ;)), as > then the conditional aspect is explicit. But it isn't conditional. Your syntax implies that the interpreter keeps some sort of flag did_we_reach_the_end_of_the_loop_without_break or something, and then it checks the state of that flag. There is no such flag. If I remember correctly, that last time this came up here was because somebody asked how they could read the value of that flag, instead of setting their own: for x in seq: ... if not did_we_reach_the_end_of_the_loop_without_break: print("break") They hoped to use the same flag the for-loop used. But if you use the dis module to decompile the CPython byte-code you will see that there is no such flag. Code like this: for i in seq: break else: foo bar compiles to something like this (results may vary according to the version of Python): py> import dis py> code = compile(""" ... for i in seq: ... break ... else: ... foo ... bar ... """, "", "exec") py> dis.dis(code) 2 0 SETUP_LOOP 19 (to 22) 3 LOAD_NAME 0 (seq) 6 GET_ITER >> 7 FOR_ITER 7 (to 17) 10 STORE_NAME 1 (i) 3 13 BREAK_LOOP 14 JUMP_ABSOLUTE 7 >> 17 POP_BLOCK 5 18 LOAD_NAME 2 (foo) 21 POP_TOP 6 >> 22 LOAD_NAME 3 (bar) 25 POP_TOP 26 LOAD_CONST 0 (None) 29 RETURN_VALUE Not a single condition to be seen, anywhere. Its all unconditional jumps. To anticipate a possible objection: it is possible that the FOR_ITER bytecode is implemented with a conditional test, but even so, all that tests for is whether to enter the main body of the for-loop (10 STORE_NAME ...) or jump to (18 LOAD_NAME ...). If you remove the break completely, or replace it with raise or (in a function) return, and you'll see the same structure to the code: for loops run the main body of the loop, then unconditionally run the else block, then unconditionally exit the loop and continue on with the rest of the program. To avoid the else block, you have to jump out of the entire for-loop. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse.

4 5

Re: Is shutil.get_terminal_size useless?
by Steve D'Aprano Feb. 1, 2017

Feb. 1, 2017

On Sun, 29 Jan 2017 04:58 am, Chris Angelico wrote: > On Sun, Jan 29, 2017 at 3:15 AM, Steve D'Aprano > <steve+python(a)pearwood.info> wrote: >> On Sat, 28 Jan 2017 10:50 pm, Chris Angelico wrote: >> >>> On Sat, Jan 28, 2017 at 9:49 PM, Steve D'Aprano >>> <steve+python(a)pearwood.info> wrote: >>>> The terminal size doesn't change just because I'm piping output to >>>> another process. Using the terminal size as a proxy for "being piped" >>>> is sheer insanity. >>> >>> In a sense, there _is no_ terminal size when you're being piped to >>> another process. >> >> In which sense, and why do you think it is relevant? >> >> There clearly is a terminal, because that's where I'm running the code. >> Regardless of whether I pipe it to grep or cat or something else, the >> output from *that* process still ends up in the same terminal that I >> typed the command in. > > No, not automatically. I've written plenty of programs that accept > input via a pipe and don't display any of it. Well of course if your program doesn't produce any output, the output doesn't go to stdout. Because there is none. > Just because someone > types "command1 | command2", you can't assume that the terminal > command2 is outputting to is the same size as the one command1 should > be outputting to. I'm not assuming anything of the sort. Obviously command2 can send output where it likes, including /dev/null, or simply not produce any output. And as I've *repeatedly* acknowledged, there are circumstances where there is no terminal at all. And shutil does the wrong thing then too: it returns an arbitrary size, instead of raising. (Of course we should be able to explicitly pass a default to shutil.get_terminal_size() if we wish.) > >>> etc, etc, etc, etc. It's >>> not a proxy for "being piped" - it's that when your output isn't going >>> to a terminal, asking "what is my terminal size" isn't particularly >>> productive. >> >> Then explain why os.get_terminal_size() returns the correct answer. No answer Chris? You've spent a lot of time telling me that asking for the terminal size doesn't even make sense for the exact circumstances where os.get_terminal_size() not only returns a size, but the *correct* size. And, of course, it also correctly raises an exception in those unusual cases where there truly is no terminal, or it is unknowable. >> The output might not be going to a terminal (not directly at least) but >> the question isn't "what's the size of the terminal that output is going >> to". The question is "what's the size of the terminal that this process >> is running in", and that has an answer regardless of where output is >> piped. > > Processes aren't always running "in" terminals, though. That's my > point. A terminal is not a fundamental feature of a process. And I have acknowledged this, oh, about a million times. But when the processes *are* running in terminals, shutil should return the size of the terminal, not some arbitrary made up size. Otherwise it is useless. >>> Would you expect a cronjob to use the terminal size when you >>> most recently edited crontab? No. >> >> Of course not -- the terminal where you edited crontab is not where the >> process is running. Why would it be the least bit relevant? > > So where *is* that process running? What terminal is it in? There isn't one. > Don't you > see how similar this is to the pipe situation No. os.get_terminal_size() is correctly able to return the terminal size in the pipe situation, even in a *double* pipe situation (you need to look at stderr rather than stdin or stout). I'm not sure if it is able to cope with situations where all three of std[in|out|err] are redirected, but that's okay. I'm a realist: if there are scenarios where the terminal size is unknowable, then so be it. But a simple pipe is not one of those cases. > - sure, there might be a > terminal that the program was invoked from, but it's utterly > irrelevant to how the program actually runs. Except that in the pipe situation, "how the program actually runs" *is* in a terminal. > >> You might well be a completely background process. >> >> And if that background process is running in a terminal? What's your >> point? > > Background processes don't have terminal access. Really? I think you are wrong: [steve@ando ~]$ cat bkg.py from os import get_terminal_size print(get_terminal_size(0)) [steve@ando ~]$ python3.5 bkg.py & [1] 13776 [steve@ando ~]$ os.terminal_size(columns=116, lines=29) [1]+ Done python3.5 bkg.py [steve@ando ~]$ Looks like the background process is perfectly able to return the terminal size, and not an arbitrary number either, it has the correct size. > Whether it's a daemon > or something started as "commandname >/dev/null 2>/dev/null </dev/null > &" from bash, it doesn't have access to a terminal. As I've repeatedly said, if there are situations where the terminal size is genuinely unknowable, then so be it. But shutil fails even when the terminal size *is* knowable. How is this useful? > Please don't assume that every program has a controlling UI. I'm not, and I haven't. > I get > enough of that mentality from the people I support on Windows, where > the fundamental assumption of people seems to be that there's a GUI > for everything. There isn't. And there isn't always a terminal either. > Processes in the middle of pipelines *do not have* terminals. Sometimes they do. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse.

2 1