🌐 Mailman 3 January 2010 - Python-list - python.org

Safe file I/O to shared file (or SQLite) from multi-threaded web server
by python＠bdurham.com Jan. 1, 2010

Jan. 1, 2010

I'm looking for the best practice way for a multi-threaded python web server application to read/write to a shared file or a SQLite database. What do I need to do (if anything) to make sure my writes to a regular file on disk or to a SQLite database are atomic in nature when multiple clients post data to my application simultaneously? Do I need to use a Queue type data structure and then run a background thread that monitors my Queue for data which it (and it alone) removes and copies to the destination file or SQLite datatbase? Note: In my specific case, the web server will be based on CherryPy 3.1 but I think this type of question is relevant across other Python based web server frameworks as well. Thank you, Malcolm

1 0

MySQL Error
by Victor Subervi Jan. 1, 2010

Jan. 1, 2010

Hi; I'm trying to avoid the mortal sin of blank excepts. I intentionally threw this error: Traceback (most recent call last): File "/var/www/html/angrynates.com/cart/createAssociations2.py", line 137, in ? createAssociations2() File "/var/www/html/angrynates.com/cart/createAssociations2.py", line 108, in createAssociations2 cursor.execute(sql) File "/usr/lib64/python2.4/site-packages/MySQLdb/cursors.py", line 163, in execute self.errorhandler(self, exc, value) File "/usr/lib64/python2.4/site-packages/MySQLdb/connections.py", line 35, in defaulterrorhandler raise errorclass, errorvalue ProgrammingError: (1146, "Table 'test.productsAssociations' doesn't exist") However, "ProgrammingError" is not an error. How do I discover the real error, so I can write the appropriate except statement? TIA, beno -- The Logos has come to bear http://logos.13gems.com/

4 5

Re: twenty years ago Guido created Python
by Steve Howell Jan. 1, 2010

Jan. 1, 2010

On Jan 1, 9:03 am, MRAB <pyt...(a)mrabarnett.plus.com> wrote: > J Peyret wrote: > > On Dec 31 2009, 2:06 pm, Steve Howell <showel...(a)yahoo.com> wrote: > >> FYI: > > >>http://twitter.com/gvanrossum > > >> Python is a truly awesome programming language. Not only is Guido a > >> genius language designer, but he is also a great project leader. What > >> an accomplishment. Congratulations to everybody who has contributed > >> to Python in the last two decades! > > > Notwithstanding all of the above, which are all true, having met > > Guido, I say he is a genuinely nice human being. > > And anyone who says he isn't will be put into the comfy chair! :-) And poked with the soft cushions!

1 0

Re: Exception as the primary error handling mechanism?
by Steven D'Aprano Jan. 1, 2010

Jan. 1, 2010

On Fri, 01 Jan 2010 11:02:28 -0500, Benjamin Kaplan wrote: > I was trying to point out that in > Python, you don't test errors for your typical conditions, but for ones > that you know still exist but don't plan on occurring often. I'm sorry, but that makes no sense to me at all. I don't understand what you are trying to say. You do understand that exceptions aren't just for errors? They are raised under specific circumstances. Whether that circumstance is an error or not is entirely up to the caller. try: n = mylist.index('fault') except ValueError: print "All is good, no fault detected" else: print "Your data contains a fault in position", n People get hung up on the idea that exceptions == errors, but they aren't. They *may* be errors, and that is one common use, but that depends entirely on the caller. -- Steven

2 1

Re: Exception as the primary error handling mechanism?
by aahz＠pythoncraft.com Jan. 1, 2010

Jan. 1, 2010

In article <mailman.300.1262323578.28905.python-list(a)python.org>, Benjamin Kaplan <benjamin.kaplan(a)case.edu> wrote: > >In Python, throwing exceptions for expected outcomes is considered >very bad form [...] Who says that? I certainly don't. -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.

4 3

Re: Exception as the primary error handling mechanism?
by Steven D'Aprano Jan. 1, 2010

Jan. 1, 2010

On Fri, 01 Jan 2010 00:26:09 -0500, Benjamin Kaplan wrote: > On Thu, Dec 31, 2009 at 11:47 PM, Peng Yu <pengyu.ut(a)gmail.com> wrote: >> I observe that python library primarily use exception for error >> handling rather than use error code. >> >> In the article API Design Matters by Michi Henning >> >> Communications of the ACM >> Vol. 52 No. 5, Pages 46-56 >> 10.1145/1506409.1506424 >> http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext >> >> It says "Another popular design flaw—namely, throwing exceptions for >> expected outcomes—also causes inefficiencies because catching and >> handling exceptions is almost always slower than testing a return >> value." >> >> My observation is contradicted to the above statement by Henning. If my >> observation is wrong, please just ignore my question below. >> >> Otherwise, could some python expert explain to me why exception is >> widely used for error handling in python? Is it because the efficiency >> is not the primary goal of python? >> -- >> http://mail.python.org/mailman/listinfo/python-list >> >> > Read the quote again "Another popular design flaw—namely, throwing > exceptions *for expected outcomes*" > In Python, throwing exceptions for expected outcomes is considered very > bad form (well, except for StopIteration but that should almost never be > handled directly by the programmer). Exceptions are *exceptional*, not "errors" or "unexpected". They are exceptional because they aren't the "normal" case, but that doesn't mean they are surprising or unexpected. Are you surprised that your "for x in range(1000)" loop comes to an end? Of course you are not -- it is completely expected, even though less than 1% of the iterations are the last loop. The end of the sequence is EXCEPTIONAL but not UNEXPECTED. If you program without expecting that keys can sometimes be missing from dictionaries (KeyError), or that you might sometimes have to deal with a list index that is out of range (IndexError), or that the denominator in a division might be zero (ZeroDivisionError), then you must be writing really buggy code. None of these things are unexpected, but they are all exceptional. The urllib2 module defines a HTTPError class, which does double-duty as both an exception and a valid HTTP response. If you're doing any HTTP programming, you better expect to deal with HTTP 301, 302 etc. codes, or at least trust that the library you use will transparently handle them for you. > To answer why people recommend using "Easier to Ask Forgiveness than > Permission" as opposed to "Look Before You Leap" : Because of the way > it's implemented, Python works quite differently from most languages. An > attribute look-up is rather expensive because it's a hash table look-up > at run time. Wrapping a small piece of code in a try block however, > isn't (relatively) very expensive at all in Python. It's not just relatively inexpensive, it's absolutely inexpensive: it costs about as much as a pass statement in CPython, which is pretty much as cheap as it gets. (If anyone can demonstrate a cheaper operation available from pure Python, I'd love to see it.) > It's only catching the exception that's expensive, True. > but if you're catching the exception, > something has gone wrong anyway and performance isn't your biggest > issue. The second try...except clause in the urllib2 module reads: try: kind = int(kind) except ValueError: pass In this case, the error is simply ignored. Nothing has gone wrong. Here's an example from my own code: I have an API where you pass a mapping (either a dict or a list of (key, value) tuples) to a function. So I do this: try: it = obj.iteritems() except AttributeError: it = obj for key, value in it: do_stuff() There's nothing wrong with catching exceptions. -- Steven

2 1

change an exception's message and re-raise it
by Phlip Jan. 1, 2010

Jan. 1, 2010

Pythonistas: I need to do this: try: deep_arcane_layer() except e: e.message = 'the deep arcane layer says: ' + e.message raise e The point is I need to augment that layer's exceptions with extra information that I know about that layer. I naturally cannot use the argless version of 'raise', because it only re-raises whatever exception object is currently in play - and it appears to be read-only or locked or something. I also should not do this... raise Exception('blah ' + e.message) ...because that stripped off the exception type itself, and higher layers need to know this. My question is a common pattern in layered architectures, where exceptions get decorated with extra info as they bubble up from the engine room to the bridge. Any ideas? -- Phlip http://zeekland.zeroplayer.com/The_Elaborate_Art_of_Play_Part_1/1

5 6

Re: Dangerous behavior of list(generator)
by Steven D'Aprano Jan. 1, 2010

Jan. 1, 2010

On Thu, 31 Dec 2009 11:34:39 -0800, Tom Machinski wrote: > On Wed, Dec 30, 2009 at 4:01 PM, Steven D'Aprano > <steve(a)remove-this-cybersource.com.au> wrote: >> On Wed, 30 Dec 2009 15:18:11 -0800, Tom Machinski wrote: >>> Bottom line, I'm going to have to remove this pattern from my code: >>> >>> foo = (foo for foo in foos if foo.bar).next() >> >> I don't see why. What's wrong with it? Unless you embed it in a call to >> list, or similar, it will explicitly raise StopIteration as expected. > > Exactly; this seems innocuous, but if some caller of this code uses it > in a list() constructor, a very subtle and dangerous bug is introduced - > see OP. This is the entire point of this post. Then don't use it in a list() constructor. That's a glib answer, of course. A better answer is to point out that the problem is not with the above expression, but with letting StopIteration bubble up as an error exception instead of dealing with it immediately. That's not what it's for, and you can't trust it not to be captured by something. If StopIteration represents an error condition, you need to deal with it immediately and convert it to an exception which isn't likely to disappear. > In a large, non-trivial application, you simply cannot afford the > assumption that no caller will ever do that. Even if you have perfect > memory, some of your other developers or library users may not. You shouldn't put the responsibility of dealing with the StopIteration on the caller, because StopIteraction is a signal not an error condition, and you can't tell when that signal will disappear. The responsibility lies on the writer of the function containing the line (that is, the Original Poster of this thread). So you need something like this: def my_function(): try: foo = (foo for foo in foos if foo.bar).next() except StopIteration: handle_empty_foos() else: handle_at_least_one_foo() handle_empty_foos may be as simple as raising a new exception and letting that bubble up to whatever layer of the application is expected to deal with it. > As for what's wrong with the "if not any" solution, Benjamin Kaplan's > post hits the nail on its head. This is a bioinformatics application, so > the iterable "foos" tends to be very large, so saving half the runtime > makes a big difference. Possibly you haven't seen my reply to Benjamin, so I'll paraphrase: that's incorrect, because any() is lazy and will return as soon as it hits a non-false item. See the docs: http://docs.python.org/library/functions.html#any If the foo items are considered true (e.g. non-empty strings), then you can guarantee that any() will return on the very first item. If the foo items are arbitrary objects which have an equal chance of being considered true or false, then on average it will have to look at half the list, which is O(N) and may be a tad expensive for large N. But how likely is that? One has to be realistic here, and consider the type of data you realistically need to deal with and not pathological cases. There's no limit to the problems you may have with sufficiently pathological data: class Evil(object): @property def bar(self): import time time.sleep(1e8) return True foos = [Evil(), "a", "b", "c", "d"] foo = (foo for foo in foos if foo.bar).next() any() is the standard, idiomatic solution for solving this sort of problem. Before rejecting it on the basis of slowness, you need to determine that long runs of false items ahead of the first true item is a realistic scenario, and that calling any() really is a bottleneck. Anything less is premature optimization. -- Steven

2 2

Re: Where is "urllib2" module in windows python3.1.1?
by Lie Ryan Jan. 1, 2010

Jan. 1, 2010

On 1/2/2010 12:46 AM, Hidekazu IWAKI wrote: > Hi; > I'd like to import "urllib2" in windows python 3.1.1, but I'm not able to > do it. > So, I researched the library directory; the result is following: > > ..... > ..... > 2009/06/07 19:11 61,749 unittest.py > 2010/01/01 22:18<DIR> urllib > 2007/12/06 09:48 6,318 uu.py > 2008/11/30 20:40 21,534 uuid.py > ..... > ..... > > Although it has "urllib", it doesn't have "urllib2". I guess to download > and install the module. But I can't find "urllib2" module for windows > with google. > > I wonder if it would be possible for me to give the location? urllib2 is merged into urllib in python3.

1 0

Where is "urllib2" module in windows python3.1.1?
by Hidekazu IWAKI Jan. 1, 2010

Jan. 1, 2010

Hi; I'd like to import "urllib2" in windows python 3.1.1, but I'm not able to do it. So, I researched the library directory; the result is following: .... .... 2009/06/07 19:11 61,749 unittest.py 2010/01/01 22:18 <DIR> urllib 2007/12/06 09:48 6,318 uu.py 2008/11/30 20:40 21,534 uuid.py .... .... Although it has "urllib", it doesn't have "urllib2". I guess to download and install the module. But I can't find "urllib2" module for windows with google. I wonder if it would be possible for me to give the location?

1 0