🌐 Mailman 3 December 2009 - Python-list - python.org

SimpleHTTPServer, shoutdown and blocked request handlers
by Miki Dec. 31, 2009

Dec. 31, 2009

Hello All, I have an instance of SimpleHTTPServer, however when I try to call "shutdown" on it and there is a request handler that is blocked - the whole process will block. It does so even if I run the "serve_forever" method in a deamon thread. See example code at http://codepad.org/cn8EYdfg Any advice? Thanks, -- Miki

1 0

Re: How to test a URL request in a "while True" loop
by Brian D Dec. 31, 2009

Dec. 31, 2009

On Dec 30, 7:08 pm, MRAB <pyt...(a)mrabarnett.plus.com> wrote: > Brian D wrote: > > Thanks MRAB as well. I've printed all of the replies to retain with my > > pile of essential documentation. > > > To follow up with a complete response, I'm ripping out of my mechanize > > module the essential components of the solution I got to work. > > > The main body of the code passes a URL to the scrape_records function. > > The function attempts to open the URL five times. > > > If the URL is opened, a values dictionary is populated and returned to > > the calling statement. If the URL cannot be opened, a fatal error is > > printed and the module terminates. There's a little sleep call in the > > function to leave time for any errant connection problem to resolve > > itself. > > > Thanks to all for your replies. I hope this helps someone else: > > > import urllib2, time > > from mechanize import Browser > > > def scrape_records(url): > > maxattempts = 5 > > br = Browser() > > user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: > > 1.9.0.16) Gecko/2009120208 Firefox/3.0.16 (.NET CLR 3.5.30729)' > > br.addheaders = [('User-agent', user_agent)] > > for count in xrange(maxattempts): > > try: > > print url, count > > br.open(url) > > break > > except urllib2.URLError: > > print 'URL error', count > > # Pretend a failed connection was fixed > > if count == 2: > > url = 'http://www.google.com' > > time.sleep(1) > > pass > > 'pass' isn't necessary. > > > else: > > print 'Fatal URL error. Process terminated.' > > return None > > # Scrape page and populate valuesDict > > valuesDict = {} > > return valuesDict > > > url = 'http://badurl' > > valuesDict = scrape_records(url) > > if valuesDict == None: > > When checking whether or not something is a singleton, such as None, use > "is" or "is not" instead of "==" or "!=". > > > print 'Failed to retrieve valuesDict' > > I'm definitely acquiring some well-deserved schooling -- and it's really appreciated. I'd seen the "is/is not" preference before, but it just didn't stick. I see now that "pass" is redundant -- thanks for catching that. Cheers.

2 1

Re: How to test a URL request in a "while True" loop
by Brian D Dec. 31, 2009

Dec. 31, 2009

On Dec 30, 12:31 pm, Philip Semanchuk <phi...(a)semanchuk.com> wrote: > On Dec 30, 2009, at 11:00 AM, Brian D wrote: > > > > > I'm actually using mechanize, but that's too complicated for testing > > purposes. Instead, I've simulated in a urllib2 sample below an attempt > > to test for a valid URL request. > > > I'm attempting to craft a loop that will trap failed attempts to > > request a URL (in cases where the connection intermittently fails), > > and repeat the URL request a few times, stopping after the Nth attempt > > is tried. > > > Specifically, in the example below, a bad URL is requested for the > > first and second iterations. On the third iteration, a valid URL will > > be requested. The valid URL will be requested until the 5th iteration, > > when a break statement is reached to stop the loop. The 5th iteration > > also restores the values to their original state for ease of repeat > > execution. > > > What I don't understand is how to test for a valid URL request, and > > then jump out of the "while True" loop to proceed to another line of > > code below the loop. There's probably faulty logic in this approach. I > > imagine I should wrap the URL request in a function, and perhaps store > > the response as a global variable. > > > This is really more of a basic Python logic question than it is a > > urllib2 question. > > Hi Brian, > While I don't fully understand what you're trying to accomplish by > changing the URL to google.com after 3 iterations, I suspect that some > of your trouble comes from using "while True". Your code would be > clearer if the while clause actually stated the exit condition. Here's > a suggestion (untested): > > MAX_ATTEMPTS = 5 > > count = 0 > while count <= MAX_ATTEMPTS: > count += 1 > try: > print 'attempt ' + str(count) > request = urllib2.Request(url, None, headers) > response = urllib2.urlopen(request) > if response: > print 'True response.' > except URLError: > print 'fail ' + str(count) > > You could also save the results (untested): > > MAX_ATTEMPTS = 5 > > count = 0 > results = [ ] > while count <= MAX_ATTEMPTS: > count += 1 > try: > print 'attempt ' + str(count) > request = urllib2.Request(url, None, headers) > f = urllib2.urlopen(request) > # Note that here I ignore the doc that says "None may be > # returned if no handler handles the request". Caveat emptor. > results.append(f.info()) > f.close() > except URLError: > # Even better, append actual reasons for the failure. > results.append(False) > > for result in results: > print result > > I guess if you're going to do the same number of attempts each time, a > for loop would be more expressive, but you probably get the idea. > > Hope this helps > Philip Nice to have options, Philip. Thanks! I'll give your solution a try in mechanize as well. I really can't thank you enough for contributing to helping me solve this issue. I love Python.

2 2

Re: class version of func_globals?
by samwyse Dec. 30, 2009

Dec. 30, 2009

On Dec 29, 5:18 am, Dave Angel <da...(a)ieee.org> wrote: > samwyse wrote: > > Is there any way to get the global namespace of the module in which a > > class was defined? Answers for both Python 2.x and 3.x will be > > cheerfully accepted. > > I don't know if it's the same in general, but consider the following > sequence in 2.6: > > import sys > > class MyClass(object): > pass > > print "class--", dir(MyClass) > print "module--", dir(MyClass.__module__) > mod = sys.modules[MyClass.__module__] > print mod > print "globals--", dir(mod) > > DaveA Excellent! Exactly what I wanted, but wasn't clever enough to figure out for myself. Thank you very much.

2 1

Re: Dangerous behavior of list(generator)
by Albert van der Horst Dec. 30, 2009

Dec. 30, 2009

In article <mailman.1818.1260694428.2873.python-list(a)python.org>, Gabriel Genellina <gagsl-py2(a)yahoo.com.ar> wrote: <SNIP> > >Despite a promise in PEP 289, generator expressions semantics isn't >explained in detail in the language reference. I can't tell if the >difference is intentional, accidental, undocumented behavior, an >implementation accident, a bug, or what... Philosophically speaking ... An important feature that is not documented is a severe defect. (important maps to severe). Before it is documented, there can be no discrepancy between specification and implementation so other defects are formally not present in relation to this situation. >-- >Gabriel Genellina > Groetjes Albert. -- -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert(a)spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

3 2

Re: Thanks for the help not given :)
by aahz＠pythoncraft.com Dec. 30, 2009

Dec. 30, 2009

In article <mailman.189.1262141677.28905.python-list(a)python.org>, J <dreadpiratejeff(a)gmail.com> wrote: >> Jon Clements wrote: >> >>> You have a bear that likes a Python? The one I have just keeps going >>> on about Piglet and eating my honey reserves... > >As for Jon, and Aahz, I'd try the teddy bear approach, but the last >one I knew led me down the dark path to Perl and thus I try to avoid >them whenever possible ;-) Don't worry, you have now been inoculated. -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.

1 0

The infamous "distutils has already been patched by" error
by David Dec. 30, 2009

Dec. 30, 2009

I am writing these notes because there is little or no documentation about this annoying problem. The error message in subject appears when you try to get the list of installed modules under the help() enviroment. >>> help() help> modules Please wait a moment while I gather a list of all available modules... ... AssertionError: distutils has already been patched by <class py2exe.Distribution at 0x019132D0> It is caused by monkeypatching of distutils module by other modules, tipically py2exe. In detail the cause is this: 0) The help() function is defined by the \python25\lib\site.py module automatically imported at interpreter startup 1) the standard module \python25\lib\site.py imports the module pydoc and call the pydoc.help() function (in the class _Helper). 2) The method run() of class ModuleScanner, used by pydoc.help() walks all installed modules using the pkgutil.walk_packages() function from module pkgutil, and calls it giving no arguments so the 'onerror' parameter takes the default value None. 3) The function pkgutil.walk_packages() tries to import each package in a try/except clause like this: try: __import__(name) except ImportError: if onerror is not None: onerror(name) except Exception: if onerror is not None: onerror(name) else: raise As you can see, if onerror is None, the function captures only the ImportError exception, but... 4) ...when pkgutil.walk_packages() tries to import dist.py module of setuptools package (\python25\lib\site-packages\setuptools\dist.py) the following piece of code gets executed... def _get_unpatched(cls): """Protect against re-patching the distutils if reloaded Also ensures that no other distutils extension monkeypatched the distutils first. """ while cls.__module__.startswith('setuptools'): cls, = cls.__bases__ if not cls.__module__.startswith('distutils'): raise AssertionError( "distutils has already been patched by %r" % cls ) return cls _Distribution = _get_unpatched(_Distribution) ...and, if the py2exe (or other modules) had already been imported and monkeypatched del setuptools module, the infamous "distutils has already been patched by" is raised, caputerd by pkgutil.walk_packages() and reraised because the onerror variable is None. That said, there are 2 workarounds to avoid this hassle: A) manually import setuptools module before typing help() >>> import setuptools >>> help() Welcome to Python 2.5! This is the.... help> modules B) Patch the run() method of pydoc module at line 1862: for importer, modname, ispkg in pkgutil.walk_packages(onerror=lambda s:None): Hope this helps David.

2 1

Re: the need for 64 bits
by aahz＠pythoncraft.com Dec. 30, 2009

Dec. 30, 2009

In article <mailman.194.1262153028.28905.python-list(a)python.org>, Cameron Simpson <cs(a)zip.com.au> wrote: >On 29Dec2009 21:42, Aahz <aahz(a)pythoncraft.com> wrote: >| In article <8950e4a5-f630-4ffb-b7ed-5c539913a7d7(a)a6g2000yqm.googlegroups.com>, >| Mensanator <mensanator(a)aol.com> wrote: >| >Ah, the 8 GB Mac wouldn't have helped. Glad I didn't spend the extra >| >$1000. >| >| It's almost always cheaper to buy your Mac and then upgrade the RAM >| separately. > >Unless it's a Macbook Air. Then you're stuck with 2GB max:-( No VM for you! I'm already unhappy with VM performance on my 4GB machine; I can't wait for the weekend to install 8GB instead. -- Aahz (aahz(a)pythoncraft.com) <*> http://www.pythoncraft.com/ Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.

1 0

Re: detect interactivity
by Steven D'Aprano Dec. 30, 2009

Dec. 30, 2009

On Wed, 30 Dec 2009 14:42:12 +0100, Vito 'ZeD' De Tullio wrote: > Steven D'Aprano wrote: > > >>> if I run 'python <<< "import sys; print(sys.ps1)"', I get an error. >> >> Of course you do, because you're not running interactively > > Excuse me, why do you say that? Two reasons. First, because sys.ps1 is documented as only existing in an interactive environment, and sure enough, sys.ps1 fails to exist when you run the above. And secondly, and much more importantly, if you run that command, you do not get an interactive prompt. The format of the command being run is a bash "here string", more-or-less equivalent to either of: echo "import sys; print(sys.ps1)" | python python -c "import sys; print(sys.ps1)" both of which also exit with AttributeError. If you really want to see what's going on, have a look at this: [steve@sylar ~]$ python -i -c "import sys; print(sys.ps1)" Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: 'module' object has no attribute 'ps1' >>> sys.ps1 '>>> ' The -c switch tells Python to execute whatever comes next, and the -i switch tells Python to enter interactive mode afterwards. At first, sys.ps1 does not exist and you get an AttributeError, but then the -i switch kicks in, sys.ps1 is defined and you get a prompt. -- Steven

1 0

Python 2.6 ftplib has timeout parameter, but how to detect a timeout
by Brendan Dec. 30, 2009

Dec. 30, 2009

I was quite happy to see that ftplib in Python 2.6 now has a timeout parameter. With large file downloads my script would often hang, presumably from timing out. Now that there is a timeout parameter, how would I detect when a timeout occurs?

4 4