🌐 Mailman 3 August 2011 - Python-list - python.org

PC locks up with list operations
by Steven D'Aprano Sept. 12, 2011

Sept. 12, 2011

Twice in a couple of weeks, I have locked up my PC by running a Python 2.5 script that tries to create a list that is insanely too big. In the first case, I (stupidly) did something like: mylist = [0]*12345678901234 After leaving the machine for THREE DAYS (!!!) I eventually was able to get to a console and kill the Python process. Amazingly, it never raised MemoryError in that time. The second time was a little less stupid, but not much: mylist = [] for x in itertools.combinations_with_replacement(some_big_list, 20): mylist.append(func(x)) After three hours, the desktop is still locked up. I'm waiting to see what happens in the morning before rebooting. Apart from "Then don't do that!", is there anything I can do to prevent this sort of thing in the future? Like instruct Python not to request more memory than my PC has? I am using Linux desktops; both incidents were with Python 2.5. Do newer versions of Python respond to this sort of situation more gracefully? -- Steven

7 7

Re: PC locks up with list operations
by Steven D'Aprano Sept. 12, 2011

Sept. 12, 2011

Chris Withers wrote: > On 31/08/2011 13:33, Steven D'Aprano wrote: >> I am using Linux desktops; both incidents were with Python 2.5. Do newer >> versions of Python respond to this sort of situation more gracefully? > > Ironically, Windows does better here and dumps you out with a > MemoryError before slowly recovering. > > Linux seems to fair badly when programs use more memory than physically > available. Perhaps there's some per-process thing that can be used to > limit things on Linux? As far as I know, ulimit ("user limit") won't help. It can limit the amount of RAM available to a process, but that just makes the process start using virtual memory more quickly. It can also limit the amount of virtual memory used by the shell, but not of other processes. In other words, Linux will try really, really, really hard to give you the 84 gigabytes you've asked for on a 2 GB system, even if it means DOSing your system for a month. Of course, I would be very happy to learn I'm wrong. -- Steven

4 3

Re: try... except with unknown error types
by Chris Torek Sept. 10, 2011

Sept. 10, 2011

In article <mailman.286.1313956388.27778.python-list(a)python.org>, Terry Reedy <tjreedy(a)udel.edu> wrote: >I would expect that catching socket.error (or even IOError) should catch >all of those. > >"exception socket.error >A subclass of IOError ... Except that, as Steven D'Aprano almost noted elsethread, it isn't (a subclass of IOError -- the note was that it is not a subclass of EnvironmentError). In 2.x anyway: >>> import socket >>> isinstance(socket.error, IOError) False >>> isinstance(socket.error, EnvironmentError) False >>> (I just catch socket.error directly for this case.) (I have also never been sure whether something is going to raise an IOError or an OSError for various OS-related read or write operation failures -- such as exceeding a resource limit, for instance -- so most places that do I/O operations on OS files, I catch both. Still, it sure would be nice to have a static analysis tool that could answer questions about potential exceptions. :-) ) -- In-Real-Life: Chris Torek, Wind River Systems Intel require I note that my opinions are not those of WRS or Intel Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603 email: gmail (figure it out) http://web.torek.net/torek/index.html

4 3

Re: killing a script
by Steven D'Aprano Sept. 9, 2011

Sept. 9, 2011

On Tue, 30 Aug 2011 08:53 am Arnaud Delobelle wrote: [...] >> Yes, but if I am not mistaken, that will require me to put a line or >> two after each os.system call. That's almost like whack-a-mole at the >> code level rather than the Control-C level. OK, not a huge deal for >> one script, but I was hoping for something simpler. I was hoping I >> could put one line at the top of the script and be done with it. > > Write a function! That's what they're for after all :) I'm not sure that this is actually as simple as that, especially using os.system. As I understand it, the scenario is this: The main script looks something like this: for x in whatever: os.system('something.py x') Each time through the loop, a new Python process is started. Each process runs in the foreground, capturing standard input, and so hitting Ctrl-C kills *that* process, not the main script. Unless, by chance, the Ctrl-C happens after the system call returns, but before the next one starts, it is completely invisible to the parent process (the main script). Wrapping os.system in a function does nothing to fix that. Possibly using the subprocess module may help. Otherwise, the only way around this I can think of is to ensure that the 'something.py' script (or scripts!) each return an error code for "User Cancelled": for x in whatever: result = os.system('something.py x') if result == 3: # User Cancelled break But if the 'something.py' scripts are arbitrary scripts, I don't think you have any easy way around it. Perhaps use threads, and have the main script ask the thread to kill the child process? Regardless, this is a hard problem, and it isn't possible to just have some magic switch at the top of your script to make it work. You actually have to do the work yourself. (But of course you can do the work inside a function, and re-use it elsewhere.) -- Steven

3 2

Installing WebDAV server
by Fokke Nauta Sept. 8, 2011

Sept. 8, 2011

Hi all, I am completely new to Python, but I'm confronted with a problem I can't solve. This is my question: I'm running a PC with XP Pro32, which acts as a file server/print server/FTP server and web server. The web server is facilitated by the Aprelium Abyss X2 server, and has Perl and PHP support on http and https. It all works fine. To do some research with some calender systems and to share the Outlook calendar I need a WebDAV server. After googling I found the Python WebDAV server. I installed Python 3.2.1 and extracted the packages PyWebDAV and PyXML. Now I have a working Python app and 2 directories called PyWebDAV-0.9.4.1 and PyXML-0.8.4. In the PyWebDAV README it says: Installation and setup of server can be as easy as follows: $ easy_install PyWebDAV $ davserver -D /tmp -n -J But of course it doesn't work like that. When I start up Python GUI I see the ">>>" prompt instead of the "$" prompt. But where do I place the two directories? And there is no easy_install script in the PyXML-0.8.4 directory, only a setup.py and ez_setup.py script. I guess the latter is the one to use. But how? How do I proceed next? Any help will be appreciated. Thanks in advance. With regards, Fokke Nauta

5 10

Why do class methods always need 'self' as the first parameter?
by T. Goodchild Sept. 6, 2011

Sept. 6, 2011

I’m new to Python, and I love it. The philosophy of the language (and of the community as a whole) is beautiful to me. But one of the things that bugs me is the requirement that all class methods have 'self' as their first parameter. On a gut level, to me this seems to be at odds with Python’s dedication to simplicity. For example, consider Python’s indent-sensitive syntax. Although other languages didn’t use indentation to specify scope, programmers always used indentation anyways. Making indentation took a common practice, made it a rule, and the result was a significantly improved signal-to-noise ratio in the readability of Python code. So why is 'self' necessary on class methods? It seems to me that the most common practice is that class methods *almost always* operate on the instance that called them. It would make more sense to me if this was assumed by default, and for "static" methods (methods that are part of a class, but never associated with a specific instance) to be labelled instead. Just curious about the rationale behind this part of the language.

19 25

Closures and Partial Function Application
by Travis Parks Sept. 5, 2011

Sept. 5, 2011

I was a little disappointed the other day when I realized that closures were read-only. I like to use closures quite a bit. Can someone explain why this limitation exists? Secondly, since I can cheat by wrapping the thing being closure-ified, how can I write a simple wrapper that has all the same members as the thing (decorator), that then applies them to the underlying thing? I also like partial function application. What is the easiest way of achieving this in Python? Would it look something like this: def foo(x, y): return x + y xFoo = lambda y: foo(10, y)

6 6

Re: [Python-ideas] allow line break at operators
by Matt Joiner Sept. 4, 2011

Sept. 4, 2011

+0.5 The "trailing \" workaround is nonobvious. Wrapping in () is noisy and already heavily used by other syntactical structures. Since a final ':' is needed anyway, i think this would be great. if a and b or c: do stuff() On Thu, Aug 11, 2011 at 11:02 PM, MRAB <python(a)mrabarnett.plus.com> wrote: > On 11/08/2011 05:16, Chris Rebert wrote: >> >> On Wed, Aug 10, 2011 at 7:52 PM, Yingjie Lan<lanyjie(a)yahoo.com> wrote: >>> >>> :And if we require {} then truly free indentation should be OK too! But >>> >>> :it wouldn't be Python any more. >>> >>> Of course, but not the case with ';'. Currently ';' is optional in >>> Python, >> >> I think of it more as that Python deigns to permit semicolons. >> >>> But '{' is used for dicts. Clearly, ';' and '{' are different in >>> magnitude. >>> >>> So the decision is: shall we change ';' from optional to mandatory >>> to allow free line splitting? >> >> Hell no, considering that the sizable majority of lines *aren't* >> split, which makes those semicolons completely redundant to their >> accompanying newlines. We'd be practicing poor Huffman coding by >> optimizing for the *un*common case. It would also add punctuational >> noise to what is otherwise an amazingly clean and readable syntax. >> Accidental semicolon omission is (IMO) the most irritating source of >> syntax (and, inadvertently, sometimes other more serious) errors in >> curly-braced programming languages. >> > +1 >> >> Such a core syntax feature is not going to be changed lightly (or likely >> ever). >> > I'm glad to hear that. :-) > > Although Python's use of indentation has its downside, we gain much > more then we lose, IMHO. > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas >

14 26

Re: Handling 2.7 and 3.0 Versions of Dict
by Gregory Ewing Sept. 2, 2011

Sept. 2, 2011

Ian Kelly wrote: > if sys.version_info < (3,): > getDictValues = dict.itervalues > else: > getDictValues = dict.values > > (which is basically what the OP was doing in the first place). And which he seemed to think didn't work for some reason, but it seems fine as far as I can tell: Python 2.7 (r27:82500, Oct 15 2010, 21:14:33) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> gv = dict.itervalues >>> d = {1:'a', 2:'b'} >>> gv(d) <dictionary-valueiterator object at 0x2aa210> % python3.1 Python 3.1.2 (r312:79147, Mar 2 2011, 17:43:12) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> gv = dict.values >>> d = {1:'a', 2:'b'} >>> gv(d) dict_values(['a', 'b']) -- Greg

3 2

Re: Help me understand this logging config
by Roy Smith Sept. 2, 2011

Sept. 2, 2011

In article <mailman.566.1314696279.27778.python-list(a)python.org>, Peter Otten <__peter__(a)web.de> wrote: > Roy Smith wrote: > > > I'm using django 1.3 and python 2.6. > > Isn't dictConfig() new in 2.7? It looks like that is what you are using... Oh, my, it turns out that django includes: # This is a copy of the Python logging.config.dictconfig module, # reproduced with permission. It is provided here for backwards # compatibility for Python versions prior to 2.7. Comparing the django copy to lib/logging/config.py from Python 2.7.2, they're not identical. It's likely they grabbed something earlier in the 2.7 series. I'll check 2.7.0 and 2.7.1 to see. > I tried your setup with the django-specific handler replaced by another > StreamHandler > [...] > and got what > > $ python2.7 tmp_logging.py > 2011-08-30 11:18:33,160: djfront.auth.facebook INFO <module> info-test > $ > > which seems to be what you expected. So I'm confused, too... I'll need to dig deeper. Not that I realize this may not be a Python issue per-se, I'll do some more experimentation and ask on the django mailing list. Thanks for your help.

2 3