🌐 Mailman 3 September 2006 - Python-list - python.org

Re: PATCH: Speed up direct string concatenation by 20+%!
by Larry Hastings Oct. 5, 2006

Oct. 5, 2006

Steve Holden wrote: > you should diff your source against the current > SVN repository and lodge that diff as a patch on SourceForge. Okay, I'll try to do that today. > Your suggested bug isn't, I think a real bug in the current > implementation because as I understand it Python strings do always > include a trailing null byte to act as a terminator when the string > is passed to C code in extensions. Ah! Excellent point. So the Python source tree *does* have an official position on what the first byte of a zero-length string should be. I'll fix my code so that is ensured. > Nice idea, though. You might also see how it does on tasks like > > s = "" > for i in range(100000): > s += "a" Sure. Here are the results, but with range (10000000): Python 2.5 release: 31.0s Python 2.5 locally built: 30.7s Python 2.5 concat: 4.4s % Improvement: *697%*! Woah! Thanks for suggesting a benchmark where it really shines :) The real big win here is storing eight pointers per concatenation object. That means 1/8th as many objects created/destroyed. In testing your suggested benchmark, it overflowed the stack when freeing the concatenation tree. So I changed deallocation of string concatenation objects so it's iterative down the left-hand side like rendering; now it's happy again. Thanks again for suggesting the test. It would still blow up if you ran s = "" for i in range(10000000): s = "a" + s because it's only clever about recursion down the left-hand side. So my advice is, don't. :) > Also note that some optimisation has recently been performed on string > concatenation, which I presume your code has already taken into account. Totally. I'm just glad there was something left for me to contribute! Rob Williscroft wrote: > On the python 3k list there is a thread about stings becoming "views", > I think this is similar to what you are doing here. As I understand it, sort-of. Definitely moreso in my section on THE FUTURE--adding another subtype of string for slices which point into other strings. I believe views are more sophisticated however. Felipe Almeida Lessa wrote: > What about "a + b"? Or "a + b + c"? Does it have a large overhead on > small string concatenations? It's *slightly* slower for two: def addTwoThings(a, b): return a + b for i in range(10000000): x = addTwoThings("aaa", "bbb") Python 2.5 release: 6.219s Python 2.5 locally built: 6.219s Python 2.5 concat: 6.234s % improvement: -0.03% But starts paying off already, even with three: def addThreeThings(a, b, c): return a + b + c for i in range(10000000): x = addThreeThings("aaa", "bbb", "ccc") Python 2.5 release: 7.9s Python 2.5 locally built: 7.7s Python 2.5 concat: 7.2s % improvement: 6.94% Note that I was lazy and only did one benchmark run apiece. Also, keep in mind that memory utilization will be a little higher with the string concatenation objects. Carl Friedrich Bolz wrote: > Robin Becker wrote: > > wouldn't this approach apply to other additions eg list+list seq+seq etc > > etc. > no, I think it depends on strings being immutable. Exactly. /larry/

4 4

Resuming a program's execution after correcting error
by Sheldon Oct. 5, 2006

Oct. 5, 2006

Hi. Does anyone know if one can resume a python script at the error point after the error is corrected? I have a large program that take forever if I have to restart from scratch everytime. The error was the data writing a file so it seemed such a waste if all the data was lost and must be recalculated again. Sincerely, Sheldon

10 13

MySQLdb for Python 2.5
by Harold Trammel Oct. 5, 2006

Oct. 5, 2006

Hi everyone, Does anyone know the status of a version of MySQLdb that will work with Python 2.5? I will accept a workaround if you know one. Thanks in advance. Harold Trammel

5 5

releasing memory to malloc
by iker.arizmendi＠gmail.com Oct. 5, 2006

Oct. 5, 2006

Is there any way to get Python to release memory back to the C allocator? I'm currently running a script that goes through the following steps: 1) Creates a very large number of Python objects to produce a relatively small data structure that sits in a C extension. The Python objects consume quite a bit of memory. 2) Releases all the Python objects. 3) Invokes a function of said C extension for further processing. This step needs as much memory as possible. I'd like step 2 to return memory to the C allocator so that it is available to the extension in step 3 (which uses malloc). Regards, Iker Arizmendi

4 5

preemptive OOP?
by John Salerno Oct. 4, 2006

Oct. 4, 2006

Ok, I have a new random question for today -- feel free to ignore and get back to your real jobs! :) Anyway, I'm creating a GUI (yes, all part of my master plan to eventually have some sort of database application working) and it's going to involve a wx.Notebook control. I think I have two options for how I can do this. Within the class for the main window frame, I can say: notebook = wx.Notebook(panel) # panel is parent of the Notebook control This uses the default wx.Notebook class, and works just fine. But I was thinking, is it a smart idea to do this instead: class MyNotebook(wx.Notebook): def __init__(self, parent): wx.Notebook.__init__(self, parent) and then call it from the main frame as: notebook = MyNotebook(panel) This seems to allow for future expansion of the customized Notebook class, but at the same time I have no idea how or why I'd want to do that. So my question in general is, is it a good idea to default to an OOP design like my second example when you aren't even sure you will need it? I know it won't hurt, and is probably smart to do sometimes, but maybe it also just adds unnecessary code to the program.

7 12

[ANN] IronPython Community Edition 1.0r2
by Sanghyeon Seo Oct. 4, 2006

Oct. 4, 2006

This is the second release of IronPython Community Edition (IPCE), 1.0 revision 2, based on IronPython 1.0. Get it here: http://sparcs.kaist.ac.kr/~tinuviel/download/IPCE-1.0r2.zip Binary is built with Mono 1.1.17.1. BIG WARNING: it won't work with Mono versions below 1.1.17. Please don't mail me or IronPython mailing list before checking your Mono version! IPCE has a new home on SourceForge: http://fepy.sourceforge.net/ And here's the license and the summary of applied patches: http://fepy.sourceforge.net/license.html http://fepy.sourceforge.net/patches.html Changes in this revision: * Includes the Python standard library from CPython 2.4.3. (Not everything is included -- that would be pointless. Those I tested and work reasonably are included.) * Includes following CPython-compatible wrappers for .NET library: md5, pyexpat, select, sha, socket, ssl, unicodedata. * Includes DB-API wrapper for ADO.NET. * Includes BeautifulSoup and ElementTree for you to test. (Both work great with IronPython!) * Does not include Doc, Src, Tutorial from the original IronPython release. You know where to get them... Reduces size by about half. * Extracts to IPCE-1.0r2. (The first revision extracted to IronPython-1.0 and it could overwrite existing installation. Thanks to Anthony Baxter for pointing this out.) -- Seo Sanghyeon

3 2

Strange xml.parsers.xml import problem
by dwelch91＠gmail.com Oct. 3, 2006

Oct. 3, 2006

Hi, c.l.p.'ers- I am having a problem with the import of xml.parsers.expat that has gotten me completely stumped. I have two programs, one a PyQt program and one a command line (text) program that both eventually call the same code that imports xml.parsers.expat. Both give me different results... The code that gets called is (print statements have been added for debugging): # this is status.py import xml.parsers.expat print xml print xml.parsers print dir(xml.parsers) print repr(xml.parsers.expat.ExpatError) print xml.parsers.expat if info_device_status: try: device_status = utils.XMLToDictParser().parseXML(info_device_status) log.debug_block("info_device_status", info_device_status) log.debug(device_status) except xml.parsers.expat.ExpatError: log.error("Device Status XML parse error") device_status = {} if info_ssp: try: ssp = utils.XMLToDictParser().parseXML(info_ssp) log.debug_block("info_spp", info_ssp) log.debug(ssp) except xml.parsers.expat.ExpatError: log.error("SSP XML parse error") ssp = {} When I run this code from my console program, it prints out: <module '_xmlplus' from '/usr/lib/python2.4/site-packages/_xmlplus/__init__.pyc'> <module 'xml.parsers' from '/usr/lib/python2.4/site-packages/_xmlplus/parsers/__init__.pyc'> ['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__path__', 'expat', 'pyexpat'] <class xml.parsers.expat.ExpatError at 0x2aaaac6c6950> <module 'xml.parsers.expat' from '/usr/lib/python2.4/site-packages/_xmlplus/parsers/expat.pyc'> and all is well... However, from the PyQt program I get: <module '_xmlplus' from '/usr/lib/python2.4/site-packages/_xmlplus/__init__.pyc'> <module '_xmlplus.parsers' from '/usr/lib/python2.4/site-packages/_xmlplus/parsers/__init__.pyc'> ['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__path__', 'sgmlop'] Traceback (most recent call last): [...snip...] File "/home/dwelch/tip/linux-imaging-and-printing/src/base/status.py", line 980, in StatusType6 print repr(xml.parsers.expat.ExpatError) AttributeError: 'module' object has no attribute 'expat' Somehow, the import has brought in "_xmlplus" rather than "xml", and "expat" is no longer in the xml.parsers module. This is on the same machine with the same user account, running each program "back to back". Running each program as a different user made no difference. Everything is also OK if I manually enter the import and print commands into a Python >>> prompt. Any help much appreciated! -Don

1 2

Escapeism
by Kay Schluehr Oct. 3, 2006

Oct. 3, 2006

Usually I struggle a short while with \ and either succeed or give up. Today I'm in a different mood and don't give up. So here is my question: You have an unknown character string c such as '\n' , '\a' , '\7' etc. How do you echo them using print? print_str( c ) prints representation '\a' to stdout for c = '\a' print_str( c ) prints representation '\n' for c = '\n' ... It is required that not a beep or a linebreak shall be printed. First of all it has be remarked that it is impossible to a certain extent. That's because e.g. c = '\a' and c = '\7' do represent the same string but this ambiguity doesn't occur for many numbers. But lets weaken the requirement and fix a canonical representation in case of ambiguity. I'm still getting stuck here.

9 11

iterator question
by Neal Becker Oct. 3, 2006

Oct. 3, 2006

Any suggestions for transforming the sequence: [1, 2, 3, 4...] Where 1,2,3.. are it the ith item in an arbitrary sequence into a succession of tuples: [(1, 2), (3, 4)...] In other words, given a seq and an integer that specifies the size of tuple to return, then for example: seq = [a,b,c,d,e,f] for e in transform (seq, 2): print e would return (a,b) (c,d) (e,f)

4 3

does anybody earn a living programming in python?
by walterbyrd Oct. 3, 2006

Oct. 3, 2006

If so, I doubt there are many. I wonder why that is?

37 42