🌐 Mailman 3 March 2003 - Python-list - python.org

unicode filenames
by Andrew Dalke March 2, 2003

March 2, 2003

Okay, I'm confused. I've been working my way through the changes put into Python 2.3. One of these is PEP 277, "Unicode file name support for Windows NT" at http://www.python.org/peps/pep-0277.html . I decided to experiment with how to use unicode filenames. I thought I understood, until I tried it out. How do I deal with possibly unicode filenames in a platform independent manner? I normally use unix. What's the right way to treat filenames under that OS? As Latin-1? Or UTF-8? As far as I can tell, filenames are simply bytes, so I can make whatever interpretation I want on the characters, and the standard viewpoint is to interpret those characters as Latin-1. [dalke@zebulon src]$ ./python Python 2.3a1 (#7, Feb 2 2003, 15:54:30) [GCC 2.96 20000731 (Red Hat Linux 7.2 2.96-112.7.2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> open("spårvägen", "w").close() >>> ^D [dalke@zebulon src]$ ls -l sp* -rw-r--r-- 1 dalke users 0 Feb 2 16:19 spårvägen [dalke@zebulon src]$ ls sp* | od -c 0000000 s p å r v ä g e n \n 0000012 [dalke@zebulon src]$ ./python Python 2.3a1 (#7, Feb 2 2003, 15:54:30) [GCC 2.96 20000731 (Red Hat Linux 7.2 2.96-112.7.2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> s = unicode("spårvägen", "latin-1") >>> s u'sp\xe5rv\xe4gen' >>> open(s, "w").close() Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character '\ue5' in position 2: ordinal not in range(128) >>> s.encode("utf8") 'sp\xc3\xa5rv\xc3\xa4gen' >>> open(s.encode("utf8"), "w").close() >>> ^D [dalke@zebulon src]$ ls -l sp* -rw-r--r-- 1 dalke users 0 Feb 2 16:19 spårvägen -rw-r--r-- 1 dalke users 0 Feb 2 16:22 spÃ¥rvÃ¤gen Does that mean Unix filenames can't contain non-Latin-1 characters? Or does it mean I need to get the info on how to interpret the filename using something from the current environment? (sys.getdefaultencoding() doesn't work since that reports 'ascii' for me.) Could different directories be encoded differently, eg /home/<encoded in ASCII>/<encoded in Latin-1>/<encoded in big5> ? And what happens when a remote file is mounted, say, from a MS Windows OS? Are they represented as UTF-8? Something else? Is that standardized or is it a property of the mount mechanism and can change accordingly? Okay, now let's see what changed in Python 2.3. According to Andrew Kuchling's "What's new in Python 2.3" at http://www.python.org/doc/2.3a1/whatsnew/node5.html On Windows NT, 2000, and XP, the system stores file names as Unicode strings. Traditionally, Python has represented file names as byte strings, which is inadequate because it renders some file names inaccessible. Python now allows using arbitrary Unicode strings (within the limitations of the file system) for all functions that expect file names, most notably the open() built-in function. If a Unicode string is passed to os.listdir(), Python now returns a list of Unicode strings. A new function, os.getcwdu(), returns the current directory as a Unicode string. ... Other systems also allow Unicode strings as file names but convert them to byte strings before passing them to the system, which can cause a UnicodeError to be raised. Applications can test whether arbitrary Unicode strings are supported as file names by checking os.path.unicode_file_names, a Boolean value. Indeed, on my Linux system, "os.path.supports_unicode_filenames" (I've sent in a bug report on the difference in attribute names) is False. Still, 'os.getcwdu()' does exist and works for my Linux system. (I removed the 'spårvägen' files and restarted Python.) >>> import os >>> os.path.supports_unicode_filenames False >>> os.getcwdu() u'/home/dalke/cvses/python/dist/src' >>> os.mkdir("spårvägen") >>> os.chdir("spårvägen") >>> os.getcwdu() Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 36: ordinal not in range(128) >>> This seems to imply that if I want to know the current working directory (eg, to display in a GUI widget, which understands how to display unicode strings) my code needs to work like this: if os.path.supports_unicode_filenames: cwd = os.getcwdu() else: encoding = .. get default filesystem encoding ... or 'latin-1' cwd = unicode(os.getcwd(), encoding) Ugly .. quite ugly. And suggestions on the proper way to handle this is not documented as far as I can find. Next I want to display the files in that directory. For MS Windows it looks like I can do that with the unicode string, as in os.listdir(cwd) but with unix I need to do [unicode(x, encoding) for x in os.listdir(s.encode(encoding))] so the portable code to list the files in a directory is something like this def my_listdir(dirname, filesystem_encoding = "ascii"): if os.path.supports_unicode_filename: return os.listdir(dirname) enc = filesystem_encoding return [unicode(x, enc) for x in os.listdir(dirname.encode(enc))] Again, that seems rather ugly, since I need to roll my own code to get what I believe to be platform independence. Similar problems hold true for mkdir and other functions. Eg, I get a unicode string from the user which is a directory to create. To work for both MS Windows and non-MS Windows machines, I need to do def my_mkdir(dirname, filesystem_encoding = "ascii"): if os.path.supports_unicode_filenames): os.path.mkdir(filename) else: os.path.mkdir(filename.encode(filesystem_encoding)) (possibly with some error catching if the new filename is in Thai for a Latin-1 filesystem.) In other words, it seems that I need to write a wrapper for all functions which might take a unicode string so when supports_unicode_filename is False I convert it to the appropriate default filesystem encoding. (Again, if different directory components can be in different character sets then this doesn't work. But I don't think anyone can reasonable expect that.) It seems, in my naive view of unicode, that there should be a system-wide function to get/set the default filesystem encoding, and the Python functions to mkdir, listdir, rmdir, etc. should use that encoding when a Unicode string is passed in to them, and that the default encoding be ASCII as it is now. But as I said, I am naive about unicode, so this post is meant as a shout for help from those more experienced, to clear up my own confusion. Andrew dalke(a)dalkescientific.com

16 33

Re: Vote on PEP 308: Ternary Operator
by Raymond Hettinger March 2, 2003

March 2, 2003

> > From: <valid email address> > > I assume those are the e-mail headers, the rest is content? Yes. > What happens on a syntax/format error? I reply to email saying that the vote needs to be resent.. > > Q. any write-in vote > > What does Q mean? Can I still select a non-seconded choice? You can select anything. Q is there in-case your favorite isn't on the list. > Is c case sensitive? Above, C was capitalized, is a lowercase c > accpepted? Yes. > > If you make a mistake or change your mind, you may re-submit > > your vote and the previous vote will be thrown out. > > ...but am I notified if I make a format or syntax mistake? Yes. Raymond Hettinger

1 0

Re: PEP 308: Pep Update
by Sheila King March 2, 2003

March 2, 2003

On Fri, 28 Feb 2003 02:36:06 +0100, Carel Fellinger <carel.fellinger(a)chello.nl> wrote in comp.lang.python in article <mailman.1046396304.1252.python-list(a)python.org>: > Serious, are you still in contact with teachers teaching Python? > If so, could you pull off another of your Guido convincing things > and get those teachers to do a little usability study with respect > to the readability of the conditional expression? Well, I do run a mailing list for AP Computer Science Teachers. It is not a very active list. (See http://www.thinkspot.net/sheila/computers/APCSlistInfo.html ) The College Board has an "official" list that gets many times the traffic of mine. The AP Comp Sci teachers do not really teach Python. A few. Probably very few. Most of them teach C++ and are getting ready to teach Java beginning next year. I have not been very visible in the community as I've not been teaching computer science, nor teaching at the high school level, for the last two years. Currently part time math lecturer at a California State Univ. and doing full time computer tech support for a web hosting company. So, I'm not sure I could really get a "turn out" if I tried to do a survey. Last time I did one, I was very visible and active in the community. At this time, I've been quite dormant. And I'm not so sure that it is appropriate, since the teachers I would survey would be, for the most part, not users of Python. In fact, I'd even fear that they'd be half in favor of the <cond>?<true>:<false> form, since most of them teach C++. -- Sheila King http://www.thinkspot.net/sheila/ http://www.k12groups.org/

1 0

Re: Accessing one's main script's global from imported libraries
by marshall＠spamhole.com March 2, 2003

March 2, 2003

czrpb <nanotech(a)europa.com> wrote in message news:<mailman.1046111768.26279.python-list(a)python.org>... > All: > > Say I have spam.py with the global taste. > Say in spam.py I import eggs. > Say I want eggs to access spam.py's taste global. > > How might I do this? > > thanks!! Quentin > As you will no doubt read, you are not _supposed_ to do this. You are _supposed_ to be passing parameters from the objects or functions in spam.py to the objects or functions in eggs.py. However, I have believe the best practice to acheive what you want is to have a separate file (e.g. spamglobals.py) and have all other files import that. I could very well be wrong about that but that is what works for me (newbie disclaimer). I think the more general question is: "What is the best way to split a large program up into several modules and still let them all have access to application global data?" What I have been doing is creating an Application instance which has all of the global data (like maxscreensize, datapath, etc.) as attributes. Then I pass this App instance as a parameter to every other object that the application creates. If there is a better way I would sure like to hear about it. Marshall

2 2

New style classes
by Travis Oliphant March 2, 2003

March 2, 2003

Forgive me if this question is naive. I am trying to understand new-style classes. In particular I want to be able to detect them in C-code. I understand that PyInstance_Check(obj) returns true in C if obj is an instance of an old-style class. However, it does not return true if obj is an instance of a new-style class. So, how does one check in C whether or not obj is an instance of a class (either old style or new style)? Thanks, -Travis -- Travis Oliphant Assistant Professor 459 CB Electrical and Computer Engineering Brigham Young University Provo, UT 84602 Tel: (801) 422-3108 oliphant.travis(a)ieee.org

2 1

cross platform application configuration files
by Lee Harr March 2, 2003

March 2, 2003

Hi; I am writing a game library on top of python and pygame ( http://www.nongnu.org/pygsear/ ) and I want to store some application defaults somewhere.... I work mostly on FreeBSD, so I would tend to create a file for my application ~/.pygsear and store app configuration there. Of course there are other platforms *cough* etc *cough* where that makes no sense. I really have no desire to learn the intricacies of using the registry, but if there were a module already built where I might say: import xpconf xpconf.store('pygsear', 'WINWIDTH', 800) xpconf.store('pygsear', 'WINHEIGHT', 600) WINWIDTH = xpconf.get('pygsear', 'WINWIDTH') WINHEIGHT = xpconf.get('pygsear', 'WINHEIGHT') and it would just "do the right thing" for the platform, I might be convinced to use that... Does such a beastie exist?

6 10

Advice for wxPython error: "'NoneType' object is not callable" in ignored
by follower＠iname.com March 2, 2003

March 2, 2003

This post is about an issue relating to wxPython, module importing and global variables referring to (native?) wxWindows objects. (It is mostly intended as Google-bait for anyone else having the problem in the future, but any further information is welcome.) The following Python error message may indicate you have this problem: Exception exceptions.TypeError: "'NoneType' object is not callable" in ignored Simplest Problem Form ===================== You have these files: myapp.py mypackage/__init__.py mypackage/mymodule.py With the following content: # ---- myapp.py ---- import mypackage.mymodule # ---- mypackage/__init__.py ---- # <empty> # ---- mypackage/mymodule.py ---- import wxPython.wx import mypackage # <1> myglobal = wxPython.wx.wxSize(780, 580) # -- <end> -- When you execute the command: python myapp.py The following is the output: Exception exceptions.TypeError: "'NoneType' object is not callable" in ignored Solutions ========= * Don't have global variables ('myglobal' in this example) referring to wxWindows objects. (Best solution.) * Don't explictly import the parent package from a module within it. (i.e. delete the line marked with "<1>"--it's not necessary in real life anyway.) While this solves the problem in certain circumstances you may run into the problem again if you have sub-packages which import the parent package. Explanation (as far as I understand things) =========== Error messages of the form "<foo> in <bar> ignored" are generated when an exception occurs in an object's __del__() method. (In this case it would seem Python can't identify where the exception occurred which is why "<bar>" is blank, leaving the message "<foo> in ignored".) The actual message indicates there are issues with the order of object deletion with wxPython and global variables referring to wxPython/wxWindows objects. Other Resources =============== See also particularly these messages on the 'wxPython-users' mailing list: * "[wxPython-users] Errors when app is closed by logging out", <http://aspn.activestate.com/ASPN/Mail/Message/1545060> <http://aspn.activestate.com/ASPN/Mail/Message/1545216> * "[wxPython-users] weird bug with imports", <http://aspn.activestate.com/ASPN/Mail/Message/1463373> <http://aspn.activestate.com/ASPN/Mail/Message/1463822> Hope this saves someone some time. :-)

1 0

UnicodeError: ASCII encoding error: ordinal not in range(128)
by Hasan Diwan March 2, 2003

March 2, 2003

I have a script to parse RSS feeds into HTML. Some of these RSS feeds contain characters outside the 7-bit ASCII range, and the page is set to display in utf8. My question is how to force the script below to output strings in UTF8 as opposed to ASCII: #!/sw/bin/python from xml.dom import minidom import string import urllib import time import sys import re import zlib #Change these to reflect your reality class news: def load(self, url): p=None try: p=minidom.parse(urllib.urlopen(url)) except: print 'unreachable URL '+url, return p DEFAULT_NAMESPACES = \ (None, # RSS 0.91, 0.92, 0.93, 0.94, 2.0 'http://purl.org/rss/1.0/', # RSS 1.0 'http://my.netscape.com/rdf/simple/0.9/' # RSS 0.90 ) def getElementsByTagName(self, node, tagName, possibleNamespaces=DEFAULT_NAMESPACES): for namespace in possibleNamespaces: children = node.getElementsByTagName(tagName) if len(children): return children return [] def first(self,node, tagName, possibleNamespaces=DEFAULT_NAMESPACES): children = node.getElementsByTagName(tagName) return len(children) and children[0] or None def textOf(self,node): return node and "".join([child.data for child in node.childNodes]) or "" DUBLIN_CORE = ('http://purl.org/dc/elements/1.1/',) -- Hasan Diwan <hdiwan(a)mac.com> OpenPGP keyID: 0x7EE3855B http://ibn.com/~hdiwan Fingerprint: 42F0 5758 C3EB BA1F ABD2 ED49 3390 CCF0 7EE3 855B

2 1

win32all for 2.3 available
by Mark Hammond March 2, 2003

March 2, 2003

I have made a new win32all for 2.3 available. As nothing significant has changed, I haven't bothered with 2.1 oe 2.2 cuts of this same build. http://starship.python.net/crew/mhammond/win32/Downloads.html Mark.

1 0

Python Audio I/O libs - Linux/Windows
by David McNab March 2, 2003

March 2, 2003

Hi, Does anyone know of a multiplatform audio lib (well, at least Linux and Windows) that can play *and capture* audio, in a system-independent (eg WAV) format? I don't want 200 different delays, 150 choruses, 300 guitar effects, spatial processing etc - I just want a simple Python API that will let me open the system default audio device, set a sampling rate and grab small chunks of raw audio, do stuff with them, then play them back. So far, have checked out audiere (play only), snack (requires Tk - unacceptable), and wx (play only). All help appreciated. Cheers David

1 0