🌐 Mailman 3 July 2011 - Python-list - python.org

Re: emacs lisp text processing example (html5 figure/figcaption)
by Xah Lee July 5, 2011

July 5, 2011

On Jul 5, 12:17 pm, Ian Kelly <ian.g.ke...(a)gmail.com> wrote: > On Mon, Jul 4, 2011 at 12:36 AM, Xah Lee <xah...(a)gmail.com> wrote: > > So, a solution by regex is out. > > Actually, none of the complications you listed appear to exclude > regexes. Here's a possible (untested) solution: > > <div class="img"> > ((?:\s*<img src="[^.]+\.(?:jpg|png|gif)" alt="[^"]+" width="[0-9]+" > height="[0-9]+">)+) > \s*<p class="cpt">((?:[^<]|<(?!/p>))+)</p> > \s*</div> > > and corresponding replacement string: > > <figure> > \1 > <figcaption>\2</figcaption> > </figure> > > I don't know what dialect Emacs uses for regexes; the above is the > Python re dialect. I assume it is translatable. If not, then the > above should at least work with other editors, such as Komodo's > "Find/Replace in Files" command. I kept the line breaks here for > readability, but for completeness they should be stripped out of the > final regex. > > The possibility of nested HTML in the caption is allowed for by using > a negative look-ahead assertion to accept any tag except a closing > </p>. It would break if you had nested <p> tags, but then that would > be invalid html anyway. > > Cheers, > Ian emacs regex supports shygroup (the 「(?:…)」) but it doesn't support the negative assertion 「?!…」 though. but in anycase, i can't see how this part would work <p class="cpt">((?:[^<]|<(?!/p>))+)</p> ? Xah

2 1

emacs lisp text processing example (html5 figure/figcaption)
by Xah Lee July 5, 2011

July 5, 2011

OMG, emacs lisp beats perl/python again! Hiya all, another little emacs lisp tutorial from the tiny Xah's Edu Corner. 〈Emacs Lisp: Processing HTML: Transform Tags to HTML5 “figure” and “figcaption” Tags〉 xahlee.org/emacs/elisp_batch_html5_tag_transform.html plain text version follows. ------------------------------------------ Emacs Lisp: Processing HTML: Transform Tags to HTML5 “figure” and “figcaption” Tags Xah Lee, 2011-07-03 Another triumph of using elisp for text processing over perl/python. ---------------------------- The Problem -------------- Summary I want batch transform the image tags in 5 thousand html files to use HTML5's new “figure” and “figcaption” tags. I want to be able to view each change interactively, while optionally give it a “go ahead” to do the whole job in batch. Interactive eye-ball verification on many cases lets me be reasonably sure the transform is done correctly. Yet i don't want to spend days to think/write/test a mathematically correct program that otherwise can be finished in 30 min with human interaction. -------------- Detail HTML5 has the following new tags: “figure” and “figcaption”. They are used like this: <figure> <img src="cat.jpg" alt="my cat" width="167" height="106"> <figcaption>my cat!</figcaption> </figure> (For detail, see: HTML5 “figure” ＆ “figurecaption” Tags Browser Support) On my website, i used a similar structure. They look like this: <div class="img"> <img src="cat.jpg" alt="my cat" width="167" height="106"> <p class="cpt">my cat!</p> </div> So, i want to replace them with the HTML5's new tags. This can be done with a regex. Here's the “find” regex: <div class="img"> ?<img src="$[^.]+?$\.jpg" alt="$[^"]+?$" width="$[0-9]+?$" height="$[0-9]+?$">? <p class="cpt">$[^<]+?$</p> ?</div> Here's the replacement string: <figure> <img src="\1.jpg" alt="\2" width="\3" height="\4"> <figcaption>\5</figcaption> </figure> Then, you can use “find-file” and dired's “dired-do-query-replace- regexp” to work on your 5 thousand pages. Nice. (See: Emacs: Interactively Find ＆ Replace String Patterns on Multiple Files.) However, the problem here is more complicated. The image file may be jpg or png or gif. Also, there may be more than one image per group. Also, the caption part may also contain complicated html. Here's some examples: <div class="img"> <img src="cat1.jpg" alt="my cat" width="200" height="200"> <img src="cat2.jpg" alt="my cat" width="200" height="200"> <p class="cpt">my 2 cats</p> </div> <div class="img"> <img src="jamie_cat.jpg" alt="jamie's cat" width="167" height="106"> <p class="cpt">jamie's cat! Her blog is <a href="http://example.com/ jamie/">http://example.com/jamie/</a></p> </div> So, a solution by regex is out. ---------------------------- Solution The solution is pretty simple. Here's the major steps: Use “find-lisp-find-files” to traverse a dir. For each file, open it. Search for the string <div class="img"> Use “sgml-skip-tag-forward” to jump to its closing tag. Save the positions of these tag begin/end positions. Ask user if she wants to replace. If so, do it. (using “delete- region” and “insert”) Repeat. Here's the code: ;; -*- coding: utf-8 -*- ;; 2011-07-03 ;; replace image tags to use html5's “figure” and “figcaption” tags. ;; Example. This: ;; <div class="img">…</div> ;; should become this ;; <figure>…</figure> ;; do this for all files in a dir. ;; rough steps: ;; find the <div class="img"> ;; use sgml-skip-tag-forward to move to the ending tag. ;; save their positions. (defun my-process-file (fpath) "process the file at fullpath FPATH ..." (let (mybuff p1 p2 p3 p4 ) (setq mybuff (find-file fpath)) (widen) (goto-char 0) ;; in case buffer already open (while (search-forward "<div class=\"img\">" nil t) (progn (setq p2 (point) ) (backward-char 17) ; beginning of “div” tag (setq p1 (point) ) (forward-char 1) (sgml-skip-tag-forward 1) ; move to the closing tag (setq p4 (point) ) (backward-char 6) ; beginning of the closing div tag (setq p3 (point) ) (narrow-to-region p1 p4) (when (y-or-n-p "replace?") (progn (delete-region p3 p4 ) (goto-char p3) (insert "</figure>") (delete-region p1 p2 ) (goto-char p1) (insert "<figure>") (widen) ) ) ) ) (when (not (buffer-modified-p mybuff)) (kill-buffer mybuff) ) ) ) (require 'find-lisp) (let (outputBuffer) (setq outputBuffer "*xah img/figure replace output*" ) (with-output-to-temp-buffer outputBuffer (mapc 'my-process-file (find-lisp-find-files "~/web/xahlee_org/ emacs/" "\\.html$")) (princ "Done deal!") ) ) Seems pretty simple right? The “p1” and “p2” variables are the positions of start/end of <div class="img">. The “p3” and “p4” is the start/end of it's closing tag </ div>. We also used a little trick with “widen” and “narrow-to-region”. It lets me see just the part that i'm interested. It narrows to the beginning/end of the div.img. This makes eye-balling a bit easier. The real time-saver is the “sgml-skip-tag-forward” function from “html- mode”. Without that, one'd have to write a mini-parser to deal with html's nested ways to be able to locate the proper ending tag. Using the above code, i can comfortably eye-ball and press “y” at the rate of about 5 per second. That makes 300 replacements per minute. I have 5000+ files. If we presume there are 6k replacement to be made, then at 5 per second means 20 minutes sitting there pressing “y”. Quite tiresome. So, now, the next step is simply to remove the asking (y-or-n-p "replace?"). Or, if i'm absolutely paranoid, i can make emacs write into a log buffer for every replacement it makes (together with the file path). When the batch replacement is done (probably under 3 minutes), i can simply scan thru the log to see if any replacement went wrong. For how to do that, see: Emacs Lisp: Multi-Pair String Replacement with Report. But what about replacing <p class="cpt">…</p> with <figcaption>…</ figcaption>? I simply copy and pasted the above code into a new file, just made changes in 4 places. So, the replacing figcaption part is considered a separete batch job. Of course, one could spend extra hour or so to make the code do them both in one pass, but is that one extra hour of thinking ＆ coding worthwhile for this one-time job? I ♥ Emacs, do you? --------------------------------- PS perl and python solution welcome. I haven't looked at perl or python's html parser libs for 5+ years. Though, 2 little requirement: 1. it must be correct, of course. Cannot tolerate the possiblility that maybe one out of a thousand replacement it introduced a mismatched tag. (but you can assume that all the input html files are w3c valid) 2. it must not change the formatting of the html pages. i.e. adding/ removing spaces or tabs. Xah

3 4

Re: emacs lisp text processing example (html5 figure/figcaption)
by Xah Lee July 5, 2011

July 5, 2011

On Jul 5, 12:17 pm, Ian Kelly <ian.g.ke...(a)gmail.com> wrote: > On Mon, Jul 4, 2011 at 12:36 AM, Xah Lee <xah...(a)gmail.com> wrote: > > So, a solution by regex is out. > > Actually, none of the complications you listed appear to exclude > regexes. Here's a possible (untested) solution: > > <div class="img"> > ((?:\s*<img src="[^.]+\.(?:jpg|png|gif)" alt="[^"]+" width="[0-9]+" > height="[0-9]+">)+) > \s*<p class="cpt">((?:[^<]|<(?!/p>))+)</p> > \s*</div> > > and corresponding replacement string: > > <figure> > \1 > <figcaption>\2</figcaption> > </figure> > > I don't know what dialect Emacs uses for regexes; the above is the > Python re dialect. I assume it is translatable. If not, then the > above should at least work with other editors, such as Komodo's > "Find/Replace in Files" command. I kept the line breaks here for > readability, but for completeness they should be stripped out of the > final regex. > > The possibility of nested HTML in the caption is allowed for by using > a negative look-ahead assertion to accept any tag except a closing > </p>. It would break if you had nested <p> tags, but then that would > be invalid html anyway. > > Cheers, > Ian that's fantastic. Thanks! I'll try it out. Xah

1 0

Question about how to get line buffering from paramiko
by Steven W. Orr July 5, 2011

July 5, 2011

I'm writing a program that uses paramiko to run a lot of commands over ssh. Some of the commands take time to run and they write to stdout and stderr as a normal part of their operation so that we can see progress happening. I can't seem to get the output from the remote commands (which is input to me) to be line buffered. If I run the commands using ssh, they line buffer nicely. If I run them through paramiko, they end up fully buffered. I stole this code I found as a base. The code that I got looked like the execute_capture method (below). I added the execute method so that I could poll for the result from both stderr and stdout. Note that I am calling channel.get_pty, but that doesn't change the fact that the results are not line buffered. Can anyone suggest a way to solve this? The code I'm using follows: #! /usr/bin/python """ Friendly Python SSH2 interface using paramiko """ import os import sys import tempfile import paramiko import select from collections import namedtuple ExecStatus = namedtuple('ExecStatus', 'status stdout stderr') class Connection(object): """ Connects and logs into the specified hostname. Arguments that are not given are guessed from the environment. """ def __init__(self, host, username = None, private_key = None, password = None, port = 22, blocking_cmds = True, verbose = False, ): self._sftp_live = False self._sftp = None if not username: username = os.environ['LOGNAME'] # Log to a temporary file if requested. if verbose: self.templog = tempfile.mkstemp('.txt', 'ssh-')[1] paramiko.util.log_to_file(self.templog) else: self.templog = False # Begin the SSH transport. self._transport = paramiko.Transport((host, port)) self._tranport_live = True # Authenticate the transport. if password: # Using Password. self._transport.connect(username = username, password = password) else: # Use Private Key. if not private_key: # Try to use default key. if os.path.exists(os.path.expanduser('~/.ssh/id_rsa')): private_key = '~/.ssh/id_rsa' elif os.path.exists(os.path.expanduser('~/.ssh/id_dsa')): private_key = '~/.ssh/id_dsa' else: raise TypeError, "You have not specified a password or key." private_key_file = os.path.expanduser(private_key) rsa_key = paramiko.RSAKey.from_private_key_file(private_key_file) self._transport.connect(username = username, pkey = rsa_key) def _sftp_connect(self): """Establish the SFTP connection.""" if not self._sftp_live: self._sftp = paramiko.SFTPClient.from_transport(self._transport) self._sftp_live = True def get(self, remotepath, localpath = None): """Copies a file between the remote host and the local host.""" if not localpath: localpath = os.path.split(remotepath)[1] self._sftp_connect() self._sftp.get(remotepath, localpath) def put(self, localpath, remotepath = None): """Copies a file between the local host and the remote host.""" if not remotepath: remotepath = os.path.split(localpath)[1] self._sftp_connect() self._sftp.put(localpath, remotepath) def execute(self, command): """ Execute the given commands on a remote machine. Return value is exit status of the remote command. """ # This execute method is similar to execute_capture below. The # difference is that this method gets the stdout and stderr of # the runnning command and forwards it on to the correct # channel within this process. # To do this, we use the poll(2) system call which comes from # the select package. def _write(fd, chan, syschan): """ _write internal method to check an fd from the list of fds for a POLLIN event, read the data that's there, and if there's anything there, then write it to the correct channel. Return True if there was something to read. """ ret = False if fd[1] & select.POLLIN: if fd[0] == chan.channel.fileno(): ss = chan.readline() ret = len(ss) != 0 if ret: # No need to strip and then print with a newline. # because every line is newline terminated. print >> syschan, ss[:-1] return ret # Open a channel of type session. Same as open_channel('session') channel = self._transport.open_session() # Calling get_pty does get us a pty # If the 2nd arg is 1, then it should be line buffered. Apparently, # in this context, line buffering refers to output to the process, # not input from the process. channel.get_pty() # Run command on the session of type channel. This returns immediately. channel.exec_command(command) # Get the stdout and stderr file descriptors. In this context, makefile # has nothing to do with the make utility. It's about making # file descriptors. stdout = channel.makefile('rb', 1) stderr = channel.makefile_stderr('rb', 1) # Create a polling object. We really only care about POLLIN events. po = select.poll() po.register(stdout.channel, select.POLLIN) po.register(stderr.channel, select.POLLIN) # Set found_output tto True to start the loop. Set it to False # as an initial value inside the loop, and OR the value on if any data # got written by either channel. found_output = True while found_output == True: found_output = False fds = po.poll() for fd in fds: found_output |= _write(fd, stdout, sys.stdout) found_output |= _write(fd, stderr, sys.stderr) status = channel.recv_exit_status() channel.close() return status def execute_capture(self, command): """ Execute the given commands on a remote machine. Return value is a 3-tuple: exit status, stdout and stderr Output is not written out. """ # Open a channel of type session. Same as open_channel('session') channel = self._transport.open_session() # Run command on the session of type channel. This returns immediately. channel.exec_command(command) # Collect stdout and stderr into lists. # The alternative would be to harvest the 3-tuple std{in,out,err} # of file descriptors returned by exec_command. stdout = channel.makefile('rb', -1).readlines() stderr = channel.makefile_stderr('rb', -1).readlines() # Not well documented, but recv_exit_status will block until # the command completes. Return value is the exit status. status = channel.recv_exit_status() channel.close() return ExecStatus(status, stdout, stderr) def close_transport(self): # Close the SSH Transport. if self._tranport_live: self._transport.close() self._tranport_live = False def close_sftp(self): # Close SFTP Connection. if self._sftp_live: self._sftp.close() self._sftp_live = False def close(self): """Closes the connection and cleans up.""" self.close_transport() self.close_sftp() # start the ball rolling. if __name__ == "__main__": def proc_exec(cmd): status = myssh.execute(cmd) print "Just ran %s with exit status %d"%(cmd, status) # Another choice is to call # def proc_exec(cmd): # run = myssh.execute_capture(cmd) # print "Just ran %s with exit status %d"%(cmd, status) # if run.stdout: # print "output:\n", ''.join(run.stdout)[:-1] # if run.stderr: # print "error:\n", ''.join(run.stderr)[:-1] # Little test when called directly. # Set these to your own details. myssh = Connection('host', username='user') proc_exec('pwd') proc_exec('false') proc_exec('xxx.sh') proc_exec('cd sorr; pwd') proc_exec('pwd') proc_exec('tty') myssh.close() -- Time flies like the wind. Fruit flies like a banana. Stranger things have .0. happened but none stranger than this. Does your driver's license say Organ ..0 Donor?Black holes are where God divided by zero. Listen to me! We are all- 000 individuals! What if this weren't a hypothetical question? steveo at syslang.net

1 0

VIDEOS FOR HOT GUYS
by SAHITHI July 5, 2011

July 5, 2011

FOR GOOD JOBS SITES TO YOU http://goodjobssites.blogspot.com/ FOR HOT PHOTO&VIDEOS KATRINA KAIF IN BEAUTIFUL RED DRESS http://southactresstou.blogspot.com/2011/05/katrina-kaif_22.html GOOD LOOKING DEEPIKA PADUKONE http://southactresstou.blogspot.com/2011/05/deepika-padukone_22.html AISHWARYA RAI UNBELIVABLE PHOTO http://southactresstou.blogspot.com/2011/05/aishwarya-rai.html TAPSEE RARE PHOTOS http://southactresstou.blogspot.com/2011/06/tapsee-rare-photos.html FOR FAST UPDATES IN TELUGU FILM INDUSTRY http://allyouwants.blogspot.com

1 0

Re: Implicit initialization is EVIL!
by nn July 5, 2011

July 5, 2011

On Jul 4, 11:35 am, Robin Becker <ro...(a)reportlab.com> wrote: > On 03/07/2011 23:21, Chris Angelico wrote: > ......... > > > var(0x14205359) x # Don't forget to provide an address where the > > object will be located > > x=42 > > ........ > did you forget to specify the memory bank and computer (and presumably planet > etc etc....) > -molly-coddled-ly yrs- > Robin Becker Ah, I see we have a mainframe programmer among us ... :-)

2 1

Re: Testing if a global is defined in a module
by Grant Edwards July 5, 2011

July 5, 2011

On 2011-07-05, Tim Johnson <tim(a)johnsons-web.com> wrote: > * Ian Kelly <ian.g.kelly(a)gmail.com> [110704 20:37]: >> >> It sounds like what you really want is to detect the names *exported* >> by the module, then. i > Yes! >> Why not do it the same way Python does it? If >> the module defines an "__all__" attribute, then it is taken to be a >> sequence of strings which are the exported names. Otherwise, the >> exported names are taken to be all the names in the module dict that >> don't begin with an underscore. > > :) Oh here we go again. Another python feature I didn't know about > or have forgotten. You could probably implement at least two more languages using nothing but Python features I don't know about or have forgotten. :) Yet I still manage to get a lot accomplished using Python. -- Grant Edwards grant.b.edwards Yow! I don't know WHY I at said that ... I think it gmail.com came from the FILLINGS in my rear molars ...

1 0

Re: Testing if a global is defined in a module
by Steven D'Aprano July 5, 2011

July 5, 2011

Tim Johnson wrote: >> It seems to me that your approach here is unnecessarily complex and >> fragile. I don't know what problem you are trying to solve, but trying to >> solve it by intraspecting differences that aren't differences is surely >> the wrong way to do it. > See my last post... Yes, but what are you actually *trying to do*? "Detecting data members" is not an end in itself. Why do you think you need to detect data members? -- Steven

4 5

module problem on windows 64bit
by miamia July 5, 2011

July 5, 2011

Hello, on 32-bit windows everything works ok but on 64-bit win I am getting this error: Traceback (most recent call last): File "app.py", line 1040, in do_this_now File "kinterbasdb\__init__.pyc", line 119, in <module> File "kinterbasdb\_kinterbasdb.pyc", line 12, in <module> File "kinterbasdb\_kinterbasdb.pyc", line 10, in __load ImportError: DLL load failed: This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem. How to get it work on 64bit windows as well? many thanks

2 1

can I package a distutil based python app in deb?
by hackingKK July 5, 2011

July 5, 2011

Hello all, I guess the subject line says it all. I want to package a python app to deb. I have 3 interesting issues with it. 1, I would want it to run on Ubuntu 10.04, Ubuntu 10.10, Ubuntu 11.04 and Debian 5. 2, the package depends on another python package which is also distutil based. 3, The second package needs to run in a virtual environment. This means that I not only have to figure out the dependencies but also have the deb to include a script to get the virtual environment script, then create one for the package, then download all dependencies (Pylons 0.9.7 in this case along with report lab) and finally put the package into this virtual environment. Is this all possible? Happy hacking. Krishnakant.

2 1