🌐 Mailman 3 August 2020 - Python-list - python.org

Re: Python 3 how to convert a list of bytes objects to a list of strings?
by Chris Green Aug. 28, 2020

Aug. 28, 2020

Richard Damon <Richard(a)damon-family.org> wrote: > On 8/28/20 7:50 AM, Karsten Hilbert wrote: > >>> No interpreation requires, since parsing failed. Then you can start > >>> dealing with these exceptions. _Do not_ write unparsable messages into > >>> an mbox! > >>> > >> Maybe I shouldn't but Python 2 has been managing to do so for several > >> years without any issues. > > I am inclined to congratulate you on that sheer amount of luck. I don't > > believe there were no issues because everything worked just right under > > py2 but rather because py2 cared less than py3 does now. > > > >> Are we saying that Python 3 really can't be made to handle things > >> 'tolerantly' like Python 2 used to? > > It sure should be possible but it will require *explicit* en/decode()s in > > more places than before because AFAICT there's less impliciteness as to > > which encoding to apply (regardless of whether it applies). > > > > Karsten > > > > > > > This might be one of the cases where Python 2's lack handling of string > vs bytes was an advantage. > > If he was just scanning the message for specific ASCII strings, then not > getting the full message decoded write is unlikely to have been causing > problems. > > Python2 handled that sort of case quite easily. Python 3 on the other > hand, will have issue converting the byte message to a string, since > there isn't a single encoding that you could use for all of it all the > time. This being 'fussier' does make sure that the program is handling > all the text 'properly', and would be helpful if some of the patterns > being checked for contained 'extended' (non-ASCII) characters. > > One possible solution in Python3 is to decode the byte string using an > encoding that allows all 256 byte values, so it won't raise any encoding > errors, just give your possibly non-sense characters for non-ASCII text. > But this will simply get some things quite wrong and produce garbage won't it? Whereas Python 2 would simply scramble the odd character. -- Chris Green ·

3 2

Re: Python 3 how to convert a list of bytes objects to a list of strings?
by Chris Green Aug. 28, 2020

Aug. 28, 2020

Stefan Ram <ram(a)zedat.fu-berlin.de> wrote: > Chris Angelico <rosuav(a)gmail.com> writes: > >But this is a really good job for a list comprehension: > >sss = [str(word) for word in bbb] > > Are you all sure that "str" is really what you all want? > Not absolutely, you no doubt have been following other threads related to this one. :-) > |>>> b = b"b" > |>>> str( b ) > |"b'b'" > > Maybe try to /decode/ the bytes? > > |>>> b.decode( "ASCII" ) > |'b' > > Therein lies the problem, the incoming byte stream *isn't* ASCII, it's an E-Mail message which may, for example, have UTF-8 or other encoded characters in it. Hopefully it will have an encoding given in the header but that's only if the sender is 'well behaved', one needs to be able to handle almost anything and it must be done without 'manual' interaction. -- Chris Green ·

2 1

Re: Python 3 how to convert a list of bytes objects to a list of strings?
by Chris Green Aug. 28, 2020

Aug. 28, 2020

Cameron Simpson <cs(a)cskk.id.au> wrote: > On 27Aug2020 23:54, Marco Sulla <Marco.Sulla.Python(a)gmail.com> wrote: > >Are you sure you want `str()`? > > > >>>> str(b'aaa') > >"b'aaa'" > > > >Probably you want: > > > >map(lambda x: x.decode(), bbb) > > _And_ you need to know the encoding of the text in the bytes. The above > _assumes_ UTF-8 because that is the default for bytes.decode, and if > that is _not_ what is in the bytes objects you will get mojibake. > > Because a lot of stuff is "mostly ASCII", this is the kind of bug which > can lurk until much later when you have less usual data. > If there's an encoding given in the header of the incoming E-Mail then one (hopefully) knows what the encoding is. However you have to be able to handle the more general case where either the encoding isn't given or it's wrong. In the real world E-Mail survives having an incorrect encoding in the header, what you see is either missing or garbled characters with the remainder being OK. Garbling the whole lot isn't a good approach. -- Chris Green ·

1 0

Re: Python 3 how to convert a list of bytes objects to a list of strings?
by Chris Green Aug. 28, 2020

Aug. 28, 2020

Chris Angelico <rosuav(a)gmail.com> wrote: > On Fri, Aug 28, 2020 at 6:36 AM Chris Green <cl(a)isbd.net> wrote: > > > > This sounds quite an easy thing to do but I can't find how to do it > > elegantly. > > > > I have a list of bytes class objects (i.e. a list containing sequences > > of bytes, which are basically text) and I want to convert it to a list > > of string objects. > > > > One of the difficulties of finding out how to do this is that 'list of > > bytes' tends to mean a bytes object with a sequence of bytes in it > > which is *not* what I'm after converting. :-) > > > > Obviously I can do:- > > > > bbb = [b'aaa', b'bbb', b'ccc'] > > sss = [] > > for i in range(0, 2): > > sss.append(str(bbb[i]) > > > > but that does seem a bit clumsy. Is there a better way? > > > > Firstly, you shouldn't iterate over the range, but over the items themselves: > > for word in bbb: > Yes, of course, it was just a 'quick and dirty' to make the point. > But this is a really good job for a list comprehension: > > sss = [str(word) for word in bbb] > That seems the way to do it, thanks. -- Chris Green ·

1 0

Re: How do I do this in Python 3 (string.join())?
by Chris Green Aug. 28, 2020

Aug. 28, 2020

Cameron Simpson <cs(a)cskk.id.au> wrote: [snip] > > >The POP3 processing is solely to collect E-Mail that ends up in the > >'catchall' mailbox on my hosting provider. It empties the POP3 > >catchall mailbox, checks for anything that *might* be for me or other > >family members then just deletes the rest. > > Very strong email policy, that one. Personally I fear data loss, and > process everything; anything which doesn't match a rule lands in my > "UNKNOWN" mail folder for manual consideration when I'm bored. It is > largely spam, but sometimes has a message wanting a new filing rule. > It's not *that* strong, the catchall is for *anything* that is addressed to either of the two domains hosted there. I.e. mail for xhghjhwd(a)isbd.net will arrive in the catchall mailbox. So I just search the To: address for anything that might be a typo for one of our names or anything else that might be of interest. I have an associated configuration file that specifies the patterns to look for so I can change things on the fly as it were. One of the scripts that I'm having trouble converting to Python 3 is the one that does this catchall management. > >> >E.g. in this case the only (well the only ready made) way to get a > >> >POP3 message is using poplib and this just gives you a list of lines > >> >made up of "bytes as text" :- > >> > > >> > popmsg = pop3.retr(i+1) > >> > >> Ok, so you have bytes? You need to know. > >> > >The documentation says (and it's exactly the same for Python 2 and > >Python 3):- > > > > POP3.retr(which) > > Retrieve whole message number which, and set its seen flag. Result > > is in form (response, ['line', ...], octets). > > > >Which isn't amazingly explicit unless 'line' implies a string. > > Aye. But "print(repr(a_pop_line))" will tell you. Almost certainly a > string-of-bytes, so I would expect bytes. The docs are probably > unchanged during the Python2->3 move. > Yes, I added some print statments to my catchall script to find out and, yes, the returned value is a list of 'byte strings'. It's a pity there isn't a less ambiguous name for 'string-of-bytes'! :-) > >> >I join the lines to feed them into mailbox.mbox() to create a mbox I > >> >can analyse and also a message which can be sent using SMTP. > > Ah. I like Maildirs for analysis; every message has its own file, which > makes adding and removing messages easy, and avoids contention with > other things using the Maildir. > > My mailfiler can process Maildirs (scan, add, remove) and add to > Maildirs and mboxes. > I've switched to maildir several times in the past and have always switched back because they have so many 'standards'. I use mutt as my MUA and that does handle maildir as well as anything but still doesn't do it for me. :-) -- Chris Green ·

1 0

Output showing "None" in Terminal
by Py Noob Aug. 27, 2020

Aug. 27, 2020

Hi! i'm new to python and would like some help with something i was working on from a tutorial. I'm using VScode with 3.7.0 version on Windows 7. Below is my code and the terminal is showing the word "None" everytime I execute my code. Many thanks! print("Conversion") def km_mi(): return answer selection = input("Type mi for miles or km for kilometers: ") if selection == "mi": n = int(input(print("Please enter distance in miles: "))) answer = (1.6*n) print("%.2f" % answer, "miles") else: n = float(input(print("Please enter distance in kilometers: "))) answer = (n/1.6) print("%.2f" % answer, "kilometers") answer = km_mi

7 8

Python 3 how to convert a list of bytes objects to a list of strings?
by Chris Green Aug. 27, 2020

Aug. 27, 2020

This sounds quite an easy thing to do but I can't find how to do it elegantly. I have a list of bytes class objects (i.e. a list containing sequences of bytes, which are basically text) and I want to convert it to a list of string objects. One of the difficulties of finding out how to do this is that 'list of bytes' tends to mean a bytes object with a sequence of bytes in it which is *not* what I'm after converting. :-) Obviously I can do:- bbb = [b'aaa', b'bbb', b'ccc'] sss = [] for i in range(0, 2): sss.append(str(bbb[i]) but that does seem a bit clumsy. Is there a better way? -- Chris Green ·

6 6

Proposal: Ignore all empty entries within $PYTHONPATH
by David A. Wheeler Aug. 27, 2020

Aug. 27, 2020

All: I propose that all empty entries with $PYTHONPATH be ignored. This would eliminate some accidents that can lead to security problems. I thought it'd be easiest to explain by drafting a PEP, so please see this first draft below. Thanks! Comments welcome. --- David A. Wheeler ============================= PEP: 9999 Title: Ignore all empty entries within ``$PYTHONPATH`` Author: David A. Wheeler <dwheeler(a)dwheeler.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Aug-2020 Python-Version: 3.7.10 Post-History: 27-Aug-2020 Abstract ======== Currently, common ways to set ``$PYTHONPATH`` also unexpectedly add the current directory as an entry. As a result, users have an increased risk of unintentionally running malicious code. This PEP proposes that all empty entries within ``$PYTHONPATH`` be ignored, instead of being treated as the current directory. In the rare cases where the current directory is intended, "." (or even better its absolute path) can continue to be used instead. This small change eliminates an easily-made and subtle mistake that can lead to a security vulnerability. Motivation ========== The essay "Never Run 'python' In Your Downloads Folder" by Glyph `Never Run`_ points out that the way Python currently implements ``$PYTHONPATH`` easily leads to users unintentionally running malicioous code. The problem is that "most places that [recommend using] PYTHONPATH recommend adding things to it like so": ``export PYTHONPATH="/new/useful/stuff:$PYTHONPATH"`` As the essay notes, "this idiom has a critical flaw: the first time it's invoked, if ``$PYTHONPATH`` was previously either empty or un-set, this then includes an empty string, which resolves to the current directory..." As a result, someone who executed this line above will quietly execute Python libraries in whatever their current directory happens to be, and that could quickly lead to running malicious code. It could be argued that this is fine, because the shell's ``$PATH`` does the same thing. After all, the current documentation for ``$PYTHONPATH`` says that it augments "the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows)." `Cmdline`_ But this argument ignores a key difference: the shell's ``$PATH`` practically *always* has an initial non-empty value, while ``$PYTHONPATH`` typically starts with an *empty* value. This means that the same patterns that are normally safe with ``$PATH`` (because ``$PATH`` is non-empty) are *dangerous* with ``$PYTHONPATH`` `Hacker News`_. Once ``$PYTHONPATH`` is set with a dangerous value, it is likely to stay dangerous after having other values appended. A far safer approach is for Python to simply skip all empty entries within ``$PYTHONPATH``. In this situation, a ``$PYTHONPATH`` with the value "``:spam::eggs:``" would be treated the same as "``spam:eggs``". Empty entries are almost never intended. They are also widely confusing, in part because they are not obvious. Users who truly want to include the runtime current directory can use "." instead, but in almost all cases they will want the absolute path to the current directory anyway. By ignoring all empty entries, Python users will be quietly protected from this mistake, making Python just a little easier to use securely. Setting ``$PYTHONPATH`` is less common (due to virtualenvs), but it's still in use and is a useful mechanism. It'd be best to quietly interpret empty entries in ``$PYTHONPATH`` as unintentional mistakes and ignore them, instead of quietly enabling security vulnerabilities. Implementing this is trivial. I have created this PEP because this is technically a behavioral change in a long-present mechanism. References ========== .. _Never Run: https://glyph.twistedmatrix.com/2020/08/never-run-python-in-your-downloads-… by Glyph .. _Cmdline: https://docs.python.org/3/using/cmdline.html .. _Hacker News: https://news.ycombinator.com/item?id=24250418 Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

1 0

Re: How do I do this in Python 3 (string.join())?
by Chris Green Aug. 27, 2020

Aug. 27, 2020

Cameron Simpson <cs(a)cskk.id.au> wrote: > On 27Aug2020 09:16, Chris Green <cl(a)isbd.net> wrote: > >Cameron Simpson <cs(a)cskk.id.au> wrote: > >> But note: joining bytes like strings is uncommon, and may indicate > >> that > >> you should be working in strings to start with. Eg you may want to > >> convert popmsg from bytes to str and do a str.join anyway. It depends on > >> exactly what you're dealing with: are you doing text work, or are you > >> doing "binary data" work? > >> > >> I know many network protocols are "bytes-as-text, but that is > >> accomplished by implying an encoding of the text, eg as ASCII, where > >> characters all fit in single bytes/octets. > >> > >Yes, I realise that making everything a string before I start might be > >the 'right' way to do things but one is a bit limited by what the mail > >handling modules in Python provide. > > I do ok, though most of my message processing happens to messages > already landed in my "spool" Maildir by getmail. My setup uses getmail > to get messages with POP into a single Maildir, and then I process the > message files from there. > Most of my mail is delivered by SMTP, I run a Postfix SMTP *serever* on my desktop machine which stays on permanently. The POP3 processing is solely to collect E-Mail that ends up in the 'catchall' mailbox on my hosting provider. It empties the POP3 catchall mailbox, checks for anything that *might* be for me or other family members then just deletes the rest. > >E.g. in this case the only (well the only ready made) way to get a > >POP3 message is using poplib and this just gives you a list of lines > >made up of "bytes as text" :- > > > > popmsg = pop3.retr(i+1) > > Ok, so you have bytes? You need to know. > The documentation says (and it's exactly the same for Python 2 and Python 3):- POP3.retr(which) Retrieve whole message number which, and set its seen flag. Result is in form (response, ['line', ...], octets). Which isn't amazingly explicit unless 'line' implies a string. > >I join the lines to feed them into mailbox.mbox() to create a mbox I > >can analyse and also a message which can be sent using SMTP. > > > >Should I be converting to string somewhere? > > I have not used poplib, but the Python email modules have a BytesParser, > which gets you a Message object; I would feed the poplib bytes to that > to parse the received message. A Message object can then be transcribed > as text via its .as_string method. Or you can do other things with it. > > I think my main points are: > > - know whether you're using bytes (uninterpreted data) or text (strings > of _characters_); treating bytes _as_ text implies an encoding, and > when that assumption is incorrect you get mojibake[1] > > - look at the email modules' parsers, which return Messages, a > representation of the message in a structure (so that MIME subparts > etc are correctly broken out, and the character sets are _known_, post > parse) OK, thanks Cameron. -- Chris Green ·

2 1

Another 2 to 3 mail encoding problem
by Chris Green Aug. 27, 2020

Aug. 27, 2020

I'm unearthing a few issues here trying to convert my mail filter and delivery programs from 2 to 3! I have a simple[ish] local mbox mail delivery module as follows:- import mailbox import logging import logging.handlers import os import time # # # Class derived from mailbox.mbox so we can override _pre_message_hook() # to do nothing instead of appending a blank line # class mymbox(mailbox.mbox): def _pre_message_hook(self, f): """Don't write the blank line before the 'From '""" pass # # # log a message # def initLog(name): log = logging.getLogger(name) log.setLevel(logging.DEBUG) f = logging.handlers.RotatingFileHandler("/home/chris/tmp/mail.log", 'a', 1000000, 4) f.setLevel(logging.DEBUG) formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') f.setFormatter(formatter) log.addHandler(f) return log # # # Deliver a message to a local mbox # def deliverMboxMsg(dest, msg, log): # # # Create the destination mbox instance # mbx = mymbox(dest, factory=None) log.info("From: " + msg.get("From", "unknown")) log.info("Destination is: " + dest) # # # Lock the mbox while we append to it # for tries in range(3): try: mbx.lock() # # # Append the incoming message to the appropriate mbox # mbx.add(msg) # # # now set the modified time later than the access time (which is 'now') so # that mutt will see new mail in the mbox. # os.utime(dest, ((time.time()), (time.time() + 5))) mbx.unlock() break except mailbox.ExternalClashError: log.info("Destination locked, try " + str(tries)) time.sleep(1) else: # get here if we ran out of tries log.warn("Failed to lock destination after 3 attempts, giving up") return It has run faultlessly for many years under Python 2. I've now changed the calling program to Python 3 and while it handles most E-Mail OK I have just got the following error:- Traceback (most recent call last): File "/home/chris/.mutt/bin/filter.py", line 102, in <module> mailLib.deliverMboxMsg(dest, msg, log) File "/home/chris/.mutt/bin/mailLib.py", line 52, in deliverMboxMsg mbx.add(msg) File "/usr/lib/python3.8/mailbox.py", line 603, in add self._toc[self._next_key] = self._append_message(message) File "/usr/lib/python3.8/mailbox.py", line 758, in _append_message offsets = self._install_message(message) File "/usr/lib/python3.8/mailbox.py", line 830, in _install_message self._dump_message(message, self._file, self._mangle_from_) File "/usr/lib/python3.8/mailbox.py", line 215, in _dump_message gen.flatten(message) File "/usr/lib/python3.8/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib/python3.8/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib/python3.8/email/generator.py", line 214, in _dispatch meth(msg) File "/usr/lib/python3.8/email/generator.py", line 432, in _handle_text super(BytesGenerator,self)._handle_text(msg) File "/usr/lib/python3.8/email/generator.py", line 249, in _handle_text self._write_lines(payload) File "/usr/lib/python3.8/email/generator.py", line 155, in _write_lines self.write(line) File "/usr/lib/python3.8/email/generator.py", line 406, in write self._fp.write(s.encode('ascii', 'surrogateescape')) UnicodeEncodeError: 'ascii' codec can't encode character '\ufeff' in position 4: ordinal not in range(128) So what do I need to do to the message I'm adding with mbx.add(msg) to fix this? (I assume that's what I need to do). -- Chris Green ·

13 15