🌐 Mailman 3 August 2016 - Python-list - python.org

Recursive csv import functions for Pandas
by andrea.botti＠gmail.com Aug. 2, 2016

Aug. 2, 2016

I have put together the following code: ## DEFINES INPUT FILES inputcsvT = ['./input_csv/A08_KI_T*.csv', './input_csv/A08_LR_T*.csv', './input_csv/A08_B1_T*.csv',] #'./input_csv/A10_KI_T*.csv', './input_csv/A10_LR_T*.csv', './input_csv/A10_B1_T*.csv', #'./input_csv/A11_KI_T*.csv', './input_csv/A11_LR_T*.csv', './input_csv/A11_B1_T*.csv', #'./input_csv/A16_KI_T*.csv', './input_csv/A16_LR_T*.csv', './input_csv/A16_B1_T*.csv'] inputcsvR = ['./input_csv/A08_KI_R*.csv', './input_csv/A08_LR_R*.csv', './input_csv/A08_B1_R*.csv',] ## DEFINES FUNCTIONS FOR CSV INPUT, MERGE AND HOURLY RESAMPLING def csv_import_merge_T(f): dfsT = [pd.read_csv(fp, index_col=[0], parse_dates=[0], dayfirst=True, names=['datetime','temp','rh'], header=0) for fp in files] dfT = pd.concat(dfsT) dfT = dfT.drop('rh', 1) #dfT[~dfT.index.duplicated()] # replaced with function below dfT_clean = dfT.reset_index().drop_duplicates('datetime').set_index('datetime') dfTH = dfT_clean.resample('H').bfill() return dfTH def csv_import_merge_R(f): dfsR = [pd.read_csv(fp, index_col=[0], parse_dates=[0], dayfirst=True, names=['datetime','rad'], header=0) for fp in files] dfR = pd.concat(dfsR) #dfR[~dfR.index.duplicated()] # replaced with function below dfR_clean = dfR.reset_index().drop_duplicates('datetime').set_index('datetime') dfRH = dfR_clean.resample('H').mean() return dfRH ## PERFORMS FUNCTIONS FOR ALL Ts AND Rs AND CALCULATES HEATING DEGREE HOURS (HDH) AS R-T WHEN XX_XX_R = XX_XX_T for csvnameT in inputcsvT: files = glob.glob(csvnameT) print ('___'); print (files) csvT = csvnameT[12:20] print csvT t = csv_import_merge_T(files) t.to_csv('./output_csv/'+ csvT + '.csv') for csvnameR in inputcsvR: files = glob.glob(csvnameR) print ('___'); print (files) csvR = csvnameR[12:20] r = csv_import_merge_R(files) print csvR while csvnameR[12:18] == csvnameT[12:18]: print csvR + "=" + csvT r.to_csv('./output_csv/'+ csvR +'.csv') hdh = r.sub(t,axis=0).dropna() hdh[hdh['temp']<=1] = 0 csvHDH = csvnameT[12:18] hdh_week = hdh.temp.resample('W-MON').sum().round(decimals=0) hdh_week.to_csv('./output_csv/HDH/' + csvHDH + '_HDH.csv') The sequence of action should be: for A08_KI - import A08_KI_T1,T2,...Tn, merge them into A08_KI_T and resample them by the hour (room temp.) - import A08_KI_R1,R2,...Rn, merge them into A08_KI_R and resample them by the hour (radiator temp.) - every time XX_XX_R = XX_XX_T (i.e. A08_KI_T and A08_KI_R, or A10_LR_T and A10_LR_R) calculate HDH = R - T. The code I developed has something wrong and the command: while csvnameR[12:18] == csvnameT[12:18]: seems to cause the loop to go on for ever.

2 1

Installing Python 3 on Windows
by Uri Even-Chen Aug. 2, 2016

Aug. 2, 2016

Hi, I want to install Python 3 on Windows, but I also need Python 2 for Google App Engine SDK. When I type a name of a Python file in command line, I want it to run with Python 3. However, I checked with "print 3/5" and it printed 0 - Python 2. I have the latest versions of Python installed - python-2.7.12.msi and python-3.5.2.exe >python --version Python 2.7.12 >assoc .py .py=Python.File >ftype Python.File Python.File="C:\Python27\python.exe" "%1" %* By the way, it there a way to print the Python version from Python? Thanks, Uri. *Uri Even-Chen* [image: photo] Phone: +972-54-3995700 Email: uri(a)speedy.net Website: http://www.speedysoftware.com/uri/en/ <http://www.facebook.com/urievenchen> <http://plus.google.com/+urievenchen> <http://www.linkedin.com/in/urievenchen> <http://twitter.com/urievenchen>

2 2

Re: Installing Python 3 on Windows
by Edward Diener Aug. 2, 2016

Aug. 2, 2016

On 8/2/2016 8:11 AM, Uri Even-Chen wrote: > Hi, > > I want to install Python 3 on Windows, but I also need Python 2 for Google > App Engine SDK. When I type a name of a Python file in command line, I want > it to run with Python 3. However, I checked with "print 3/5" and it printed > 0 - Python 2. I have the latest versions of Python installed > - python-2.7.12.msi and python-3.5.2.exe > >> python --version > Python 2.7.12 > >> assoc .py > .py=Python.File > >> ftype Python.File > Python.File="C:\Python27\python.exe" "%1" %* > > By the way, it there a way to print the Python version from Python? Use the Python Launcher for Windows.

2 2

Re: Why not allow empty code blocks?
by Rustom Mody Aug. 2, 2016

Aug. 2, 2016

On Saturday, July 30, 2016 at 4:56:01 PM UTC+5:30, Chris Angelico wrote: > On Sat, Jul 30, 2016 at 8:15 PM, BartC wrote: > > Anyway, if you're going to talk about annoying things forced upon you by the > > language, what about: > > > > "()" in "print (x)" for Python 3 > > Why are you singling out print? It's just a function like any other. > Are you complaining about the way function calls need parentheses? Its a function… ok. Its ‘just’ a function… Arguable For example: - Prior Art: Its builtin and special in Fortran, Pascal, Basic - More immediate : It was a special in python2 - Poorer error catching: What was a straight syntax error is now a lint-catch (at best) [print (x) for x in range(20)]

6 10

Re: Why not allow empty code blocks?
by Steven D'Aprano Aug. 2, 2016

Aug. 2, 2016

On Sat, 30 Jul 2016 04:32 am, Antoon Pardon wrote: > Op 29-07-16 om 16:38 schreef Steven D'Aprano: >> On Fri, 29 Jul 2016 11:55 pm, Antoon Pardon wrote: >> >>> Op 29-07-16 om 15:43 schreef Steven D'Aprano: >>>> Of course it won't, which is why I don't believe all these folks who >>>> claim that they regularly ("all the time", "fairly often") replace >>>> except blocks with `pass`. I call shenanigans -- perhaps you do it >>>> *occasionally*, but as a general rule, you can rarely replace the >>>> exception handler with a do-nothing clause and expect your code to >>>> work: >>>> >>>> try: >>>> block >>>> except SomeException: >>>> handle error >>>> process >>>> >>>> If you remove the error handling block, replacing it with `pass`, in >>>> general your code will just break again as soon as it continues >>>> processing. >>> >>> I think the case where you just want to ignore the exception, but it can >>> at times be useful to get some extra logging information for debuging >>> purposes, is not that rare as you seem to suggest. >> >> I think I can count the number of times I've wanted to ignore an >> exception on the fingers of one hand. And when I did, there was no need >> to debug it, because I only ignore expected failures. There's nothing to >> debug. > > You really should stop, generalising your own situation. I'm not just generalising from code I've written. I'm generalising from code I've reviewed, code I've read, code I've seen others admire, code I've seen others mock and laugh at, and best practices I've read about. That's called "learning to be a better programmer". >> But even if you do it all the time, how do you "get some extra logging >> information" by replacing the "except" block with `pass`? If you replaced >> the "except" block with a call to the logger, that would be one thing. >> But "pass", that won't do the job. > > You don't seem to understand. There is no real except block to be replaced > by pass. There is sometime an except block added (or uncommented) to get > logging information. Perhaps its *you* who doesn't understand. The subject line talks about "empty code blocks", and Bart has suggested that instead of having to write try: something except SomeException: pass process (which he considers far too difficult) we should allow empty code blocks: try: something except SomeException: process because apparently saving that one indented line with "pass" makes all the difference in language usability. That lead to Chris asking how common it was to do this. That's when Bart said that he frequently *replaced* an existing except block with "pass": # before try: something except SomeException: handle error handle it some more process # after try: something except SomeException: pass process At which point *you agreed with him*: [quote] As BartC already mentions it happens fairly often during debugging. Something like. try: Some code except Some_Exception: # Commented code for when I am debugging <Some code> pass [end quote] So now you're changing your story, and claiming that you're not actually replacing an existing except block with pass (because I think any experienced Python developer, which Bart is not, realises that's not likely to work). Now you've got a use-case where we start with: # before some code more processing and add an entire try...except around it: # step 1 try: some code except SomeException: debugging code more processing *and then* comment out the debugging code and replace it with pass: # after, as stated by you try: some code except SomeException: # debugging code pass more processing Of course it's legitimate to wrap some failing code in a temporary try...except block for debugging purposes. But that's not an empty code block, and its not "pass", and it's not relevant to Chris' question. And what happens once you have finished debugging the code? According to your post, you comment out the debugging code, replacing it by "pass". And you do this *frequently*. Okay, if you say you do that, I have to believe you. If we take you at your word, then we can only conclude that your code is littered with the detritus of old debugging code and unnecessary try...except blocks which do nothing. That's practically the definition of *poor programming*, leaving old code commented out to rot, but if you say that's what you do, who am I to argue that you don't? Since I don't actually believe you are that poor a programmer, I suspect that what you actually do is not replace the debugging code with "pass", like you claim, but that once you have finished debugging that section of the code you remove the unneeded try...except block and dead, unused debugging code, returning it to the "before" state. D'Arcy's suggestion of leaving the debugging code in with a runtime switch to turn it on and off also makes sense. I'm partial to switching on the special __debug__ global. But we're getting further and further away from Bart's scenario, of a bare "pass". *That's* what I don't believe in: this supposed scenario where people have existing code blocks that they have to replace with "pass", and that happens *frequently enough to matter*. And of course as D'Arcy has pointed out, these two are not the same: # before some code # after try: some code except SomeException: pass To make the second one equivalent to the first, one should use *raise*, not pass. In which case you could even leave your debugging code in there, uncommented out: try: some code except SomeException: raise debugging code all the better to confuse those who have to maintain the program in your absence. -- Steven “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse.

2 1

Re: Python text file fetch specific part of line
by honeygne＠gmail.com Aug. 2, 2016

Aug. 2, 2016

On Thursday, July 28, 2016 at 1:00:17 PM UTC+5:30, c...(a)zip.com.au wrote: > On 27Jul2016 22:12, Arshpreet Singh <arsh840(a)gmail.com> wrote: > >I am writing Imdb scrapper, and getting available list of titles from IMDB > >website which provide txt file in very raw format, Here is the one part of > >file(http://pastebin.com/fpMgBAjc) as the file provides tags like Distribution > >Votes,Rank,Title I want to parse title names, I tried with readlines() method > >but it returns only list which is quite heterogeneous, is it possible that I > >can parse each value comes under title section? > > Just for etiquette: please just post text snippets like that inline in your > text. Some people don't like fetching random URLs, and some of us are not > always online when reading and replying to email. Either way, having the text > in the message, especially when it is small, is preferable. > > To your question: > > Your sample text looks like this: > > New Distribution Votes Rank Title > 0000000125 1680661 9.2 The Shawshank Redemption (1994) > 0000000125 1149871 9.2 The Godfather (1972) > 0000000124 786433 9.0 The Godfather: Part II (1974) > 0000000124 1665643 8.9 The Dark Knight (2008) > 0000000133 860145 8.9 Schindler's List (1993) > 0000000133 444718 8.9 12 Angry Men (1957) > 0000000123 1317267 8.9 Pulp Fiction (1994) > 0000000124 1209275 8.9 The Lord of the Rings: The Return of the King > (2003) > 0000000123 500803 8.9 Il buono, il brutto, il cattivo (1966) > 0000000133 1339500 8.8 Fight Club (1999) > 0000000123 1232468 8.8 The Lord of the Rings: The Fellowship of the > Ring (2001) > 0000000223 832726 8.7 Star Wars: Episode V - The Empire Strikes Back > (1980) > 0000000233 1243066 8.7 Forrest Gump (1994) > 0000000123 1459168 8.7 Inception (2010) > 0000000223 1094504 8.7 The Lord of the Rings: The Two Towers (2002) > 0000000232 676479 8.7 One Flew Over the Cuckoo's Nest (1975) > 0000000232 724590 8.7 Goodfellas (1990) > 0000000233 1211152 8.7 The Matrix (1999) > > Firstly, I would suggest you not use readlines(), it pulls all the text into > memory. For small text like this is it ok, but some things can be arbitrarily > large, so it is something to avoid if convenient. Normally you can just iterate > over a file and get lines. > > You want "text under the Title." Looking at it, I would be inclined to say that > the first line is a header and the rest consist of 4 columns: a number > (distribution?), a vote count, a rank and the rest (title plus year). > > You can parse data like that like this (untested): > > # presumes `fp` is reading from the text > for n, line in enumerate(fp): > if n == 0: > # heading, skip it > continue > distnum, nvotes, rank, etc = split(line, 3) > ... do stuff with the various fields ... > > I hope that gets you going. If not, return with what code you have, what > happened, and what you actually wanted to happen and we may help further. Thanks I am able to do it with following: https://github.com/alberanid/imdbpy/blob/master/bin/imdbpy2sql.py (it was very helpful) python imdbpy2sql.py -d <.txt files downloaded from IMDB> -u sqlite:/where/to/save/db --sqlite-transactions

1 0

Re: `exec`-based routine crashes app upon migration from 3.4.3 to python 3.5.2.
by Lawrence D’Oliveiro Aug. 2, 2016

Aug. 2, 2016

On Friday, July 29, 2016 at 6:25:51 AM UTC+12, Enjoys Math wrote: > exec('obj = ' + objType + '(self)', None, _locals) > obj = _locals['obj'] Why? Why not just obj = objType(self) ?

2 1

usage of functools.partial in in parallelism
by Sivan Greenberg Aug. 2, 2016

Aug. 2, 2016

Hi all, I'm wondering about the use of partial in writing parallel code. Is is it quicker than re-evaluating arguments for a multiple session.get()'s method with different , for example (of requests) ? Or maybe it is used to make sure the arguments differ per each invocation ? (the parallel invocation is supposedly using tasks / co-routine support in Python 3. I can't publish the code I spotted that in. What are ups and downs of using them when are they in context? -Sivan

4 4

python and open office
by Crane Ugly Aug. 2, 2016

Aug. 2, 2016

I try to create some scripts that will help me to open and manipulate OpenOffice documents. Calc in particular. But I have some problems finding right packages or libraries that offer such interface. So far I was trying uno and unotools but the first step is to import them failed. Here is the output: UNO tools are installed: $ pip list | grep uno uno (0.3.3) unotools (0.3.3) Try to import them: $ python Python 2.7.12 (default, Jun 29 2016, 12:53:15) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import uno >>> import unotools Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Volumes/home/lshanin/Dropbox/Python/ve/accounting/lib/python2.7/site-packages/unotools/__init__.py", line 16 def __init__(self, name: str): ^ SyntaxError: invalid syntax >>> I would appreciate is somebody help me to find what is wrong with untools package. Or point me to some other available libraries. I expect to work with OpenOffice (LibreOffice) files only not with MS Excel files. Regards Leonid

5 4

Re: use import *
by Lawrence D’Oliveiro Aug. 2, 2016

Aug. 2, 2016

On Tuesday, August 2, 2016 at 3:32:11 AM UTC+12, Ganesh Pal wrote: > is it a good programming practice to use import * ? No. Think of the module you’re importing as a can, and each of the names it defines as a worm. So you when you do “from «module» import *”, it’s like emptying all the worms in the can onto the floor, and letting them crawl all over your program, polluting your global namespace. Now imagine you change your mind, and want to get all the worms back into the can, because people reading your program (yourself included) cannot be sure where each name is coming from. Not a pleasant process. So the wise programmer avoids getting into this situation. Yes, it can be annoying to have to write module names over and over. Python lets you use alternative names, e.g. import long_module_name as lmn then you say “lmn.«name»” instead of “long_module_name.«name»”.

1 0