🌐 Mailman 3 February 2019 - Python-list - python.org

Scraping multiple web pages help
by Drake Gossi Feb. 27, 2019

Feb. 27, 2019

Hello everyone, For a research project, I need to scrape a lot of comments from regulations.gov https://www.regulations.gov/docketBrowser?rpp=25&so=DESC&sb=commentDueDate&… But partly what's throwing me is the url addresses of the comments. They aren't consistent. I mean, there's some consistency insofar as the numbers that differentiate the pages all begin after that 0064 number in the url listed above. But the differnetiating numbers aren't even all the same amount of numbers. Some are 4 (say, 4019) whereas others are 5 (say, 50343). But I dont think they go over 5. So this is a problem. I dont know how to write the code to access the multiple pages. I should also mention I'm new to programing, so that's also a problem (if you cant already tell by the way I'm describing my problem). I should also mention that, I think, there's an API on regulations.gov, but I'm such a beginner that I dont evem really know where to find it, or even what to do with it once I do. That's how helpless am right now. Any help anyone could offer would be much appreciated. D

3 2

Re: Quirk difference between classes and functions
by jfong＠ms4.hinet.net Feb. 27, 2019

Feb. 27, 2019

Chris Angelico於 2019年2月27日星期三 UTC+8上午11時29分04秒寫道： > On Wed, Feb 27, 2019 at 2:21 PM <jfong(a)ms4.hinet.net> wrote: > > > > Chris Angelico於 2019年2月27日星期三 UTC+8上午9時25分11秒寫道： > > > On Wed, Feb 27, 2019 at 12:21 PM <jfong(a)ms4.hinet.net> wrote: > > > > > > > > Gregory Ewing at 2019/2/27 AM 5:26:49 wrote: > > > > > Thomas Jollans wrote: > > > > > > I imagine there's a justification for the difference in behaviour to do > > > > > > with the fact that the body of a class is only ever executed once, while > > > > > > the body of a function is executed multiple times. > > > > > > > > > > I suspect there isn't any deep reason for it, rather it's just > > > > > something that fell out of the implementation, in particular > > > > > the decision to optimise local variable access in functions > > > > > but not other scopes. > > > > > > > > > > When compiling a function, the compiler needs to know which > > > > > variables are local so that it can allocate slots for them in > > > > > the stack frame. But when executing a class body, the locals > > > > > are kept in a dict and are looked up dynamically. > > > > > > > > If the compiler can do any decision on the variable's name, when it goes to line of print, how it handle it? > > > > > > > > x = 0 > > > > def f(): > > > > print(x) > > > > > > > > def g(): > > > > print(x) > > > > x = 1 > > > > print(y) > > > > > > > > > > Not sure what you mean by "decision", but the definition is that since > > > x is assigned to in g(), it is local to g(). It's not local to f(), so > > > the global is visible. > > > > So, may I say that the Python compiler is a multi-pass one? > > > > At this point, you're veering away from "Python-the-language" and > towards "CPython-the-implementation" (or whichever other > implementation you want to explore). Yes, it's a little away:-) I was curious about how the compiled print(x) can be influenced by x = 1 after it. Anyway, thank you for your confirmation. --Jach > But I would say that, yes, most > or all Python implementations will use multi-pass compilation. It's > the easiest way to guarantee correct behaviour. For instance, when you > reach a "try" statement, you don't know whether there'll be an > "except" block after it (there might just be a "finally"); the > bytecode that CPython produces is different for setting up a > finally-only block than for setting up a try/except. Easiest to find > out that sort of thing by running multiple passes. > > Plus, depending on how you define "pass", there are additional steps > such as constant folding and peephole optimization that can be done > after everything else. So, yeah, it's highly unlikely that there are > any performant implementations of Python that run a single compilation > pass. > > ChrisA

1 0

Re: Quirk difference between classes and functions
by jfong＠ms4.hinet.net Feb. 27, 2019

Feb. 27, 2019

Chris Angelico於 2019年2月27日星期三 UTC+8上午9時25分11秒寫道： > On Wed, Feb 27, 2019 at 12:21 PM <jfong(a)ms4.hinet.net> wrote: > > > > Gregory Ewing at 2019/2/27 AM 5:26:49 wrote: > > > Thomas Jollans wrote: > > > > I imagine there's a justification for the difference in behaviour to do > > > > with the fact that the body of a class is only ever executed once, while > > > > the body of a function is executed multiple times. > > > > > > I suspect there isn't any deep reason for it, rather it's just > > > something that fell out of the implementation, in particular > > > the decision to optimise local variable access in functions > > > but not other scopes. > > > > > > When compiling a function, the compiler needs to know which > > > variables are local so that it can allocate slots for them in > > > the stack frame. But when executing a class body, the locals > > > are kept in a dict and are looked up dynamically. > > > > If the compiler can do any decision on the variable's name, when it goes to line of print, how it handle it? > > > > x = 0 > > def f(): > > print(x) > > > > def g(): > > print(x) > > x = 1 > > print(y) > > > > Not sure what you mean by "decision", but the definition is that since > x is assigned to in g(), it is local to g(). It's not local to f(), so > the global is visible. So, may I say that the Python compiler is a multi-pass one? --Jach > > (Incidentally, "print" is handled exactly the same way. Since neither > function assigns to print, it's not local, so the built-in is found.) > > ChrisA

2 1

Re: Quirk difference between classes and functions
by Gregory Ewing Feb. 27, 2019

Feb. 27, 2019

Thomas Jollans wrote: > I imagine there's a justification for the difference in behaviour to do > with the fact that the body of a class is only ever executed once, while > the body of a function is executed multiple times. I suspect there isn't any deep reason for it, rather it's just something that fell out of the implementation, in particular the decision to optimise local variable access in functions but not other scopes. When compiling a function, the compiler needs to know which variables are local so that it can allocate slots for them in the stack frame. But when executing a class body, the locals are kept in a dict and are looked up dynamically. The compiler *could* be made to treat class bodies the same way as functions in this regard, but it would be extra work for little or no benefit. Most code in class bodies just defines new names without referring to anything else in the same scope. -- Greg

5 4

missing 1 required positional argument error
by vergos.nikolas＠gmail.com Feb. 27, 2019

Feb. 27, 2019

I'm receiving the following error: Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/bottle.py", line 862, in _handle return route.call(**args) File "/usr/lib64/python3.6/site-packages/bottle.py", line 1740, in wrapper rv = callback(*a, **ka) File "/usr/lib64/python3.6/site-packages/bottle.py", line 2690, in wrapper return func(*a, **ka) TypeError: listall() missing 1 required positional argument: 'pymydb' and this is my route definition: @app.route( '/' ) @auth_basic(counters.is_authenticated_user) def listall( pymydb ): The way i understand this error is that i'am trying to call 'listall()' without giving it an argument while in its definition i do have 'pymydb' as a parameter. BUT from inside my script i do NOT call listall at all, so how can it miss an argument?

3 2

Trying to read in data for a file to a python application
by Paul Sutton Feb. 26, 2019

Feb. 26, 2019

Hi I have been trying to write a small application that is essentially user information application. https://raw.githubusercontent.com/zleap/AboutMe/master/Aboutme.py So far I have managed to write the data generated to a file, what I want to do now, is read this data back in when the user opens the program. Trying to use https://stackoverflow.com/questions/3277503/how-to-read-a-file-line-by-line… with open(fname) as f: content = f.readlines() # you may also want to remove whitespace characters like `\n` at the end of each line content = [x.strip() for x in content] Trying to use this as a guide (hence my code is a little different). But am really stuck So far the GUI window appears but no widgets. So something is clearly getting stalled. To begin with If I can read the text in, and just display in the console this is a start, I can then see if I can figure out how to take that and insert the data in to the right places. Can anyone help please. Thanks Paul

2 2

Re: How to format a datetime MySQL database field to local using strftime()
by vergos.nikolas＠gmail.com Feb. 26, 2019

Feb. 26, 2019

Can you help me rewrite this function, which when iam using 'pymysql' conncector works normally, it does not when iam using 'bottle_pymysql' def coalesce( data ): newdata = [] seen = {} for host, ref, location, useros, browser, visits, hits, downloads, authuser in data: # Here i have to decide how to group the rows together # I want an html row for every unique combination of (host) and that hits should be summed together key = host if key not in seen: newdata.append( [ [host], [ref], location, useros, browser, [visits], hits, [downloads], authuser ] ) seen[key] = len( newdata ) - 1 # Save index (for 'newdata') of this row else: # This row is a duplicate row with a different referrer & visit datetime & torrent download rowindex = seen[key] newdata[rowindex][0].append( host ) newdata[rowindex][1].append( ref ) newdata[rowindex][5].append( visits ) newdata[rowindex][6] += hits newdata[rowindex][7].append( downloads ) return newdata

1 2

Re: missing 1 required positional argument error
by vergos.nikolas＠gmail.com Feb. 26, 2019

Feb. 26, 2019

Τη Τρίτη, 26 Φεβρουαρίου 2019 - 5:21:14 μ.μ. UTC+2, ο χρήστης Calvin Spealman έγραψε: > you call it by visiting the route you've mapped to it, so when you run this > app and navigate to the root / URL it gets called. What do you expect this > parameter to be and where did you expect it to come from when you wrote the > listall() function? > > On Tue, Feb 26, 2019 at 10:15 AM <vergos.nikolas(a)gmail.com> wrote: > > > I'm receiving the following error: > > > > Traceback (most recent call last): > > File "/usr/lib64/python3.6/site-packages/bottle.py", line 862, in _handle > > return route.call(**args) > > File "/usr/lib64/python3.6/site-packages/bottle.py", line 1740, in > > wrapper > > rv = callback(*a, **ka) > > File "/usr/lib64/python3.6/site-packages/bottle.py", line 2690, in > > wrapper > > return func(*a, **ka) > > TypeError: listall() missing 1 required positional argument: 'pymydb' > > > > and this is my route definition: > > > > @app.route( '/' ) > > @auth_basic(counters.is_authenticated_user) > > def listall( pymydb ): > > > > The way i understand this error is that i'am trying to call 'listall()' > > without giving it an argument while in its definition i do have 'pymydb' as > > a parameter. > > > > BUT from inside my script i do NOT call listall at all, so how can it miss > > an argument? I expect the parameter to be pass from the top of my script where i have this line: # dbhost is optional, default is localhost plugin = bottle_pymysql.Plugin( dbuser='nikos', dbpass='*****', dbname='clientele', dictrows=False ) app.install(plugin) It works as with another script of mine does.

1 0

Re: missing 1 required positional argument error
by vergos.nikolas＠gmail.com Feb. 26, 2019

Feb. 26, 2019

Τη Τρίτη, 26 Φεβρουαρίου 2019 - 5:21:14 μ.μ. UTC+2, ο χρήστης Calvin Spealman έγραψε: > you call it by visiting the route you've mapped to it, so when you run this > app and navigate to the root / URL it gets called. What do you expect this > parameter to be and where did you expect it to come from when you wrote the > listall() function? > > On Tue, Feb 26, 2019 at 10:15 AM <vergos.nikolas(a)gmail.com> wrote: > > > I'm receiving the following error: > > > > Traceback (most recent call last): > > File "/usr/lib64/python3.6/site-packages/bottle.py", line 862, in _handle > > return route.call(**args) > > File "/usr/lib64/python3.6/site-packages/bottle.py", line 1740, in > > wrapper > > rv = callback(*a, **ka) > > File "/usr/lib64/python3.6/site-packages/bottle.py", line 2690, in > > wrapper > > return func(*a, **ka) > > TypeError: listall() missing 1 required positional argument: 'pymydb' > > > > and this is my route definition: > > > > @app.route( '/' ) > > @auth_basic(counters.is_authenticated_user) > > def listall( pymydb ): > > > > The way i understand this error is that i'am trying to call 'listall()' > > without giving it an argument while in its definition i do have 'pymydb' as > > a parameter. > > > > BUT from inside my script i do NOT call listall at all, so how can it miss > > an argument? Yes, '/' route is being called when i visir it with my browser. Ok, but that function needs to work with the 'pymydb' 'bottle-pymysql' keyword so it must have it as a parameter. In another script of mine i have mapped '/' to def index( pymydb ) and its working without error. How come it works there and not work here? They are both '/' routes.

1 0

Re: Quirk difference between classes and functions
by Gregory Ewing Feb. 26, 2019

Feb. 26, 2019

Chris Angelico wrote: > Classes and functions behave differently. Inside a function, a name is > local if it's ever assigned to; but in a class, this is not the case. Actually, it is. Assigning to a name in a class body makes it part of the class namespace, which is the local namespace at the time the class body is executed. The unusual thing about a class namespace is that it doesn't form part of the scope of closures created within the class. So methods, for example, can't directy see attributes of the class they're defined in. -- Greg

2 1