🌐 Mailman 3 February 2022 - Python-list - python.org

Re: Threading question .. am I doing this right?
by Robert Latest Feb. 25, 2022

Feb. 25, 2022

Chris Angelico wrote: > Depending on your database, this might be counter-productive. A > PostgreSQL database running on localhost, for instance, has its own > caching, and data transfers between two apps running on the same > computer can be pretty fast. The complexity you add in order to do > your own caching might be giving you negligible benefit, or even a > penalty. I would strongly recommend benchmarking the naive "keep going > back to the database" approach first, as a baseline, and only testing > these alternatives when you've confirmed that the database really is a > bottleneck. "Depending on your database" is the key phrase. This is not "my" database that is running on localhost. It is an external MSSQL server that I have no control over and whose requests frequently time out. > Hmm, it's complicated. There is another approach, and that's to > completely invert your thinking: instead of "request wants data, so > let's get data", have a thread that periodically updates your cache > from the database, and then all requests return from the cache, > without pinging the requester. Downside: It'll be requesting fairly > frequently. Upside: Very simple, very easy, no difficulties debugging. I'm using a similar approach in other places, but there I actually have a separate process that feeds my local, fast DB with unwieldy data. But that is not merely replicating, it actually preprocesses and "adds value" to the data, and the data is worth retaining on my server. I didn't want to take that approach in this instance because it is a bit too much overhead for essentially "throwaway" stuff. I like the idea of starting a separated "timed" thread in the same application. Need to think about that. Background: The clients are SBCs that display data on screens distributed throughout a manufacturing facility. They periodically refresh every few minutes. Occasionally the requests would pile up waiting for the databsase, so that some screens displayed error messages for a minute or two. Nobody cares but my pride was piqued and the error logs filled up. I've had my proposed solution running for a few days now without errors. For me that's enough but I wanted to ask you guys if I made some logical mistakes.

2 1

Threading question .. am I doing this right?
by Robert Latest Feb. 25, 2022

Feb. 25, 2022

I have a multi-threaded application (a web service) where several threads need data from an external database. That data is quite a lot, but it is almost always the same. Between incoming requests, timestamped records get added to the DB. So I decided to keep an in-memory cache of the DB records that gets only "topped up" with the most recent records on each request: from threading import Lock, Thread class MyCache(): def __init__(self): self.cache = None self.cache_lock = Lock() def _update(self): new_records = query_external_database() if self.cache is None: self.cache = new_records else: self.cache.extend(new_records) def get_data(self): with self.cache_lock: self._update() return self.cache my_cache = MyCache() # module level This works, but even those "small" queries can sometimes hang for a long time, causing incoming requests to pile up at the "with self.cache_lock" block. Since it is better to quickly serve the client with slightly outdated data than not at all, I came up with the "impatient" solution below. The idea is that an incoming request triggers an update query in another thread, waits for a short timeout for that thread to finish and then returns either updated or old data. class MyCache(): def __init__(self): self.cache = None self.thread_lock = Lock() self.update_thread = None def _update(self): new_records = query_external_database() if self.cache is None: self.cache = new_records else: self.cache.extend(new_records) def get_data(self): if self.cache is None: timeout = 10 # allow more time to get initial batch of data else: timeout = 0.5 with self.thread_lock: if self.update_thread is None or not self.update_thread.is_alive(): self.update_thread = Thread(target=self._update) self.update_thread.start() self.update_thread.join(timeout) return self.cache my_cache = MyCache() My question is: Is this a solid approach? Am I forgetting something? For instance, I believe that I don't need another lock to guard self.cache.append() because _update() can ever only run in one thread at a time. But maybe I'm overlooking something.

3 3

Extract specific no of data from nedCDF files using python
by Smital Fulzele Feb. 24, 2022

Feb. 24, 2022

Hi.I am learning python and I am working with some netCDF files. Suppose I have temperature data from 1950-2020 and I want data for only 1960-2015. How should I extract it.

2 1

Coding help
by Richard Pullin Feb. 24, 2022

Feb. 24, 2022

I know next to nothing about computer coding nor Python. However, I am working on a mathematical challenge in which coding is required to calculate and generate different potential solutions. Can anyone help? If so, please private message me and we can discuss in more detail. Many thanks, Richard

2 1

Re: Best way to check if there is internet?
by Robert Latest Feb. 23, 2022

Feb. 23, 2022

Abdur-Rahmaan Janhangeer wrote: > I've got my answers but the 'wandering' a bit > on this thread is at least connected to the topic ^^. > > Last question: If we check for wifi or ethernet cable connection? > > Wifi sometimes tell of connected but no internet connection. > So it detected that there is or there is no internet ... Your application obviously needs to make a certain network connection. Why don't you just check if the connection can be made? Why would you care if it's via cable or Wifi?

1 0

Re: Best way to check if there is internet?
by Robert Latest Feb. 23, 2022

Feb. 23, 2022

Abdur-Rahmaan Janhangeer wrote: > Well, nice perspective. > > It's a valid consideration, sound theory > but poor practicality according to me. On the contrary: It is absolutely the right and only way to do it. > It you view it like this then between the moment > we press run or enter key on the terminal, > our Python interpreter might get deleted. Yeah, but by your logic you'd have to check the existence of /usr/bin/python each time before you run it. > Though it's nice to go in and catch exceptions, > if you have a long operation upcoming it might be nice > to see if your key element is present. That is true, but you still must make sure that your long operation fails in a controlled manner if the necessary ressource becomes unavailable in the meantime. > Checking by exceptions is what the snippet > i shared does. But we check for the ability to do it > before we start. Nothing wrong with checking before. > Of course, it can change in the seconds that follow. But it's too much pure > logic at work. No. It's the correct way of doing it.

1 0

Re: Long running process - how to speed up?
by Robert Latest Feb. 23, 2022

Feb. 23, 2022

Shaozhong SHI wrote: > Can it be divided into several processes? I'd do it like this: from time import sleep from threading import Thread t = Thread(target=lambda: sleep(1)) t.run() # do your work here t.wait()

1 0

Re: Best way to check if there is internet?
by Dennis Lee Bieber Feb. 23, 2022

Feb. 23, 2022

On Wed, 23 Feb 2022 07:08:27 +0300, Abdur-Rahmaan Janhangeer <arj.python(a)gmail.com> declaimed the following: >The upcoming operation is short enough and >though we cannot assume that it will be the case >when we are running the operation, we sensibly >assume it will still be the case. > >This is more an extra check to not start an operation than >either relying to check at the start or when query. >It's both. So you have a success/fail tree of... 1 attempt a connection to some IP/port 1a success -- attempt connection for desired operation 1a1 success -- operation completes 1a2 failure -- report that the operation can not be completed 1b failure -- report that you will not attempt the operation because your test connection to some site failed EVEN IF THE OPERATION COULD SUCCEED (if the problem is with "some IP/port", not the rest of the network) vs 1 attempt connection for desired operation 1a success -- operation completes 1b failure -- report that the operation cannot be completed Personally the second tree is simpler -- AND NEEDS TO BE PART OF THE OTHER TREE ANYWAY! [short tree 1/1a/1b are long tree 1a/1a1/1a2] The only way the longer tree makes sense is by making step 1 a connection to the IP/port requested in the desired operation -- and if that succeeds you've basically performed the shorter tree. > >> -- Wulfraed Dennis Lee Bieber AF6VN wlfraed(a)ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

1 0

Re: Best way to check if there is internet?
by Chris Green Feb. 22, 2022

Feb. 22, 2022

Chris Angelico <rosuav(a)gmail.com> wrote: > On Tue, 22 Feb 2022 at 19:33, Abdur-Rahmaan Janhangeer > <arj.python(a)gmail.com> wrote: > > > > As discussed here but, it would have been nevertheless great to have this > > tiny function instead of > > nothing > > > > Here's a function that determines whether or not you have an internet > connection. It's almost as reliable as some of the other examples > given - I know this, because I tried it ten times, and it gave the > correct result every time! > > def has_internet(): > return True > > Tell me, is it useful to have something that doesn't always give the > right answer, even if it usually does? Is there any value whatsoever > in a lie? > That's sort of in the same area as a stopped clock being right more often than one that runs just a bit slow (or fast). :-) -- Chris Green ·

2 1

Re: Why does not Python accept functions with no names?
by Greg Ewing Feb. 22, 2022

Feb. 22, 2022

On 21/02/22 6:27 am, Abdur-Rahmaan Janhangeer wrote: > Well Python deliberately throws an exception if we do not > pass in a function name. Just wanted to know why it should. As the > above case is a 100% pure anonymous function. The syntax for a function definition is defined in the grammar as requiring an identifier. An identifier in turn is defined as consisting of at least one character. So the grammar would need to be changed to make the name optional. Also, the compiler would need a special case to treat a missing name there as though it were an empty string. So again, why *should* it be allowed, given that the parser and compiler would have to go out of their way to treat it as a special case, only to create a function that there is no easy way to call? BTW, this is not what is usually meant by the term "anonymous function". An anonymous function is one that is not bound to *any* name. The thing you're proposing wouldn't be anonymous -- it would have a name, that name being the empty string. -- Greg

5 5