🌐 Mailman 3 April 2021 - Python-list - python.org

Re: memory consumption
by Alexey April 1, 2021

April 1, 2021

четверг, 1 апреля 2021 г. в 15:27:01 UTC+3, Chris Angelico: > On Thu, Apr 1, 2021 at 10:56 PM Alexey <zen.su...(a)gmail.com> wrote: > > > > Found it. As I said before the problem was lurking in the cache. > > Few days ago I read about circular references and things like that and > > I thought to myself that it might be the case. To build the cache I was > > using lots of 'setdefault' methods chained together > > > > self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})... > > > > and instead of wring a long lines I decided to divide it to increase > > readability > > > > cluster = self.__cache.setdefault(cluster_name, {}) > > database = database.setdefault(database_name, {}) > > ... > > and I guess that was the problem. > > > > First thing I did was to rewrite this back to single line. > If the cache is always and only used in this way, it might be cleaner > to use a defaultdict(dict) instead of the setdefault calls. Or, since > this appears to be a two-level cache: > > self.__cache = defaultdict(lambda: defaultdict(dict)) > > and then you can simply reference > self.__cache[cluster_name][database_name] to read or update the cache. I agree > Having that be more efficient than either self.__cache=None or del > self.__cache (which will be equivalent), I can understand. But better > than clearing the dict? Seems very odd. In this particular case 'cache.clear()' just don't work (in context of releasing memory). If someone can tell me why, I'll be very thankful > Ideally, though, you'd want to NOT have those reference loops. I > presume the database objects need to have a reference to whatever > 'self' is, but perhaps the cache can be done externally to the object, > which would make all the references one-way instead of circular. But > that's something only you can investigate. I did some refactoring and changed my code already.

1 0

Re: memory consumption
by Alexey April 1, 2021

April 1, 2021

Found it. As I said before the problem was lurking in the cache. Few days ago I read about circular references and things like that and I thought to myself that it might be the case. To build the cache I was using lots of 'setdefault' methods chained together self.__cache.setdefault(cluster_name, {}).setdefault(database_name, {})... and instead of wring a long lines I decided to divide it to increase readability cluster = self.__cache.setdefault(cluster_name, {}) database = database.setdefault(database_name, {}) ... and I guess that was the problem. First thing I did was to rewrite this back to single line. And it helped. In the morning I tried different approach and decided to clear cache with different way. So instead of doing self.__cache.clear(), self.__cache = None or even 'del self.__cache' I did: for item in list(self.__cache.keys()): del self.__cache[item] and againg effect was positive. As a result I decided to rewrite all the methods to build,update and get from cache without 'setdefault' and to use "for loop" instead of dict.clear(). I don't understand underlying processes and why I had this issues. In my opinion if you're leaving the function scope you don't have to worry about what variables you're leaving there. There are other guys at StackOverflow who has similar problems with big dictionaries and memory but they're using different approaches. Thanks you everyone for your time and advices! I think the problem is solved and I hope some one will find this helpful

3 2

Re: memory consumption
by Alexey April 1, 2021

April 1, 2021

среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg: > On Tue, Mar 30, 2021 at 1:25 AM Alexey <zen.su...(a)gmail.com> wrote: > > > > > I'm sorry. I didn't understand your question right. If I have 4 workers, > > they require 4Gb > > in idle state and some extra memory when they execute other tasks. If I > > increase workers > > count up to 16, they`ll eat all the memory I have (16GB) on my machine and > > will crash as soon > > as system get swapped. > > > What if you increase the machine's (operating system's) swap space? Does > that take care of the problem in practice? I can`t do that because it will affect other containers running on this host. In my opinion it may significantly reduce their performance.

4 4

Re: memory consumption
by Alexey April 1, 2021

April 1, 2021

четверг, 1 апреля 2021 г. в 14:57:29 UTC+3, Barry: > > On 31 Mar 2021, at 09:42, Alexey <zen.su...(a)gmail.com> wrote: > > > > среда, 31 марта 2021 г. в 01:20:06 UTC+3, Dan Stromberg: > >>> On Tue, Mar 30, 2021 at 1:25 AM Alexey <zen.su...(a)gmail.com> wrote: > >>> > >>> > >>> I'm sorry. I didn't understand your question right. If I have 4 workers, > >>> they require 4Gb > >>> in idle state and some extra memory when they execute other tasks. If I > >>> increase workers > >>> count up to 16, they`ll eat all the memory I have (16GB) on my machine and > >>> will crash as soon > >>> as system get swapped. > >>> > >> What if you increase the machine's (operating system's) swap space? Does > >> that take care of the problem in practice? > > > > I can`t do that because it will affect other containers running on this host. > > In my opinion it may significantly reduce their performance. > Assuming this a modern linux then you should have control groups that allow you to set limits on memory and swap for each container. > > Are you running with systemd? I really don't know.

2 1

Re: Retrieving non-/etc/passwd users with Python 3?
by Loris Bennett April 1, 2021

April 1, 2021

Christian Heimes <christian(a)python.org> writes: > On 31/03/2021 14.45, Loris Bennett wrote: >> Chris Angelico <rosuav(a)gmail.com> writes: >> >>> On Wed, Mar 31, 2021 at 11:21 PM Loris Bennett >>> <loris.bennett(a)fu-berlin.de> wrote: >>>> >>>> Hi, >>>> >>>> I want to get a list of users on a Linux system using Python 3.6. All >>>> the users I am interested in are just available via LDAP and are not in >>>> /etc/passwd. Thus, in a bash shell I can use 'getent' to display them. >>>> >>>> When I try to install the PyPi package >>>> >>>> getent >>>> >>>> I get the error >>>> >>>> File "/tmp/pip-build-vu4lziex/getent/setup.py", line 9, in <module> >>>> long_description = file('README.rst').read(), >>>> NameError: name 'file' is not defined >>>> >>>> I duckduckwent a bit and the problem seems to be that 'file' from Python >>>> 2 has been replaced by 'open' in Python 3. >>>> >>>> So what's the standard way of getting a list of users in this case? >>>> >>> >>> I don't have LDAP experience so I don't know for sure, but is the >>> stdlib "pwd" module suitable, or does it only read /etc/passwd? >>> >>> https://docs.python.org/3/library/pwd.html >>> >>> Failing that, one option - and not as bad as you might think - is >>> simply to run getent using the subprocess module, and parse its >>> output. Sometimes that's easier than finding (or porting!) a library. >> >> D'oh! Thanks, 'pwd' is indeed exactly what I need. When I read the >> documentation here >> >> https://docs.python.org/3.6/library/pwd.html >> >> I mistakenly got the impression that it was only going to give me the >> local users. It doesn't actually say that, but it mentions /etc/shadow >> and not getent. However, it does talk about the "account and password >> database", which is a clue (although our passwords are on an other >> system entirely), since "database" is more getent terminology. >> >> In any case, I think 'pwd' is hiding its light under a bushel a bit >> here. > > Please open a documentation bug :) I'll have a look :) > The pwd and grp module use the libc API to get users from the local > account database. On Linux and glibc the account database is handled by > NSS and nsswitch.conf. > > By the way I recommend that you use SSSD instead of talking to LDAP > directly. You'll have a much more pleasant experience. Yes, we do use SSSD, but my grasp of what it does is pretty much limited to "as well as looking at the local /etc/passwd it can also talk to LDAP" :/ Cheers, Loris -- This signature is currently under construction.

1 0

Re: Canonical conversion of dict of dicts to list of dicts
by Greg Ewing April 1, 2021

April 1, 2021

On 31/03/21 7:37 pm, dn wrote: > Python offers mutable (can be changed) and immutable (can't) objects > (remember: 'everything is an object'): > https://docs.python.org/3/reference/datamodel.html?highlight=mutable%20data While that's true, it's actually irrelevant to this situation. > $ a = "bob" > $ b = a > $ b = "bert" > $ a > 'bob' Here, you're not even attempting to modify the object that is bound to b; instead, you're rebinding the name b to a different object. Whether the object to which b was previously bound is mutable or not makes no difference. You can see this if you do the equivalent thing with lists: >>> a = ["alice", "bob", "carol"] >>> b = a >>> b ['alice', 'bob', 'carol'] >>> b = ['dave', 'edward', 'felicity'] >>> a ['alice', 'bob', 'carol'] >>> b ['dave', 'edward', 'felicity'] -- Greg

1 0