🌐 Mailman 3 February 2022 - Python-list - python.org

Re: Data unchanged when passing data to Python in multiprocessing shared memory
by Dennis Lee Bieber Feb. 3, 2022

Feb. 3, 2022

On Wed, 2 Feb 2022 00:40:22 +0100 (CET), Jen Kris <jenkris(a)tutanota.com> declaimed the following: > > breakup = int.from_bytes(byte_val, "big") >print("this is breakup " + str(breakup)) > >Python prints: this is breakup 32894 > >Note that I had to switch from little endian to big endian. Python is little endian by default, but in this case it's big endian. > Look at the struct module. I'm pretty certain it has flags for big or little end, or system native (that, or run your integers through the various "network byte order" functions that I think C and Python both support. https://www.gta.ufrj.br/ensino/eel878/sockets/htonsman.html >However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know. <pondering>MMU cache lines not writing through to RAM? Can't find anything on Google to force a cache flush</pondering> Can you test on a different OS? (Windows vs Linux) -- Wulfraed Dennis Lee Bieber AF6VN wlfraed(a)ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

5 6

Re: Waht do you think about my repeated_timer class
by Cecil Westerhof Feb. 3, 2022

Feb. 3, 2022

Cameron Simpson <cs(a)cskk.id.au> writes: > You have: > > def _check_interval(self, interval): > if not type(interval) in [int, float]: > raise TypeError('{} is not numeric'.format(interval)) > > This check is better written: > > if not isinstance(interval, (int,float)): > > which handles subclasses of these types (but note that bool subclasses > int :-) Done, thanks. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof

1 0

Re: Data unchanged when passing data to Python in multiprocessing shared memory
by Dennis Lee Bieber Feb. 2, 2022

Feb. 2, 2022

On Wed, 2 Feb 2022 19:16:19 +0100 (CET), Jen Kris <jenkris(a)tutanota.com> declaimed the following: >It's not clear to me from the struct module whether it can actually auto-detect endianness. I think it must be specified, just as I had to do with int.from_bytes(). In my case endianness was dictated by how the four bytes were populated, starting with the zero bytes on the left. Which is why I also suggested maybe looking at the various network/host translation calls. They are in the socket module of Python, and should also be available in most C standard libraries... https://docs.python.org/3/library/socket.html#other-functions """ socket.ntohl(x) Convert 32-bit positive integers from network to host byte order. On machines where the host byte order is the same as network byte order, this is a no-op; otherwise, it performs a 4-byte swap operation. socket.ntohs(x) Convert 16-bit positive integers from network to host byte order. On machines where the host byte order is the same as network byte order, this is a no-op; otherwise, it performs a 2-byte swap operation. Changed in version 3.10: Raises OverflowError if x does not fit in a 16-bit unsigned integer. socket.htonl(x) Convert 32-bit positive integers from host to network byte order. On machines where the host byte order is the same as network byte order, this is a no-op; otherwise, it performs a 4-byte swap operation. socket.htons(x) Convert 16-bit positive integers from host to network byte order. On machines where the host byte order is the same as network byte order, this is a no-op; otherwise, it performs a 2-byte swap operation. Changed in version 3.10: Raises OverflowError if x does not fit in a 16-bit unsigned integer. """ https://docs.python.org/3/library/struct.html """ Byte Order, Size, and Alignment By default, C types are represented in the machine’s native format and byte order, and properly aligned by skipping pad bytes if necessary (according to the rules used by the C compiler). Alternatively, the first character of the format string can be used to indicate the byte order, size and alignment of the packed data, according to the following table: Character Byte order Size Alignment @ native native native = native standard none < little-endian standard none > big-endian standard none ! network (= big-endian) <<<<<< standard none <<<<<< If the first character is not one of these, '@' is assumed. """ Since all the programs in your situation are running on the same machine, it would appear that at least one of them is NOT formatting integers in native host mode -- and I don't think it is Python. https://www.tutorialspoint.com/unix_sockets/network_byte_orders.htm """ These functions are macros and result in the insertion of conversion source code into the calling program. On little-endian machines, the code will change the values around to network byte order. On big-endian machines, no code is inserted since none is needed; the functions are defined as null. """ -- Wulfraed Dennis Lee Bieber AF6VN wlfraed(a)ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

1 0

Re: http.client and dns lookups
by Dieter Maurer Feb. 2, 2022

Feb. 2, 2022

Michael Welle wrote at 2022-2-1 19:28 +0100: > ... >That doesn't happen when the 'real' issue occurs. Attaching strace to >the Python process I can see that resolv.conf is stat'ed and open'ed. I >guess now I'm more confused than before ;). There must be an additional >condition that I'm missing. The DNS service routinely uses caches. Outdated cache values can cause (apparently non deterministic) failures.

1 0

Why dict.setdefault() has value as optional?
by Marco Sulla Feb. 2, 2022

Feb. 2, 2022

Just out of curiosity: why dict.setdefault() has the default parameter that.... well, has a default value (None)? I used setdefault in the past, but I always specified a value. What's the use case of setting None by default?

2 2

Data unchanged when passing data to Python in multiprocessing shared memory
by Jen Kris Feb. 2, 2022

Feb. 2, 2022

I am using multiprocesssing.shared_memory to pass data between NASM and Python. The shared memory is created in NASM before Python is called. Python connects to the shm: shm_00 = shared_memory.SharedMemory(name='shm_object_00',create=False). I have used shared memory at other points in this project to pass text data from Python back to NASM with no problems. But now this time I need to pass a 32-bit integer (specifically 32,894) from NASM to Python. First I convert the integer to bytes in a C program linked into NASM: unsigned char bytes[4] unsigned long int_to_convert = 32894; bytes[0] = (int_to_convert >> 24) & 0xFF; bytes[1] = (int_to_convert >> 16) & 0xFF; bytes[2] = (int_to_convert >> 8) & 0xFF; bytes[3] = int_to_convert & 0xFF; memcpy(outbuf, bytes, 4); where outbuf is a pointer to the shared memory. On return from C to NASM, I verify that the first four bytes of the shared memory contain what I want, and they are 0, 0, -128, 126 which is binary 00000000 00000000 10000000 01111110, and that's correct (32,894). Next I send a message to Python through a FIFO to read the data from shared memory. Python uses the following code to read the first four bytes of the shared memory: byte_val = shm_00.buf[:4] print(shm_00.buf[0]) print(shm_00.buf[1]) print(shm_00.buf[2]) print(shm_00.buf[3]) But the bytes show as 40 39 96 96, which is exactly what the first four bytes of this shared memory contained before I called C to overwrite them with the bytes 0, 0, -128, 126. So Python does not see the updated bytes, and naturally int.from_bytes(byte_val, "little") does not return the result I want. I know that Python refers to shm00.buf, using the buffer protocol. Is that the reason that Python can't see the data that has been updated by another language? So my question is, how can I alter the data in shared memory in a non-Python language to pass back to Python? Thanks, Jen

3 3

Re: http.client and dns lookups
by Barry Feb. 1, 2022

Feb. 1, 2022

> On 1 Feb 2022, at 19:21, Michael Welle <mwe012008(a)gmx.net> wrote: > > Hello, > > Barry <barry(a)barrys-emacs.org> writes: > >>>> On 31 Jan 2022, at 19:22, Dieter Maurer <dieter(a)handshake.de> wrote: >>> >>> Michael Welle wrote at 2022-1-30 09:18 +0100: >>>> ... >>> The machine this is running on regularly switches >>>> its network configuration without restarting the Python application. Now >>>> it turns out that the application is still using an old, outdated dns >>>> server after such a network configuration switch. >>> >>> It is unlikely that Python performs the host -> IP address translation >>> itself. Almost surely, the translation is delegated to functions >>> of the underlying C runtime library, e.g. "gethostbyname". >> >> And glibc does not have the ability to reload the /etc/resolv.conf I recall. >> If you rely on that you will need to change to using something like >> dnsmasq or systemd-resolved so that resolv.conf uses localhost. > hmmm, I don't know if that's all. For a test I just changed > /etc/resolv.conf manually while the application was sleeping. After > returning from sleep the application knew the change I had made in > resolv.conf and, interestingly, an exception (name resolution failure, > which is correct with the change I made) surfaces in the application. > That doesn't happen when the 'real' issue occurs. Attaching strace to > the Python process I can see that resolv.conf is stat'ed and open'ed. I > guess now I'm more confused than before ;). There must be an additional > condition that I'm missing. That is good to know, last time I hit this was on centos 6 a little while ago. Barry > > Thanks for helping > hmw > -- > https://mail.python.org/mailman/listinfo/python-list

1 0

Re: http.client and dns lookups
by Dieter Maurer Feb. 1, 2022

Feb. 1, 2022

Michael Welle wrote at 2022-1-30 09:18 +0100: > ... The machine this is running on regularly switches >its network configuration without restarting the Python application. Now >it turns out that the application is still using an old, outdated dns >server after such a network configuration switch. It is unlikely that Python performs the host -> IP address translation itself. Almost surely, the translation is delegated to functions of the underlying C runtime library, e.g. "gethostbyname".

2 1