🌐 Mailman 3 October 2024 - Python-list - python.org

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
by Greg Ewing Oct. 2, 2024

Oct. 2, 2024

On 2/10/24 12:26 pm, avi.e.gross(a)gmail.com wrote: > The real problem is how the JSON is set up. If you take umpteen data > structures and wrap them all in something like a list, then it may be a tad > hard to stream as you may not necessarily be examining the contents till the > list finishes gigabytes later. Yes, if you want to process the items as they come in, you might be better off sending a series of separate JSON strings, rather than one JSON string containing a list. Or, use a specialised JSON parser that processes each item of the list as soon as it's finished parsing it, instead of collecting the whole list first. -- Greg

1 0

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
by Greg Ewing Oct. 1, 2024

Oct. 1, 2024

On 2/10/24 10:03 am, Left Right wrote: > Consider also an interesting > consequence of SCSI not being able to have infinite words: this means, > besides other things that fsync() is nonsense! :) If you aren't > familiar with the concept: UNIX filesystem API suggests that it's > possible to destage arbitrary large file (or a chunk of file) to disk. > But SCSI is built of finite "words" and to describe an arbitrary large > file you'd need to list all the blocks that constitute the file! I don't follow. What fsync() does is ensure that any data buffered in the kernel relating to the file is sent to the storage device. It can send as many blocks of data over SCSI as required to achieve this. There's no requirement for it to be atomic at the level of the interface between the kernel and the hardware. Some devices do their own buffering in ways that are invisible to the software, so fsync() can't guarantee that the data is actually written to the storage medium. But that's a problem stemming from the design of the hardware, not the design of the protocol for communicating with the hardware. > the only way to implement fsync() in compliance with the > standard is to sync _everything_ Again I'm not sure what you mean here. It may be difficult for the kernel to track down exactly what data is relevant to a particular file, and so the kernel programmers take the easy way out and just implement fsync() as sync(). But again that has nothing to do with the protocol. -- Greg

1 0

[RELEASE] Python 3.13.0rc3 and 3.12.7 released.
by Thomas Wouters Oct. 1, 2024

Oct. 1, 2024

This is not the release you’re looking for… (unless you’re looking for 3.12.7.) Because no plan survives contact with reality, instead of the actual Python 3.13.0 release we have a new Python 3.13 release candidate today. Python 3.13.0rc3 rolls back the incremental cyclic garbage collector (GC), which was added in one of the alpha releases. The incremental GC had more significant performance regressions in specific workloads than we expected. Rather than try to fiddle with its details in the hope of fixing them (and not making anything else worse) we decided to revert back to the old GC in 3.13. Work on the incremental GC will continue in 3.14. We also took the opportunity to fix some other (rare) bugs and issues found in 3.13.0rc2. The final release of Python 3.13.0 will now happen next week, Monday October 7th . In an effort to return to normalcy, we’ve also released Python 3.12.7 as scheduled, despite the expedited release a month ago. It’s important to be regular! 3.13.0rc3 https://www.python.org/downloads/release/python-3130rc3/ The final cut of 3.13.0 (really, honest). Besides the incremental GC revert it contains a small number of other fixes, as well as many documentation improvements and testsuite improvements (~145 changes in total). <https://discuss.python.org/t/python-3-12-7-and-3-13-0rc3-released/66306#p-1…>Call to action We strongly encourage maintainers of third-party Python projects to prepare their projects for 3.13 compatibilities during this phase, and where necessary publish Python 3.13 wheels on PyPI to be ready for the final release of 3.13.0. Any binary wheels built against Python 3.13.0rc1 and later will work with future versions of Python 3.13. As always, report any issues to the Python bug tracker <https://github.com/python/cpython/issues> . Please keep in mind that this is a preview release and while it’s as close to the final release as we can get it, its use is not recommended for production environments. Next week, though! <https://discuss.python.org/t/python-3-12-7-and-3-13-0rc3-released/66306#p-1…>New features in Python 3.13 - A new and improved interactive interpreter <https://docs.python.org/3.13/whatsnew/3.13.html#a-better-interactive-interp…>, based on PyPy <https://pypy.org/>’s, featuring multi-line editing and color support, as well as colorized exception tracebacks <https://docs.python.org/3.13/whatsnew/3.13.html#improved-error-messages> . - An *experimental* free-threaded build mode <https://docs.python.org/3.13/whatsnew/3.13.html#free-threaded-cpython>, which disables the Global Interpreter Lock, allowing threads to run more concurrently. The build mode is available as an experimental feature in the Windows and macOS installers as well. - A preliminary, *experimental* JIT <https://docs.python.org/3.13/whatsnew/3.13.html#experimental-jit-compiler>, providing the ground work for significant performance improvements. - The locals() builtin function (and its C equivalent) now has well-defined semantics when mutating the returned mapping <https://docs.python.org/3.13/whatsnew/3.13.html#defined-mutation-semantics-…>, which allows debuggers to operate more consistently. - A modified version of mimalloc <https://github.com/microsoft/mimalloc>is now included, optional but enabled by default if supported by the platform, and required for the free-threaded build mode. - Docstrings now have their leading indentation stripped <https://docs.python.org/3.13/whatsnew/3.13.html#other-language-changes>, reducing memory use and the size of .pyc files. (Most tools handling docstrings already strip leading indentation.) - The dbm module <https://docs.python.org/3.13/library/dbm.html>has a new dbm.sqlite3 backend <https://docs.python.org/3.13/whatsnew/3.13.html#dbm>that is used by default when creating new files. - The minimum supported macOS version was changed from 10.9 to 10.13 (High Sierra). Older macOS versions will not be supported going forward. - WASI is now a Tier 2 supported platform <https://peps.python.org/pep-0011/#tier-2>. Emscripten is no longer an officially supported platform <https://peps.python.org/pep-0011/#no-longer-supported-platforms>(but Pyodide <https://pyodide.org/> continues to support Emscripten). - iOS is now a Tier 3 supported platform <https://peps.python.org/pep-0730/>. - Android is now a Tier 3 supported platform <https://peps.python.org/pep-0738/>as well. <https://discuss.python.org/t/python-3-12-7-and-3-13-0rc3-released/66306#p-1…>Python 3.12.7 https://www.python.org/downloads/release/python-3127/ A small release since 3.12.6 was only a month ago, but nevertheless 3.12.7 contains ~120 bug fixes, build improvements and documentation changes. <https://discuss.python.org/t/python-3-12-7-and-3-13-0rc3-released/66306#p-1…>More resources - Python 3.13 Online Documentation <https://docs.python.org/3.13/> - PEP 719 <https://peps.python.org/pep-0719/>, Python 3.13 Release Schedule - Report bugs at Issues · python/cpython · GitHub <https://github.com/python/cpython/issues>. - Help fund Python directly <https://www.python.org/psf/donations/python-dev/>(or via GitHub Sponsors <https://github.com/sponsors/python>), and support the Python community <https://www.python.org/psf/donations/>. <https://discuss.python.org/t/python-3-12-7-and-3-13-0rc3-released/66306#p-1…>Enjoy the new releases Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Regards from a positively *melting* Menlo Park for some reason <https://social.coop/@Yhg1s/113051321976759729>this time, Your release team, Thomas Wouters Łukasz Langa Ned Deily Steve Dower

1 0

Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
by Keith Thompson Oct. 1, 2024

Oct. 1, 2024

2QdxY4RzWzUUiLuE(a)potatochowder.com writes: [...] > In Common Lisp, you can write integers as #nnR[digits], where nn is the > decimal representation of the base (possibly without a leading zero), > the # and the R are literal characters, and the digits are written in > the intended base. So the input #16fFFFF is read as the integer 65535. Typo: You meant #16RFFFF, not #16fFFFF. -- Keith Thompson (The_Other_Keith) Keith.S.Thompson+u(a)gmail.com void Void(void) { Void(); } /* The recursive call of the void */

2 1