🌐 Mailman 3 February 2023 - Python-list - python.org

Re: Organizing modules and their code
by transreductionist Feb. 4, 2023

Feb. 4, 2023

On Friday, February 3, 2023 at 5:31:56 PM UTC-5, Thomas Passin wrote: > On 2/3/2023 4:18 PM, transreductionist wrote: > > Here is the situation. There is a top-level module (see designs below) containing code, that as the name suggests, manages an ETL pipeline. A directory is created called etl_helpers that organizes several modules responsible for making up the pipeline. The discussion concerns the Python language, which supports OOP as well as Structural/Functional approaches to programming. > > > > I am interested in opinions on which design adheres best to standard architectural practices and the SOLID principles. I understand that this is one of those topics where people may have strong opinions one way or the other. I am interested in those opinions. > Well, you have pretty well stacked the deck to make DESIGN 1 the > obviously preferred choice. I don't think it has much to do with Python > per se, or even with OO vs imperative style. > > As a practical matter, once you got into working with > extract_transform_load.py (for the other designs), I would expect that > you would start wanting to refactor it and eventually end up more like > DESIGN 1. So you might as well start out that way. > > The reasons are 1) what you said about separation of concerns, 2) a > desire to keep each module or file relatively coherent and easy to read, > and 3, as you also suggested, making each of them easier to test. > Decoupling is important too (one of the SOLID prescriptions), but you > can violate that with any architecture if you don't think carefully > about what you are doing. > > On the subject of OO, I think it is a very good approach to think about > architecture and design in object terms - meaning conceptual objects > from the users' point of view. For example, here you have a pipeline (a > metaphorical or userland object). It will need functionality to load, > transform, and output data so logically it can be composed of a loader, > one or more transformers, and one or more output formatters (more > objects). You may also need a scheduler and a configuration manager > (more objects). > > (*Please* let's not have any quibbling about "class" vs "object". We > are at a conceptual level here!) > > When it comes to implementation, you can choose to implement those > userland objects with either imperative, OO, or functional techniques, > or a mixture. > > Allow me to give my thoughts. First, I don't think there would be much difference if I was using OOP for the functionality, or using a structural paradigm. A structural paradigm in my opinion, along the lines of Rich Hickey's comments on simple versus complex, would be a simpler implementation. In this case there is no reason to create a construct with state. So let's assume the code is structural and not OOP. > > > > I would go with Design I. Succinctly stated, Design I supports readability and maintainability at least as well, if not better than the other designs. The goal of the SOLID principles are the creation of mid-level software structures that (Software Architecture: SA Martin). I think Design I best adheres to these principles of: > > ---- Tolerate change, > > ---- Are easy to understand, and > > ---- Are the basis of components that can be used in many software systems. > > > > I could point to the Single Responsibility Principle which is defined as (SA Martin): a module should be responsible to one, and only one, actor. It should satisfy the Liskov Substitution Principle as well. Further, each module in the etl_helpers directory is at the same level of abstraction. > > > > I could also mention that as Dijkstra stressed, at every level, from the smallest function to the largest component, software is like a science and, therefore, is driven by falsifiability. Software architects strive to define modules, components, and services that are easily falsifiable (testable). To do so, they employ restrictive disciplines similar to structured programming, > > albeit at a much higher level (SA Martin). > > > > One can point to multiple reasons why Design I might be preferred, but what are the compelling reasons, if there are any, that would suggest another design was superior. > > > > Finally, let me reference an interesting research paper I read recently that seems to support the other designs as anti-patterns: Architecture_Anti-patterns_Automatically.pdf > > > > ---- (https://www.cs.drexel.edu/~yfcai/papers/2019/tse2019.pdf) > > > > SEVERAL DESIGNS FOR COMPARISON > > > > DESIGN I: > > > > ---- manage_the_etl_pipeline.py > > ---- etl_helpers > > ---- extract.py > > ---- transform.py > > ---- load.py > > > > Of course one could also > > > > DESIGN II: > > > > ---- manage_the_etl_pipeline.py > > ---- etl_helpers > > ---- extract_transform_load.py > > > > or probably even: > > > > DESIGN III: > > > > ---- manage_the_etl_pipeline.py > > ---- extract_transform_load.py On point that I think is worth making ,and I forgot to make it, is that namespaces are ubiquitous in Python: Built-in, Global, Function, and Enclosing namespaces, as well as user namespaces, e.g. dictionaries, the SimpleNamespace, and DataClasses to list just a few. Modules ARE namespaces. Namespaces organize programming constructs like classes, functions, variables, etc. into coherent groups of "things". To have a namespace that complects extract constructs with transform constructs, and load constructs in one module seems un-pythonistic.

1 0

Re: evaluation question
by rbowman Feb. 3, 2023

Feb. 3, 2023

On Thu, 2 Feb 2023 16:09:09 -0000 (UTC), Muttley wrote: > What if its not a few scripts? What if its 10s of thousands of lines of > core production code? If the company it belongs to wants to add new > Python 3 features it can't just plug them into the code because it won't > run under Python 3, they have to do a full overhaul or even complete > rewrite and that costs a lot of time and money. Tell me about it... Esri is the 500 pound gorilla in the GIS industry. They haven't been secretive about their roadmap but the tools people have been using for almost 20 years are going, going, GONE. Part of that is their scripting language ArcPy moved to 3.8. It's a minor inconvenience for me to update some old scripts and to develop with 3 but a lot of firms have those thousands of lines of code they've developed over the years in ArcPy for GIS data manipulation. More painful for me is the C++ API is also gone. Legacy sites have a couple of more years before it's all over. That leaves me with a foot in both worlds. > Unfortunately a lot of people who've never worked in large institutions > with huge code bases don't understand this. I don't work in a large company but I deal with about 25 years of legacy code every day. The core functionality is good but the time and money is spent putting lipstick on a pig. Technical debt is an industry wide problem.

1 0

Re: Licensing?
by Jon Ribbens Feb. 3, 2023

Feb. 3, 2023

On 2023-02-02, Stefan Ram <ram(a)zedat.fu-berlin.de> wrote: > Many licenses in the Python world are like: "You can make > changes, but have to leave in my Copyright notice.". > > Would it be possible that the original author could not > claim a Copyright anymore when code has been changed? No. If you change someone else's code then you have created a derived work, which requires permission from both the original author and you to copy. (Unless you change it so much that nothing remains of the original author's code, of course.)

2 2

Re: evaluation question
by Greg Ewing Feb. 3, 2023

Feb. 3, 2023

On 3/02/23 5:09 am, Muttley(a)dastardlyhq.com wrote: > What if its 10s of thousands of lines of > core production code? If the company it belongs to wants to add new Python 3 > features it can't just plug them into the code because it won't run under > Python 3, they have to do a full overhaul or even complete rewrite and that > costs a lot of time and money. A possible strategy in that case would have been to incrementally rewrite it in such a way that the code would run in both 2.7 and 3.x (various features were added to 2.7 to make that possible). When that point is reached, you can then switch to running it with Python 3 and start using the new features. Also, if you're a company whose business is totally reliant on some piece of code, it would be prudent to plan ahead and budget for rewriting or replacing it at some point. People seem to think that because code doesn't wear out like hardware, you don't have to budget for replacing it. But you can't expect third party software to be maintained forever -- particularly when, as with Python, the maintenance is mainly being done by *volunteers*. -- Greg

1 0

Upgrading Python on Ubuntu 22.04.1 LTS
by אורי Feb. 2, 2023

Feb. 2, 2023

Hi, I have a server with Ubuntu 22.04.1 LTS and the Python version there is Python 3.10.6. Is there a safe way to upgrade to the latest version of Python 3.10 (3.10.9)? I tried with the OS update and upgrade but the Python version remained the same. Thanks, אורי uri(a)speedy.net

6 10

Re: evaluation question
by rbowman Feb. 2, 2023

Feb. 2, 2023

On 1 Feb 2023 17:31:02 GMT, Stefan Ram wrote: > rbowman <bowman(a)montana.com> writes: >><Venting> Why does every language have to invent their own function to >>print to the console that is very similar but not the same as the rest >>of the herd?</Venting> > > Why do there have to be different languages at all? https://homepages.cwi.nl/~steven/abc/programmers/introduction.html "Why ABC? The answer to the question 'Why a new language?' is the same as the answer to the question 'Why new computers?': because they can help you do the job better. With the choice between a language where it will take a week to write a program, and a language where it will take an afternoon, most people will choose the latter." That leads to the question of when Van Rossum was looking for a hobby project, why not extend ABC? Or Pike? https://pike.lysator.liu.se/about/history/ Then there is the question of how a new language becomes popular. When Matsumoto developed Ruby it was almost 4 years before there was any coherent English documentation. How did it get traction? How about Go? Thompson and Pike hate C++ (with cause) so they went back to C and reworked it. Then there is C++ itself, which was released before its time. There are many more obscure languages when someone saw a need. Then there are features the propagate like lambdas. Everyone came down with lambda envy and shoehorned them into the language one way or the other.

1 0

Re: evaluation question
by Muttley＠dastardlyhq.com Feb. 1, 2023

Feb. 1, 2023

On Wed, 1 Feb 2023 13:17:33 +1300 dn <PythonList(a)DancesWithMice.info> wrote: >On 01/02/2023 11.59, Greg Ewing wrote: >> On 31/01/23 10:24 pm, Muttley(a)dastardlyhq.com wrote: >>> All languages have their ugly corners due to initial design mistakes >>> and/or >>> constraints. Eg: java with the special behaviour of its string class, C++ >>> with "=0" pure virtual declaration. But they don't dump them and make >>> all old >>> code suddenly cease to execute. >> >> No, but it was decided that Python 3 would have to be backwards >> incompatible, mainly to sort out the Unicode mess. Given that, >> the opportunity was taken to clean up some other mistakes as well. > >+1 >and the move to Unicode has opened-up the Python community beyond the >US, to embrace 'the world' - a proposition (still) not well-recognised >by (only) English-speakers/writers/readers. > > >Even though the proposition has a troll-bait smell to it:- > >1 nothing "ceased to execute" and Python 2 was maintained and developed >for quite some time and in-parallel to many Python 3 releases. MacOS only comes with python3 now. If you have a whole load of python2 code you want to run you now have to manually install python2 yourself. >2 the only constant in this business is 'change'. I'd rather cope with >an evolution in this language (which we know and love), than one day Its not evolution, its revolution. Evolution retains old functionality.

4 3

Re: evaluation question
by Greg Ewing Feb. 1, 2023

Feb. 1, 2023

On 1/02/23 7:33 am, Stefan Ram wrote: > Thomas Passin <list1(a)tompassin.net> writes: > Some people say it is a function now so that you can redefine it. Well, that's one benefit, but I wouldn't say it's the main one. The point is really that you can do *anything* with it now that you can do with a regular function -- pass it as an argument, wrap it with another function, define your own function with a similar signature for duck-typing purposes, etc. > It would still be possible to have a special syntax for the outermost > expression of an expression statement that would allow one to omit > the parentheses, That's only one of the syntactic oddities of the old print statement, thogh. There was also the >> thing, special treatment of trailing commas, etc. Also, introducing a paren-less call syntax would be a very big and controversial change that would be way out of proportion to the problem. -- Greg

4 3

Re: evaluation question
by Greg Ewing Feb. 1, 2023

Feb. 1, 2023

On 1/02/23 1:17 pm, dn wrote: > 1 nothing "ceased to execute" and Python 2 was maintained and developed > for quite some time and in-parallel to many Python 3 releases. And a lot of effort was put into making the transition as easy as possible, e.g. 2to3, and the features added to 2.7 to make it easier to write code that would work in both versions. -- Greg

1 0