Automatic Generation of Python Class Files

I was even thinking of automatically generating the shell for doc
strings and epydoc tags.

Is there existing scripts that do something like this? If not, I will
try to come up with something. If I'm sucessful I'll release the code
under the GPL and will report back to the list.

However, I thought I would check here first so that I don't reinvent
the wheel.

Thanks,

Scott Huey

I can't think of a single reason why you would ever want to do this,
since your "list of method and property names" would be just as
verbose as just typing the actual python code.

Auto generated documentation stubs are considered harmful because they
take the place of real documentation.

Chris,

You wrote: " can't think of a single reason why you would ever want to

I don't think I understand how this would be the same amount of
typing. Consider the following example that would generate a Monster
class file from an input text file (I only did one property and method
in generated class file.):

Contents of input text file:

[Name]
Fire Breathing Dragon

[Properties]
Strength
Scariness
Endurance

[Methods]
eatMaiden argMaiden
fightKnight argKnight

Generated Python Class File:

def class FireBreathingDragon:

def getStrength(self):
"""
Docstring goes here.

@return
@rtype
"""
return self.strength

def setStrength(self, argStrength):
"""
Docstring goes here.

@param argStrength
@ptype
"""
return self.strength

def eatMaiden(self, argMaiden):
"""
Docstring goes here.

@param argMaiden
@ptype
"""

What's a "Python Class File" ?

> a
> little less painful.

If you find writing classes in Python "painful", then either you have no
experience with any mainstream language or you are doing something wrong.

Once again, there's no such thing as a "class file" in Python. A module
usually contains several (classes, functions etc) definitions. And since
there's very few boilerplate code in Python, I don't see what your
system would be good for - directly writing the code will not take much
more time than writing the "file with a list of property names and
method names".

My 2 cents...

Really a "monster class".

Okay this could be written as:

class FireBreathingDragon(object):
def __init__(self, strength):
self.strength = strength
self.scariness = scariness
self.endurance = endurance

def eat_maiden(self, maiden):
pass

def fight_knight(self knight):
pass

Even with all the stuff from your input file it's shorter than your
generated class.

I left out the docs because having no docs is as valuable as your stubs if
not even better. This way you at least see that there is no real docs and
tools like `pylint` can point out the missing docs.

There are editors with good template support for the small amount of
boilerplate that's left.

Just in case you want to defend the default getters and setters: read up
on properties, i.e. the `property()` function, and the discussions in this
group about Python not being Java.

Ciao,
Marc 'BlackJack' Rintsch

Your getters/setters are useless (Python isn't Java) and so are the
docstrings, because you have to write them over again anyway. Removing
them results in 3 lines of code, quite a bit shorter than your input
file, and it doesn't have any of the quite well known issues with this
sort of boiler plate code generation.

Thank you for all of the feedback. As you can no doubt tell I am still
learning to think in "Python" instead of Java.

My intention is to copy the generated text into the module that will
containt the class definition. (I should have said "class definition"
not "class file".

I also intended to add statements creating properties from the getter
and setter methods. I understand that getters and setters aren't
really necessary if you aren't making a property. I just forgot to add
the property statements to my example.

I don't know much about pylint, but I will check it out. It sounds
like a nifty tool. At any rate, I always try to add documentation to
my code, but I can see how the docstring stubs could cause more harm
than good if this wasn't a programmer's habbit.

Thank you for all of the good responses. I hope I didn't offend anyone
with my "Javaness". I think Python is a great language and I hope to
continue learning and using it.

Scott Huey

This should instead generate::

# Inherit from object. There's no reason to create old-style classes.
class FireBreathingDragon(object):

# Python is not Java. You don't need getters and setters.
# Use public attributes. If you ever decide later that you
# need different attributes, you can always use property()
# to make your getters and setters look like public attributes
def __init__(self, stregth, scariness, endurance):

STeVe

Others have already given you reasons why that is a bad idea. Another
one is this: how do you deal with changes to your generated classfile?
It's complicated to ensure that you don't override modified code with
generated.

Diez

Thank you for the advice Steve. I didn't know that I should inherit
from Object. I will change my existing Python code accordingly.

Scott Huey

You still don't want to define the getters and setters if they're just
going to set an attribute. Just use an attribute itself.

Java makes you define getters and setters because there's no way to make
method calls look like an attribute access. So in Java if you start
with a public attribute, you can never adjust the API later.

In Python, you can use property() to make method calls look like
attribute access. This could be necessary if you have an existing API
that used public attributes, but changes to your code require those
attributes to do additional calculations now.

But if you're creating a class for the first time, it should *never* use
property(). There's no need to retrofit anything.

STeVe

For any well-designed, non-trivial code, my guess is that your solution
will end up needed *at least* as much typing. Trivial code doesn't of
course need any kind of code generation - doing it directly will be
faster anyway. I won't of course talk about badly-designed code...

Why do you put spaces in the name ?

should be:

class FireBreathingDragon(object):

> def getStrength(self):

You don't need these getters and setters. Python has support for
computed attributes (look for 'property'), so until you need to control
access, a plain attribute is all you need. IOW:

def eat_maiden(self, maiden):
raise NotImplementedError

<ot>following pep08 (coding style) might be a good idea</ot>

Now if you're doing RPG, there will certainly be some mandatory
attributes for all characters, both players and non-players - IOW,
things like strength, endurance etc belong to a base 'Character' class.
And I seriously doubt that methods like 'eat_maiden' or 'fight_knight'
will survive the first design round... But anyway, a somewhat more
realistic example would look like:

from MyRPG import NonPlayerCharacter, FireBreather

class FireBreathingDragon(NonPlayerCharacter, FireBreather):
def __init__(self, *args, **kw):
super(FireBreathingDragon, self).__init__(*args, **kw)
# other class-specific code here, else might as well
# just skip the initializer

def eat_maiden(self, maiden):
# code here

def fight_knight(self, knight):
# code here

Sorry, but I don't see the benefit of your code-generator here.

I don't need nor want such a docstring in my code, thanks.

Don't take it bad - there are languages that requires so much
boilerplate that without code-generators, productivity tends to drop
near zero. But that's not the case with Python.

Bruno wrote: "Why do you put spaces in the name ?"

I made a mistake here.

Bruno wrote: "You don't need these getters and setters. Python has

So I don't use getters, setters, and the property() function unless I
need to control how a property is accessed. Is this correct?

Bruno wrote: "And I seriously doubt that methods like 'eat_maiden' or

I'm not really designing an RPG. I was jsut trying to come up with a
simple example. What you said about inheriting from a base class like
Character makes sense. I am curious thought, what sounds wrong with
methods like "eatMaiden" and "fightKnight". Are you thinking they
should have been something like "eatCharacter" and "fightCharacter"
instead? I only ask because I am trying to learn how to write well
designed Python code. :]

Bruno wrote: "I don't need nor want such a docstring in my code,
thanks."

I guess epydoc isn't for everyone? :] Maybe I like it because it
reminds me of Javadoc. It seemed useful to me because it helped define
a "protocol" that could be followed for proper use of an object. Maybe
I am trying to replace Java's static typing system with source code
comments?

Bruno wrote: "Don't take it bad - there are languages that requires so

This thread is helping me to realize this. I suppose the only thing my
generator would be good for is the doc comments, and it doesn't sound
like most of you would want those anyways.

Thanks for the feedback.

Scott Huey

Actually, it's a class. But anyway...

It is. The default is to use a plain attribute unless you already know
you need a computed one.

Also, properties are just one example use of the descriptor protocol,
which you can use to define your own 'smart properties' (it's a common
idioms for ORMs and like).

I think I did guess this - but nonetheless, it looked to much like a bad
example of OO design to me.

Looks somehow better. Now what about 'eat' and 'fight' ?-)

It's not really Python-specific - it's about not hard-wiring what should
be generic.

Fact is that I don't really like all those XXXdoc systems. But mostly,
it's because, as other pointed out, no docstring is better than this
template code.

Which is certainly a sensible concern. Now as far as I'm concerned, a
listing of arguments names and type is not the best approach. Good
naming, textual explanations and code examples are IMHO better.

Documenting what 'kind' of objects your function waits for and/or
returns - and/or what's the function's intent - is certainly not a bad
idea in itself. But you don't necessarily needs that much formalism.
Most of the time, the source code is readable enough for those who
want/need a very formal and *exact* definition of the function !-)

Then welcome onboard !-)

Indeed. Another point is that, when it comes to 'boilerplate reduction',
Python has far more to offer than textual code generation. Using a
combination of custom descriptors, decorators, metaclasses and closures
(and of course more common patterns like the template method...), you
can abstract out pretty much of the 'plumbing' *without* any textual
source-code generation. You may want to have a look at Elixir (an
ActiveRecord inspired tiny declarative layer above SQLAlchemy) or
FormEncode (a validation/conversion package) to see what I mean.

> Thanks for the feedback.

Thanks for reading the feedback !-)

May I kindly disagree here ?-)

Computed attributes are IMHO not only a life-saver when it comes to
refactoring. There are cases where you *really* have - by 'design' I'd
say - the semantic of a property, but know from the start you'll need
computation (for whatever reason). Then what would be the rationale for
using explicit getters/setters ?

Bruno,

Thanks for the detailed response. I need some time to digest what you
wrote. I will read this thread again and think carefully about what
you have saaid.

Scott Huey

Diez,

I didn't realize all of the shortcomings with my example until the
others responded. I was just trying to save myself some typing, but I
realize know the problem was my lack of understanding Python syntax.

I will consider myself schooled! :]

Scott Huey

Of course. ;-)

I'd be interested to hear what these use cases are.

Generally, people expect that attributes are used for simple state, and
methods are used when some sort of computation is required. If you use
attributes when some computation is required, you violate your users'
expectations.

Of course, you can always document that your attributes require
computation. But you're definitely adding some cognitive load by
violating the normal expected behavior of attributes and methods.

STeVe

Stupid little example: A circle object with `radius` and `diameter`
attributes. It doesn't make sense to store both a as normal attributes
because they will eventually get out of sync. One can be implemented as a
property.

Another one is delegation of attribute access. I'm thinking of a wrapper
class around an object with an attribute, say `is_active`, and a wrapper
that has a property with the same name that returns the value of the
wrapped objects attribute.

Or lazy computation of an attribute. Breaks expectations for the first
access -- long calculation for simple attribute access -- but meets it for
every subsequent access.

Ciao,
Marc 'BlackJack' Rintsch

I once wrote a small ORM-like wrapper for LDAP access, and, for reasons
that might be obvious for anyone having worked with LDAP, I choosed to
keep the record values in the format used by the lower level LDAP lib
and let user code access them thru computed attributes (actually
custom-made descriptors). Talking about ORMs, most of them surely use a
similar scheme.

I could also list the CS101 examples, like Shape.area,
Rect.bottom_right, Person.age, etc... And yes, some of these attributes
are obviously read-only. That doesn't prevent them from being
semantically *properties*, not *behaviour*.

Depends on which kind of people and on their background. I obviously
don't advocate the use of computed attributes for heavy, non-cachable
computations (please don't call me stupid !-), but not suffering from a
too long exposures to some low-level mainstream languages, I never
assume that attribute syntax implies direct attribute access.

"normal expected behaviour" ? "normal" according to which norm, and
"expected" by who ? In a language with a very strong builtin support for
computed attributes, the only thing to expect is that you don't have to
call an attribute to get to the value.

Could be. But I'd tend to define a get_diameter() method here. As a
user of your class, I'd like to know which one is doing the computation.
What if I'm using ridiculously large ints, and I care about the
multiplication?

This is retrofitting to an API, so it already falls under the only time
I think you *should* be using property(). Although in this case, it
might be that __getattr__ is more appropriate.

There are descriptors better than property() at achieving this behavior.
See for example:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/363602

My personal experience is that every time I write stuff this way (and I
have many times), I end up re-writing it later as a method because it's
less confusing for a method to take a long time than it is for an
attribute access.

STeVe

But this is trying to match an existing API, the only case that I think

> I could also list the CS101 examples, like Shape.area,

But as I mentioned in another email here, I'd rather know which ones
require computation, on the off chance that even the small amount of
additional calculation matters (say, for large integers or in a very
tight loop).

I guess as long as your documentation is clear about which attributes
require computation and which don't...

STeVe

Certainly not. There's *no* existing API here. It's just exposing as
attributes some values that are *not* attributes, and that requires some
computation to get at.

And then ? If you need to get this value, and getting it requires
computation, then you'll have to do that computation - whatever the
context. Looks like premature optimization to me. And then, what then if
the implementation changes and what was plain attributes get turned into
properties and vice-versa ?

Why should it ? FWIW, I mentionned that I would obviously not use
properties for values requiring heavy, non cachable computation. This
set aside, the fact that the attribute is computed (or not) is none of
the concerns of client code. I think you know that concept, it's called
'encapsulation' !-)

Encapsulation doesn't mean having no understanding of the object you're
using. We all use Python lists, knowing that adding to the end is
cheap, and adding to the front is expensive. If you're using someone's
data type, but you have no idea where the performance costs are, you're
just asking for trouble. That's true even if you're not prematurely
optimizing -- in the case of lists, doing the wrong sort of insert can
make an O(N) algorithm an O(N**2) algorithm. So I don't think this has
anything to do with encapsulation.

This really only has to do with what you and I think are natural
programming intuitions. I agree that Python is not Java, or C, or
whatever, but there are a wide variety of precedents in a wide variety
of languages (the Python stdlib included) that lead many programmers to
expect only minimal computation to be performed when accessing
attributes. I believe we simply disagree on whether or not Python
programmers need to be trained out of these expectations.

STeVe

Nope (or not exactly !-) but it means that interface should be decoupled
from implemantation, and that, as a user, you shouldn't have have to
worry that much about implementation details.

Indeed. But
1/ this is true weither you use properties only for refactoring plain
attributes or you use them righ from the start
2/ I'm sorry to have to repeat it, but I never advised using computed
attributes for heavy computations.

I can only agree on this. But is this really what we're talking about ?
I thought you said - well, let me quote it, it will be simpler (nb:
emphasis is yours):

Now you're talking about not _abusing_ this feature - a point on which,
FWIW, I wholefully agree from the start. Are you saying that using
properties is an abuse in itself ?

If then, one should only use explicit accessors ? But then, what would
prevent anyone to put heavy computations into accessors ?

And else - ie, if you think using computed attributes is legitimate for
simple things - why not using them right from the start when it makes
sens instead of having a strange mix of attribute syntax and explicit
accessors syntax (for an implementation that may finally end up using
properties under the hood and have explicit accessors just, well,
accessing a plain attribute) ?

Should I mention *once again* that I never advertized using properties
for heavy computations ?-)

I believe we simply disagree on weither properties should be used when
it makes sens (from a semantic POV, which doesn't prevent common sens,
thanks) or if they should be restricted to refactoring-life-saver. Now
since the second use implies that some attributes-looking properties of
an object may not be what they looks like - and even worth, may suddenly
become (a little bit) more costly without client code being warned - I
don't see what's your problem with the first one.

Ok, I think we all understood your (quite sensible) concerns about not
calling super-heavy-computations without knowing, but (did I say it ?)
that's certainly not what I'm talking about, nor how anyone in it's own
mind would use this feature. No, don't tell me, I know there are a
couple ho-so-very-smart peoples out there that actually will. But what ?
There's no cure for stupidity. So far, it seems that most Python users
don't suffer from such major brain disorders - I'd even find them
suprisingly reasonnable when it comes to black magic -, so I don't think
there's a need for such a rule as the one you were suggesting.

Ho, and wrt/ training and expectations, I'm afraid you have to live with
the fact that computed attributes (either properties and custom
descriptors) are alreay widely used. FWIW, may I remind you that a
method is actually a callable returned by a computed attribute ?-)

I said those kind of expectation violations should be documented, and
you said "Why should it?" I gather you really only meant the "Why
should it?" in the situation that the computation is extremely minimal.
I can understand that, but I'd still like it documented for any
objects I use.

Yep. And it's documented. ;-)

STeVe

Did you failed to spot some kind of pattern in my previous posts ?-)

More seriously: I do assume that what looks like an attribute will not
require *heavy* computation - a bit more than 'extremely minimal' is ok
for me, but we may not have the same definition of 'extremely minimal'.
And FWIW, I do assume too that computation-heavy functions will be
either obvious (either from the domain and/or wrt/ what I do pass as
arguments - ie, I expect that sorting a 10K items list with a custom cmp
callback will be heavier that naturally sorting a 10 words list) or
documented as such. Else, I just write my code and, if and when I have a
real problem, I profile the whole thing... strange enough, but so far
it seems to work just fine.

There are quite a lot of things I'd like to see documented - but
hopefully I'm mostly working with OSS, so I usually use the source when
I need a very exact and detailed documentation !-)

Now how does your desire for documentation imply that "if you're
creating a class for the first time, it should *never* use property()" ?
I still wonder...

How many Python users ever bothered to read that part of the doc ? How
many are not even aware of this ?

Of course, there's *never* any such thing as "never" in Python. ;-)

STeVe

P.S. If you really don't understand what I was getting at there, feel
free to contact me off-list. I think I've polluted c.l.py enough. ;-)

We recently had to change an object pipeline from new style classes to
old style. A lot of these objects were being created and the *extra
overhead* of new style classes was killing us. :-)

Michael
http://www.manning.com/foord

I'm curious. Was it memory or speed overhead that was the problem?

Steve

Can you please expand on this? What extra overhead of new-style
classes are you referring to?

Memory overhead seems to firmly favor new-style classes, especially
for small object. A trivial small-object allocation test such as

# After running this, use "top" or "ps" to see the ballpark memory
# consumption. Remove "(object)" to measure for old-style class.
python -c 'import time
class A(object): pass
l=[A() for n in xrange(2000000)]
time.sleep(100)'

shows the Python process growing to 78M with a new-style class and to
354M with an old-style class. And that is without even starting to
use actual optimizations, such as using __slots__! Instantiation time
difference is less drastic, but still shows significant improvement
with the use of new-style classes: 0.27 usec for new-style, 0.40 usec
old-style (measured with timeit, subtracted baseline overhead).

It was speed overhead - I have to add a disclaimer that this is
IronPython, but my understanding is that the situation is similar with
CPython. Creating new style class instances is slower than for old
style classes.

This was in the execution of spreadsheets, where we had a pipeline of
three objects for each spreadsheet element (not sure if the
implementation is still the same now).

Michael
http://www.manning.com/foord

>
> Steve

The obvious timeit test shows new-style classes being created in half
the time of old style ones. I certainly would not generalize to
CPython from IronPythons performance characteristics, especially with
something that's so dependent on both the .NET runtime and on the
details of IronPython integration as object creation.

C:\>python -m timeit -s "class C: pass" "c = C()"
1000000 loops, best of 3: 0.315 usec per loop

C:\>python -m timeit -s "class C(object): pass" "c = C()"
10000000 loops, best of 3: 0.175 usec per loop