Python, learnt backwards

Why this tutorial?

From time to time I like to idly answer questions on StackOverflow. There are two sorts of questions I look for:

1. specific questions in a niche area in which I want to continue developing expertise
2. beginner questions in Python or SQL that I can answer quickly (maybe saving someone else a lot of time)

The former are more enriching for me personally. A good answer takes research time and writing time, and I always learn something along the way.

The latter are what inspired me to write this.

After a while, you notice patterns.

Many first-time questioners don’t ‘get’ StackOverflow; in my opinion, you should write a question there when all else has failed. It’s not well-suited as a source of on-demand tutorials. If you are a beginner your question has almost certainly been answered; learning to search old questions is a better use of your time.

An aside: I don’t mean this in a derogatory way — searching is a skill, one acquired through experience. And in the end, spending more time searching first will help you write a better question later — that is, one more likely to get good answers, promptly.

How? Well, why is searching frustrating? Because people have written sub-par questions: unclear titles, minimal detail, spelling errors, too specific when they could have been general. These things hurt searchability and answerability. The art of searching entails getting around such limitations.

I wrote this tutorial as an alternative for a certain group of novice Pythonistas who post questions in the form of a certain mysterious error. By the end, you should have a much deeper understanding of such errors and know how to act on them.

It won’t be short, but I hope it will be enlightening.

AttributeError: ‘tutorial’ object has no attribute ‘get_to_the_point’

How did you learn Python? My route was circuitous.

I had the good fortune to spend a few hours a week one undergraduate term being taught programming — and the misfortune to be taught in C.

I then had to learn IDL for my Masters research project in Astronomy, because at the time Astronomers were still in the thrall of it.

Fortunately, I saw sense halfway through my PhD and switched to Python. I rewrote a lot of code. I received a limited amount of instruction, asked questions when I got stuck, and everything was very focused on ‘how do we accomplish a particular task?’

It is no exaggeration to say that I learnt far more about writing good code in my first six months in industry than across the whole of the half-decade just described.

(the problem there is that I’d have got loads more research done if I’d known how to write good code earlier …)

Look. Of course it’s satisfying to finish some goal-oriented instruction and accomplish something tangible, something real. I don’t have a problem with that! However. If you never, ever bother with the fundamentals, if you never, ever go back and fill in the gaps left by your mad rush to solve today’s problem, you are limiting yourself.

Case in point: my intended audience are often stuck because they can’t parse the meaning of an actually-pretty-helpful error.

Why not? No-one ever taught them how to!

In short, teaching people how to do stuff with Python, expecting deeper understanding to follow later is the natural way to do it. But I contend that there is also a case for doing it the other way around: for teaching Python backwards.

This is my first attempt.

Finally, some code

When I’ve taught Python ‘basics’ before, I focused first on a few different types of thing Python understands — numbers, text, true vs false — then moved on to

1. variable assignment (x=1)
2. simple expressions (like addition, multiplication, etc)
3. collections (lists, dictionaries, tuples, sets)
4. logic (if … else)
5. functions …

This was intended as the quick way to get people ready to do useful things. The idea was that my students would know just enough to import functions they needed from various libraries, then string them together into a useful programme.

That’s fine, but the result was that showing them anything like this

class Saucepan:
    handle_length = 15  # cm

… was a mere afterthought.

What if we try learning that first?

Keeping it classy

Have you heard of Plato’s Theory of Forms? A brief account (albeit horribly warped for my own purposes) might go like this:

• how do you recognise a tree?
• true, you’ve seen other trees before, and you were told that some of them were trees, but none of those trees looks exactly like the oak you’ve just come across on your walk
• and yet you successfully generalise from the examples of trees you have seen and recognise the new thing as being a tree!
• therefore you must have some more fundamental idea of a tree, of which all the ones you’ve seen are mere imperfect examples
• this ‘form of the tree’, being perfect, is more real than any of the actual trees
• you probably encountered it before you were born, when your soul travelled in the world of forms, and that’s how you recognise trees so easily now, in the debased physical realm … ahem.

This is (essentially) how Python works.

When I write

class Saucepan:
    ...

… I am defining a class called ‘Saucepan’. It’s not an actual Saucepan (whatever that is …), it’s more the idea of a Saucepan. The form of the Saucepan, if you will.

Saucepans come in all shapes and sizes. You can recognise them because they have certain things in common, like having a handle, although details may differ between saucepans, such as

• material of construction
• shape
• size
• etc.

In Python we call these ‘attributes’:

class Saucepan:
    material = "Aluminium"
    nonstick = True
    handle_length = 15  # cm

Classes, all the way down

Now at this point, if we were really doing this from scratch, I’d have to go on a digression and explain some things, such as “what is this True thing?” Instead, I’m going to assume that you know that.

What may not have occurred to you is that when I tell you

• "Aluminium" is a character string
• True is a Boolean value
• 15 is an integer

… I am really telling you what class of thing each of these is. Hold on to that thought.

There’s method in this madness

Why am I so obsessed with Saucepans? Well, you can do a lot of great stuff with them. Similarly, it’s what you can do with your classes that makes them really useful.

Certain special attributes capture these ‘things you can do’. We call them methods. The syntax follows:

class Saucepan:
    def make_spaghetti(self):
        print("Yum!")

Again, now would be the time for a digression, but in reality you likely know plenty about functions, arguments, outputs and so on — and will therefore recognise most of the syntax.

In the spirit of thinking backwards, however, I encourage you to start thinking of functions (such as print) as methods that have been detached from any class.

Similarly, you could think of regular variables as detached attributes.

Objects are the subject

The special self argument of make_spaghetti refers to the particular Saucepan you’re using to make_spaghetti (please note: not the class, but an example of the class).

We’d better talk about how to create such an example, which we call an instance of a class (‘Saucepans, for instance this one …’).

The generic term for class instances is ‘objects’. Creating them is simple enough:

class Saucepan:
    handle_length = 15  # cm
    def make_spaghetti(self):
        print("Yum!")


my_saucepan = Saucepan()

print(my_saucepan.handle_length)  # prints '15'

my_saucepan.make_spaghetti()  # hopefully you can guess what this does

There. A simple pair of brackets and we have a saucepan we can use. Yum!

To recap:

• my_saucepan is an object
• handle_length is an attribute
• make_spaghetti is a method, which is a special kind of attribute
• Saucepan is a class

Initially, it seemed simple enough…

“Wait!” I hear you cry, “any saucepan object I make using this class will have a fifteen centimetre handle! What if I want several different saucepans that aren’t all the same?”

You’re right. This means I need to introduce a special method:

class Saucepan:
    def __init__(self, material, nonstick, handle_length):
        self.material = material
        self.nonstick = nonstick
        self.handle_length = handle_length

my_saucepan = Saucepan("Aluminium", True, 15)

print(my_saucepan.material)  # prints 'Aluminium'

What’s this __init__? Unlike other methods, it gets called when we create an instance (a process we call ‘initialisation’).

Things to note:

• when I created my_saucepan I passed in some arguments "Aluminium", True, 15, which get sent to this special method (self is always the first argument in the method definition, and it’s included implicitly when we call the method)
• some assignments happen inside the method: self.material = material is equivalent to my_saucepan.material = material (the point is, it should work the same way for any instance of Saucepan)

Our attributes in the earlier examples were class attributes — they are the same for any instance of the class. They can’t be changed (at least, not without changing them for all Saucepans, everywhere).

These new attributes are instance attributes — they are specific to the instance of the class, don’t exist until initialisation, and can be changed afterwards.

Some case studies

This is not a full discussion of object-oriented programming (else this long tutorial would be longer still), but it equips us to handle these problems.

It’s not the case that every AttributeError: … object has no attribute … question is rooted in misunderstanding, but enough of them are that it’s worth unpacking.

These errors are generally a ‘case of mistaken identity’. Either an object wasn’t of the expected class, or the class lacked a method or attribute you thought it ought to have.

Let’s illustrate with some examples.

Let me preface this by saying: zero blame lies on any of the linked questioners. I wrote this for them, but also for the many others who ask similar questions at an early stage in their journey. It’s up to us old hands to make clearer paths for new travellers.

An example with pandas

Take this one— AttributeError: 'DataFrameGroupBy' object has no attribute 'to_csv'.

The popular pandas library can be especially confusing, because of its heavy use of method chaining.

Imagine this:

class SelfReplicatingSaucepan:
    def clone(self):
        return SelfReplicatingSaucepan()

first_saucepan = SelfReplicatingSaucepan()

final_saucepan = first_saucepan.clone().clone().clone().clone()

We can make that chain as long as we like, because every one returns a fresh instance of the SelfReplicatingSaucepan. They are resolved left to right:

final_saucepan = first_saucepan.clone().clone().clone().clone()
final_saucepan = second_saucepan.clone().clone().clone()
final_saucepan = third_saucepan.clone().clone()
final_saucepan = fourth_saucepan.clone()
final_saucepan = fifth_saucepan

In pandas this enables a syntax wherein a series of transformations are chained together in order. This is at once very intuitive and deeply confusing, if you aren’t following closely.

In the case of the linked question, the poster was confused because the DataFrame.groupby method doesn’t return a DataFrame object at all; it returns a DataFrameGroupBy object, a completely different class with different methods.

Armed with the above details, the poster may deduce that they need to use one of those methods to convert back to a DataFrame … which does have a to_csv method. The right thing would then be to consult the documentation.

An example with numpy

Secondly, let’s examine this one — AttributeError: 'numpy.ndarray' object has no attribute 'append'.

In this case the poster has taken some code and tried to replace the builtin Python list, which does have an append method, with the higher-performance numpy N-dimensional array ndarray.

Unfortunately, ndarrays pre-allocate their memory and aren’t so flexible; they do not have an append method.

An aside: the NumPy library does have a function append, which creates a brand new array instead of modifying the existing one, new = append(array, new_values). That said, in principle NumPy could have given ndarray an append method returning a new array. This would enable method chaining wont_work = array.append(new_values). Perhaps the developers thought it would be confusing.

I would suggest this poster ought to look at the ndarray documentation and/or search for ‘append to a numpy array’, which comes back with some great, useful results.

An unexpected NoneType example

Perhaps most common of all are AttributeError: 'NoneType' object has no attribute ... errors. There are usually several posted per day.

Brief digression: Python has a special value, None, which you can think of as an example of the NoneType class. When a function or method doesn’t return anything, None is implicitly returned. If you haven’t seen this before:

def some_function():
    print("I return nothing!")

result = some_function()

if result is None:
    print("Function returned nothing!")

Similarly, Python dictionaries have a method get which safely retrieves the value for a particular key. By default, if the key is not present, the retrieved value will be None:

my_dictionary = {"key1": "value1", "key2": "value2"}

k1_val = my_dictionary["key1"]  # the standard way to retrieve a value

k2_val = my_dictionary.get("key2")  # the safe way

k3_val = my_dictionary.get("key3")  # won't throw an error

if k3_val is None:
    print("my_dictionary did not contain 'key3'")

In short, None is Python’s built-in way of representing a missing value.

So when you find yourself faced with AttributeError: 'NoneType' object has no attribute '...' errors, usually the object you expected to be working with just isn’t there.

To debug the error, you must look at the line where the ‘missing’ method is invoked, and locate the object that was supposed to have it. For example, if this code threw AttributeError: 'NoneType' object has no attribute 'cool_method' …

my_object = something_that_returns_an_object()

my_object.cool_method()

… then I would know that something_that_returns_an_object had actually given me None rather than whatever I expected. I could then investigate that part of the code.

Conclusion

And so we reach the end.

This is a bit of an experiment. I’m interested to hear whether you found this approach helpful.

I’m open to the idea of writing more articles like this in the future, and have an idea for how a series of them could hang together, but of course I want to be sure that this longform, more abstract treatment actually adds something!

So, if you enjoyed, let me know below.