Which Python 2 -> Python 3 changes caught you out?

March 4th, 2015 | Categories: programming, python | Tags:

If you really want to learn the differences between Python 2 and Python 3, I suggest you try converting a non-trivial software project. I’m in the middle of doing one now and am learning all kinds of little gotchas over and above the standard stuff that everyone knows such as changes to print, integer division and removal of xrange.

The most recent one I learned about (about 10 minutes ago) amounted to this

#Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
9

compared to

Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'x' is not defined

This is well documented (This StackOverflow Q+A is great!) but I didn’t know about it and, in the code I was looking at, there was a heck of a lot of complication between the list comprehension and when ‘x’ was used. As such, it took me a while to figure out!

Another change that had me scratching my head for a while is the fact that Python 3 ignores the __metaclass__ hook. I didn’t know this little fact but discovered it while debugging failing tests!

Of course, once you know these little gotchas, you’ll probably not be caught out by them again in your next Python 2->Python 3 porting project but they got me wondering…..

What changes from Python 2->Python 3 have really caught you out at some point?

  1. Mike Croucher
    March 5th, 2015 at 09:50
    Reply | Quote | #1

    The following comments about this post came in from Twitter

    @michiexile: ‘I’m teaching Python3 this semester, and suddenly kinda enjoy it. I was caught by dict.keys() not being a normal iterable tho’.

    ‏@chendaniely: had the same experience when I first ran `range()`

    @michiexile: “map objects” have been tripping me up yesterday. Makes me go back to [f(x) for x in l] instead

  2. Mike Croucher
    March 5th, 2015 at 09:51
    Reply | Quote | #2

    Another gotcha for me is scipy.weave

    If you find yourself working on a project that makes heavy use of it, you’ve got work ahead of you for your Python 3 conversion. Cython is usually the solution.

  3. Mike Croucher
    March 7th, 2015 at 09:18
    Reply | Quote | #3
  4. March 9th, 2015 at 07:39
    Reply | Quote | #4

    I just wrote a brief blog post about my own experiences:
    http://ilovesymposia.com/2015/03/08/experiences-porting-a-medium-sized-library-from-python-2-to-3/

    In summary:

    – dictionary iteration order has changed (which was never guaranteed to be conserved, but in actuality was, throughout 2.7’s long life)
    – pickle compatibility is tricky
    – code may run way slower in 3

    Your scoping one is nicely illustrated though! Thanks!

  5. Royi
    March 10th, 2015 at 07:27
    Reply | Quote | #5

    Hi Mike,
    Wanted to ask you,
    Have you worked on an Image Processing algorithm which creates a large Laplacian Matrix?
    Something like:

    http://scicomp.stackexchange.com/questions/11387/interpolation-by-solving-a-minimization-problem-optimization

    I’m wondering if it can be solved efficiently with the NAG library.

    Thank You.

  6. NAGfan
    March 10th, 2015 at 20:26
    Reply | Quote | #6

    HI Royi, I know some of the guys at NAG very well. My guess is without knowing more about your particular problem (such as sparsity of images) it would be difficult for them or Mike to judge what from the NAG Library would be most useful to you. Almost certainly NAG’s Large Scale Linear Systems chapter F11 will contain some direct/iterative solvers that you could use. You might want to review the Library manual http://www.nag.co.uk/numeric/CL/nagdoc_cl24/html/FRONTMATTER/manconts.html and in particular their chapter introduction guides are very good http://www.nag.co.uk/numeric/CL/nagdoc_cl24/html/F11/f11intro.html

    Other than that suggest you trial the NAG Library if you don’t have access already. Appreciate this is not a great help, but…

  7. Royi
    March 11th, 2015 at 09:44
    Reply | Quote | #7

    Hi NAGfan,

    The above link describe the Matrix pretty accurately.
    This is what I need, a solve which is quick (Uses SSE, AVX), memory efficient with API to preconditioners.

  8. March 11th, 2015 at 17:28
    Reply | Quote | #8

    When working with CSVs, csv.reader accepts a `delimiter` argument in case you want to use tabs or pipes or something like that instead of commas.

    If you use `from __future__ import unicode_literals` in a module where you try to use this argument, Python 2 will complain that `delimiter` should be a str, not a unicode. If you use a bytestring literal (e.g., b’\t’) to try to make Python 2 happy, Python 3 will complain that it should be a string, not bytes.

    Before you start looking around for a while in the docs for six, like I did, it turns out the solution is just to wrap your string literal in a str() call (e.g., str(‘\t’)) so each version sees what it wants to see.

  9. Will Furnass
    March 12th, 2015 at 19:08
    Reply | Quote | #9

    Something that caught me out and took a little debugging was the change in behaviour of the round function:
    – Py 2.x: always round the mid-point between two whole numbers up to the next integer (away from zero)
    – Py 3.x: always round the mid-point between two whole number to the nearest _even_ integer

  10. April 22nd, 2015 at 13:43

    Good exercise. Hopefully we’ll all be doing more of this in 2015!

    The unicode / str thing always seems to generate more work than the “advertising” would have you believe.

    Even though I know about it, that map() and .items() return iterators rather than lists still trips me up. Especially when I want to insert print calls.