Python's string formatting syntax

As you are surely aware of, you can use python's % operator to do string formatting. It's part of the reasons why the language is so good at text processing.

The left operand is the format string, the right operand is a single format argument, or an iterable of format arguments. In the format string, you can use %s to insert a string, %d to insert an integer, %f for a float, etc. It's much like C's printf family of functions.

    >>> '%s %d' % ('a string', 123)
    'a string 123'

This you almost surely know. You might not know that the string can take a %r argument which calls repr on the arguments. Now that's useful!

    >>> 'a repr: %r' % ['', ""]
    "a repr: ['', '']"
    >>> 

Or that the format parameters can be in a dictionary.

    >>> '%(a_string)s %(an_int)d' % {'a_string':'a string', 'an_int': 123}
    'a string 123'
    >>> 

If the parameters were passed as a dictionary, the format string will not raise exceptions for extra parameters. So you can use locals() on your (trusted) format string, and format it easily. In a class method, self.__dict__ would also be useful.

    >>> a_number = 3
    >>> '%(a_number)d' % locals()
    '3'

String formatting parameters will also take their own parameters. The parameters are decimal characters (and an optional separation dot) between the % sign and the character indicating the type.

Here is an ugly syntax representation. Bear with me.

% [0|-][leading/trailing character count][An optional dot][Truncation](Type character)

That's all, I think. Be careful to add no spaces. The only characters allowed between % and the type indicator are [0-9\.\-]

• First, the usual percentage sign.

• Then, add a single zero if you want to fill the string with zeroes (you specify how many characters you would like to fill the string with in the next argument). If you want the string to have trailing spaces instead of leading spaces, add a minus sign.

• After that, add the number of characters you want to fill the string with. You can skip this.

• If you want to use the truncation argument (explained below), add a dot here.

• Add the truncation argument, which is an integer.

For obvious reasons, you can't have trailing zeroes in anything.

The "Truncation" argument is used to left-truncate a string to that size, to grow (only grow) an integer string by zero-filling the left to that size, or to use that many decimal digits in a float.

Here are some examples with %s. %s doesn't let you put leading zeroes on your string. You can use str.zfill() for that.

    >>> s = '1234'
    >>> '%.1s' % s
    '1'
    >>> '%.4s' % s
    '1234'
    >>> '%.6s' % s
    '1234'
    >>> '%10s' % s
    '      1234'
    >>> '%10.2s' % s
    '        12'
    >>> '%10.4s' % s
    '      1234'
    >>> '%10.6s' % s
    '      1234'

%f examples:

    >>> f = 10.123
    >>> '%f' % f
    '10.123000'
    >>> '%.4f' % f
    '10.1230'
    >>> '%4f' % f
    '10.123000'
    >>> '%20f' % f
    '           10.123000'
    >>> '%20.2f' % f
    '               10.12'
    >>> '%020.2f' % f
    '00000000000000010.12'
    >>> 

%d can also be told to have leading zeroes or leading spaces. As mentioned above the "Truncation" part of the parameter can only make it grow. It wouldn't make sense to let it shrink, since that would shrink the number's value.

    >>> i = 10
    >>> '%d' % i
    '10'
    >>> '%.1d' % i
    '10'
    >>> '%.4d' % i
    '0010'
    >>> '%4d' % i
    '  10'
    >>> '%4.1d' % i
    '  10'
    >>> '%4.3d' % i
    ' 010'
    >>> '%4d' % i
    '  10'
    >>> '%04d' % i
    '0010'
    >>> 

As you know, in python, characters are indeed strings with len() of 1, so if you want to represent a character you just use %s. But you can also use %c. %c will raise an exception if the input string is not a character, and the extra assertion might prove a little useful. You can add leading and trailing spaces to this parameter too.

Finally, it's worth something to note that python has a newer formatting syntax. I haven't seen it used much, and don't use it either. It's not so practical as the one I described.

More information on string formating can be found in the python documentation

This has hopefully been a thorough dissection of python's string formatting syntax. Now go out and enjoy the sun!

Python Discovery: Named Tuples

I had known of the named tuple idiom for ages. It is used sparingly in some python API's, and I feel it is very pythonic, practical, and simple to read and code with.

It's used like this (this example is for the timetuple in datetime.datetime, which is a namedtuple, or looks a lot like one):

>>> tt = datetime.datetime.now().timetuple()
>>> tt
time.struct_time(tm_year=2012, tm_mon=10, tm_mday=15, tm_hour=20, ...)
>>> tt.tm_year
2012
>>> tt[0]
2012
>>> tt[3]
20
>>> tt.tm_mon
10
>>> 

And it's iterable.

>>> list(tt)
[2012, 10, 15, 20, 24, 8, 0, 289, -1]

It's basically a tuple which has attributes accessible through dot notation.

The timetuple idiom is perfect for API design, and for those times where a class is what you want, but a tuple would be okay too. For example, 2D dimensions. You could store them in a class with height, width attrs, or in a tuple.

I thought it was just some pythonic pattern, but today I needed to automate it because I found myself making dumb __init__ methods like these:

def __init__(self, left, right):
    self.left = left
    self.right = right

It is a useful pattern. It's pythonic. It's implementable in pure python with just a little metaprogramming. It could be in the standard library, I thought.

I opened up my python interpreter and entered import collections and dir(collections) looking for NamedTuple or something. namedtuple was there. It wasn't just a coincidence.

collections.namedtuple is a factory for creating namedtuple types, which you can then use. The resulting classes include methods like _asdict, _fields, index and _replace.

>>> a = collections.namedtuple('a', 'b, c')
>>> a(1,2)._asdict()
OrderedDict([('b', 1), ('c', 2)])
>>> a(1,2)._fields()
('b', 'c')
>>> a(1,2).index(1)
0
>>> a(1,2).index(2)
1
>>> a(1,2)._replace(b=3)
a(b=3, c=2)
>>> 

It is hashable, unlike a dict, and pickleable.

Here is how you create one:

>>> collections.namedtuple('SomeNamedTuple', 't1 t2')

The first argument is the name of the output class, and the second argument is the list of parameter names, whitespace - and/or comma - delimited.

This creates a named tuple with the __name__ SomeNamedTuple, and the attributes t1 and t2.

Extend it

When you need more stuff out of your namedtuple you should really subclass. Keep in mind that it is immutable, so you should return a copy of it in every method.

class Dimensions(collections.namedtuple('DimensionsTuple', 'height width')):
    pass

>>> Dimensions(10, 30).height
10
>>> Dimensions(10, 30).width
30

It is rather useful sometimes, so don't forget about the named tuple! You might need it sooner or later.

jQuery.showWhen

I've created a small jQuery plugin to allow for more responsive forms. Sometimes you only want to show parts of a form when other parts are complete. This helps you simplify forms and only go into detail when the user has already filled in some field. Here is an example, and an example with a checkbox instead
Think about a survey:

1. Do you smoke?
   1. Yes
   2. No (skip to 4.)
2. How often do you smoke?
   1. Every hour
   2. Every two hours
3. Do you realize ...
   1. Not really
4. Is question four a question?
   1. Yes
   2. No

In this survey, we could have tried to follow the standard of interactive and intuitive forms to hide questions number 3. and 4.
This could be done in a non-obstrusive way: If the "(skip to 4.)" text was inside its own span with its own CSS class, it would be trivial to add data-skip-questions to that span, and have jQuery look for span[data-skip-questions], and use String.split with jQuery.add to register with this plugin.
html code:

<ol class="questions">
    <li>Do you smoke?
        <ol>
            <li><input type="radio" name="q1" value="y" />Yes</li>
            <li><input type="radio" name="q1" value="n" />No <span data-skip-questions="2 3">(skip to 4.)</span></li>
        </ol>
    </li>
    <li>How often do you smoke?
        <ol>
            <li><input type="radio" name="q2" value="1" /> Every hour</li>
            <li><input type="radio" name="q2" value="2" /> Every two hours</li>
        </ol>
    </li>
    <li>Do you realize ...
        <ol>
            <li><input type="radio" name="q3" value="n" /> Not really...</li>
        </ol>
    </li>
    <li>Is question four a question?
        <ol>
            <li><input type="radio" name="q4" value="y" /> Yes</li>
            <li><input type="radio" name="q4" value="n" /> No</li>
        </ol>
    </li>
</ol>

javascript code:

$('span[data-skip-questions]').each(function() {
    var questionsToSkip = $(this).data('skip-questions').split(' '),
        q = $(null),
        watchedInput = $(this).parents('li:first').find('input[type="radio"]');

    $.each(questionsToSkip, function(i, val) {
        q = q.add($('ol.questions > li')
              .eq(val - 1));
    });

    $(q).hide().showWhen(watchedInput, true);

})
// Hide the span, since the user doesn't have to skip the questions snymore.
.hide();

Here's a fiddle for the survey example.

Edit:
The plugin is now on github. It was released under the WTFPL. Feel free to fork and use!

Python: "or" and "and" operators yield more stuff than bool

Python's and and or operators don't yield bool. They don't calculate the result of the expressions into booleans, and most certainly do not give us True or False.

They return one of the objects we put in. So they are not useful only in if statements, and can't convert to a boolean by themselves (without bool(), that is).

You might be familiar with using or like this since it's a common alternative to ... if ... else .... Take this __init__ method for example:

    def __init__(self, form=None):
        self.form = form or self.make_form()

It is straightforward, readable, and short. Pythonic indeed.

If the right operand is an expression which would throw an error, as long as the left operand is true, you don't have to worry. The right expression will not be evaluated. This is not old. We have seen it in if statements in the C language. I have used it countless times in Java to avoid NullPointerException. It wouldn't make any sense for a machine or virtual machine to evaluate both expressions if the first one already evaluates to true.

>>> def get_sth():
...     raise NotImplementedError
... 
>>> [1] or get_sth()
[1]
>>> [] or get_sth()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in get_sth
NotImplementedError

A very good use of this is to try to tentatively get a resource in one manner which is more likely to succeed, and then try a more risky or less desireable fallback if the first manner yields nothing.

This is awesome for creating intuitively usable classes. For example, a make_form implementation could be a single-line "raise NotImplementedError", so client code can initialize the class without passing in a form object every time, but if they want to use the defaults they just have to inherit the class and make their own make_form method. It's very practical, intuitive and informative behavior just for a single expression, isn't it?

Here is the or expression behavior, described:

• When one of the operands evaluate to True, return the one which evaluates to True.

• >>> 1 or 0
  1
  >>> 0 or 1
  1

• When both evaluate to True, returns the left one (python won't even look at the second operand, so you can rest assured no methods will be called, if any.)

• >>> 1 or [1]
  1
  >>> [1] or 1
  [1]

• When both evaluate to False, returns the right one.
• >>> 0 or []
[]
>>> [] or 0
0

We can use the last caracteristic, too. Sometimes two values are False, but we want to use one over the other. (standard falsy values are the empty list, the empty tuple, 0, False, None, and the empty string).

The and operator is rather interesting. It gives you:

• The right operand, when both evaluate to True:

• >>> True and 1
  1
  >>> 1 and True
  True 

• the operand which evaluates to False, when one of them is falsy:

• >>> 19 and 0
  0
  >>> 0 and 19
  0

• The left operand, when both evaluate to False

• >>> 0 and False
  0
  >>> False and 0
  False

Interesting-c

I'm creating a new programming language!

This language will be created through transcompilation to C. Inspired by the concept of CoffeeScript, and including a lot of Python philosophies and ideas, I will try to create a language which is really easy to use, and yet fast as C.

One of the most important features will be reflection and metaprogramming facilities. This will help developers by describing the structure of their programs at runtime. This makes for good ORM's, serializers, etc.

It will not use classes for encapsulation, but instead a new concept I have created. It will build upon the capabilities of classes and the concept of "is-a" relationships. More on these soon.

interesting-c aims to be able to interpret 99% of existing C programs as interesting-c. So a C program will (almost) always be a interesting-c program. This allows developers to convert gradually to interesting-c.

interesting-c will have a module system. modules will be able to import other modules as well as pure C source or header files. The syntax will be something like this:

    import "c_module.c";
    import icmodule;

When importing, interesting-c will just add a #include directive to the compiled source, but it will simulate the concept and behavior of a namespace. It will parse the #included file and look for identifiers which the current module can use. Optionally, users will be able to use import module_name as alias syntax to assign the module to an identifier so its namespace can be isolated from the local module namespace. This poses a new problem: C has no concept of a namespace and will not allow the programmer to choose between foo in module1.c and foo in module2.c. It's unclear how I will solve this

interesting-c will discourage (but still support) preprocessor directives. But there will be a lot more space for safer, more interesting pre-compile-time magic as well as runtime magic. And yet, you will still be able to shoot yourself in the foot and use macros as much as you want.

Early work

I have started creating interesting-c, and it really gives me new interesting problems to solve.

I am using PLY (python-lex-yacc)'s lex to create the lexer. My challenge right now is to preserve whitespace as possible, as well as most of the code appearance, since it's important for the users to be able to opt in and out of interesting-c at any time, so it will be something easy to adopt.

I have been creating small C programs to test C language features. Programs like these help me test whether and how the language supports something.

    /* Test whether floating point numbers can be octal */

    int main(int argc, char* argv[]){
        float oct = 01.1f;
        return 0;
    }

Python: str.split has a "limit" argument

I have recently made this little discovery. It seems that str.split has a maxsplit argument, which tells it to only split into a certain amount of parts. This could be really useful for text parsing.
I have in the past run into some (rare) situations where I needed to do this, but didn't know of the maxsplit parameter, and ended up using str.join and slices, to recreate the rest of the string with the delimiters.

It's a little boring to do, and it is ugly.

>>> url = '/posts/blog-1/10251/'
>>>
>>> #problem: split the URL into two parts
... #such that first_part == 'posts' and second_part == 'blog-1/10251'
... #first solution: split and join with slices.
...
>>> first_part = url.strip('/').split('/')[0]
>>> second_part = '/'.join(url.strip('/').split('/')[1:])
>>> first_part, second_part
('posts', 'blog-1/10251')

However, if we do this using the split limit argument, it becomes much more readable.

>>> #second solution: use unpacking, and str.split() with the limit argument
...
>>> first_part, second_part = url.strip('/').split('/',1)
>>> first_part, second_part
('posts', 'blog-1/10251')
>>>

The "limit" argument asks you how many splits you would like, not how many fragments you would like. So specify the n-1 when you want n fragments.

What about splitting by whitespace?

Splitting by whitespace is one of the most powerful features in str.split(). Since I usually invoke this functionality using "".split() without any arguments, I was worried about splitting by whitespace, with the limit argument being a positional-only argument, but you can also use "".split(None).
This is nice since the exact whitespace that used to be there would be impossible to recover with the above tactic (since it's not just a delimiter character).

>>> 'OneFragment TwoFragments ThreeFragments'.split()
['OneFragment', 'TwoFragments', 'ThreeFragments']
>>> 'OneFragment TwoFragments ThreeFragments'.split(maxsplit=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: split() takes no keyword arguments
>>> 'OneFragment TwoFragments ThreeFragments'.split(None, 1)
['OneFragment', 'TwoFragments ThreeFragments']

Split by whitespace, and preserve it.

When you split by whitespace, str.split splits by spaces, tabs, carrier returns and newlines. There are many whitespace characters, and sometimes you want to preserve this information. When using string.split and joining it back, you have no way of getting that information back. It's gone. However, the maxsplit argument allows you to preserve the existing whitespace.

>>> 'And together we fled.\r\nWe said:\r\n\t"Hello!"'.split(None, 1)
['And', 'together we fled.\r\nWe said:\r\n\t"Hello!"']
>>> print _[1]
together we fled.
We said:
        "Hello!"

Python: Unpacking works with one item!

One of the advantages in python is that it is very practical to unpack variables out of short lists and tuples. Some pieces of code which would otherwise be repetitive and ugly (a = lst[0]; b = lst[1]) end up clean, short and easy to read.

>>> a,b = [1,2]
>>> a
1
>>> b
2
>>> 

It's the reason behind python's multi-return values making sense. When we create code that is going to be used by other modules, it opens up a lot of possibilities, and eases the writing and understanding of the client code that uses them.

>>> #create_monkey will use the monkey count in its calculations.
... This value is very useful for client code, but it is rather expensive to obtain.
... monkey, new_monkey_count = create_monkey()
>>>

My discovery today is that you can "unpack" a list or tuple even when it contains only one item. Check it out:

ActivePython 2.7.2.5 (ActiveState Software Inc.) based on
Python 2.7.2 (default, Jun 24 2011, 12:22:14) [MSC v.1500 64 bit (AMD64)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> a, = [1]
>>> a
1
>>> a, = []
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: need more than 0 values to unpack
>>> 

This opens up some possibilities. If we are one hundred percent sure that the list contains a single value, we can unpack that into a variable instead of using [0], which is inherently ugly.

When you are not sure whether the container is empty (or more than one item), you can always catch the resulting ValueError. Or use this:

>>> (a,) = [] or ['default']
>>> a
'default'
>>> 

However, the comma is very subtle and future code readers (including the smartass who decided to use this obscure thang) will probably not notice it. This effectively makes refactoring affected code tricky.

Furthermore, someone may think that it is a syntax error, edit the comma out, and TypeErrors and ValueErrors start popping up everywhere. Subtle bug hunting fun!

There is a little workaround for these readability issues, which is to use full tuple syntax:

>>> (a,) = [1]
>>> a
1
>>>

So it seems like I can use this single-unpacking to do good and not just evil.
It seems to be really interesting, but I am not sure whether I should use it in real code. Seems like I have some meditation to do.
In the meantime, I can show off a little.

>>> a, = 1,
>>> a
1
>>>