Planet Pypefitters

Here are some random things I've learned while writing technical documentation:

• When talking about concepts, using singulars makes for clearer reading than using multiples. For example, rather than composing a sentence like this:

  "Applications built with foobar may be frobnobbed using fleebars."

  It's usually clearer to say something like:

  "An application built using a foobar can be frobnbbed using a fleebar."

  Personally, I find keeping to the singular keeps things concrete, and prevents the text from careening into the abstract too quickly.

• Don't be afraid to repeat yourself. Use pronouns sparingly. For example, it's often clearer to say: "The cat sleeps on the sofa. The cat is a tommy" rather than "The cat sleeps on the sofa. It is a tommy." Is the cat or the the sofa a tommy? The first form is repetitive, but it's crystal clear that the cat is a tommy.
• Examples beat any amount of purely narrative explanation.

I just uploaded a tutorial on using Storm with BFG to the BFG tutorial bin. The application is traversal-based, and uses the storm.zope-based transaction integration, in combination with repoze.tm2, to make for a very pleasing little CRUD app.

Gustavo Narea reports:

A few months ago I wrote an article on repoze.who and repoze.what, which has just been published on Python Magazine: http://pymag.phparch.com/c/issue/view/98.

I believe it's a good resource for those who are new to both frameworks (even if they aren't familiar with WSGI or auth in general yet), as well as for current users to better understand how repoze.who/what work and so make the most out of them.

I just whipped up a tiny bit of middleware for OSI's version of the KARL project: they needed to put markup for Google Analytics into each page served. We wanted to avoid adding anything to the core KARL software which imposed such a policy on other users of KARL; we might have added it via the "customization package" for OSI, but it would have still required changing the core software in ways that seemed too invasive.

This seemed like a natural use for WSGI middleware, which could intercept the outbound respond and add the additional markup, without requiring any changes to the application. It turns out that the implementation is simple, cheap and generic: it would be useful for any WSGI-based deployment which wants to use Google Analytics.

So, I released it to the Cheeseshop, with the minimal docs also served from there.

Enjoy!

- Tres Seaver

I’ve been working on some Javascript to enhance the TextArea elements on the PylonsHQ Snippets section, and have noticed that… well, TextArea’s suck.

The hack I’ve seen is to use one of the newer features of browsers, the editable or ‘design’ attribute’s for div elements I believe. This lets one build a very snazzy amount of features, such as syntax highlighting, code completion, etc., but I don’t think I needed to go that far.

I only have one main design goal, this TextArea is for the user to enter RestructuredText so it’d be awesome if the TextArea acted in a way that made rst a bit nicer. The obvious two things that came to mind:

1. Tab key indents 4-spaces
2. Hitting return on an indented line, will retain the indentation on the next line

I’ve actually gotten some Javascript, hobbled together from various parts of the net, along with an ‘enter’ key handler I wrote myself on bitbucket.

Course, it’d also be nice to have a button or key combo, that will indent/unindent a selection in the TextArea as well.

Anyone else have any Javascript they’ve hobbled together in the past to make TextArea a little nicer for restructured text?

According to my Ohloh stats:

• Zope Repository commits, January 2000-Present: 1476
• Repoze Repository commits, September 2007-Present: 1485

In November, I moved to New Zealand, and I was very happy to have been graciously invited to Kiwi Foo by Nat Torkington. As a Python programmer, it was quite a bit of fun to mingle with a wide range of people on the cutting edge of technology

It was held up near Auckland, New Zealand area, and it was an incredibly fun weekend. The Webstock conference was held the same weekend, so a lot of the people in town for that conference also came to Kiwi Foo. This was great, as I got to meet several interesting people who might not have been there otherwise.


View Larger Map

One thing I regret a tad, is that I didn't do a talk on anything in Python, as the board filled up so quickly I couldn't squeeze anything in by the time I decided to talk about something. I guess this is one of the lessons you learn as a newbie to a Foo style camp, you have to be quick!

One of the interesting things about being one of the few, if not only, people who made there living programming in Python was to listen to some of the conversations about Python that sprung up here and there. Google App Engine in particular seems like it is bringing a huge assortment of people to the Python language, and these aren't your everyday Joe Blows, a lot of them help make the technical decisions for their company.


While I wasn't able to give a talk on Python, I do think I convinced quite a few people to take a second look. My "sales technique", that I have been polishing for a while, revolves around getting someone to easy_install the IPython shell, and then walking them through how to write a program in Python. If you haven't tried this technique, yet, do, it is has a very big wow factor.

To wrap up, I really enjoyed myself, and was happy to hear so much talk about Python, as a fly on the wall in a country around the world. If am so lucky to be invited back to Kiwi Foo next year, I would be a very happy man, and, more importantly, I would shove in a talk on Python Generators, right after I dropped off my bags.

A couple of funny things to note on the way out was that the whole camp had a group debate in auditorium on the topic, "Is New Zealand Fucked?". It was a riot, and worth the trip alone. Those kiwi have foul mouths, I think one person managed to fit "Fuck" into a sentence 6 times, and I love them for it. Second, when one of many discussions on App Engine popped up, I heard someone say, I think it is going to fail. I replied, I hope not, because I have been writing a book about it for the last year :)

We will be presenting a half-day tutorial on repoze.bfg at the 2009 Plone Symposium East, hosted by the awesome WebLion group at Penn State. The tutorial is a hands-on (bring a laptop, and be ready to code) introduction to developing applications using BFG.

The tutorial runs on Thursday, 26 May 2009 (exact times TBD). The cost for the tutorial is $75.00. Register now!

Paul will also be talking about an ongoing BFG customer project, and how it relates to Plone. This talk starts at 11:00 AM on Friday, 28 May 2009.

Hope to see you there!

On the Python Artificial Intelligence SIG, the topic of the week was statistics versus logic.

Graham Higgins dropped quite a few links:

[1] http://languagelog.ldc.upenn.edu/nll/?p=1374
[2] http://en.wikipedia.org/wiki/Conditional_entropy
[3] http://www.safarmer.com/fsw2.pdf
[4] http://www.cs.washington.edu/homes/rao/ScienceIndus.pdf
[5] http://homepages.inf.ed.ac.uk/s0450736/maxent_toolkit.html
[6] http://r.research.att.com/tools/
[7] http://www.amazon.com/Gods-Demons-Symbols-Ancient-Mesopotamia/dp/0292...
[8] http://www.prehistory.it/ftp/winn.htm
[9] http://www.flonnet.com/fl1801/18010730.htm
[10] http://blog.wired.com/wiredscience/2009/04/indusscript.html
[11] http://www.newscientist.com/article/dn17012-scholars-at-odds-over-mysterious-indus-script.html
[12] http://www.acl-ijcnlp-2009.org/main/acceptedfullpapers.html

Rick Thomas responded back with:

http://code.google.com/p/pebl-project/
http://code.google.com/p/python-dlp/
http://www.betaversion.org/~stefano/linotype/news/275/
http://ite.gmu.edu/~klaskey/papers/Costa_Laskey_MEBN_wo_Tears.pdf

AIPY Mailing List: http://groups.google.com/group/aipy?hl=en

Calvin Hendryx Parker of Six Feet Up writes a blog entry about integrating the Zine blog engine together with Plone and Deliverance, using repoze.zope2. The blog entry itself is hosted on that setup.

One of the things I don't like about closures [1](via nested functions) is how they obscure intent in code. For example, if you just want to retain state why use a closure if you could just use a class? Sure a closure sounds cooler, but a class or a regular group of functions is often more flexible and readable than a closure.


Example 1: Simple Persistent State
Closure That Stores State: (1A)

In [39]: def outer():
   ....:     x = 1
   ....:     def inner():
   ....:         return x
   ....:     return inner
   ....: 

In [40]: func = outer()

In [41]: func
Out[41]: 

In [42]: func()
Out[42]: 1

Class That Stores State (1B)

In [46]: class State(object):
   ....:     def __init__(self):
   ....:         self.x = 1
   ....:     def func(self):
   ....:         return self.x
   ....:     
   ....:     

In [47]: func = State().func

In [48]: func()
Out[48]: 1

Score: +1 Class
Summary: If you simply want to persistent state, why use a closure that
has an odd signature, when you can simple use a class?


Example 2: Flat is better than nested, closures taken to an extreme with multiple nesting

Nested function w/ function that operates on state (2A)

In [18]: def way_outer():
   ....:     wo = 1
   ....:     def inner():
   ....:         i = 2
   ....:         def way_inner():
   ....:             return i + wo
   ....:         return way_inner
   ....:     return inner
   ....: 

In [19]: out = way
way_inner  way_outer  

In [19]: out = way_outer()

In [20]: out
Out[20]: 

In [21]: out()
Out[21]: 

In [22]: way_way_out = out()

In [23]: way_way_out()
Out[23]: 3

Class functions that operate on state (2B)

In [24]: class Addition(object):
   ....:     def __init__(self):
   ....:         self.x = 1
   ....:         self.y = 2
   ....:     def add(self):
   ....:         return self.x + self.y
   ....:     
   ....:     

In [25]: a = Addition().add

In [26]: a
Out[26]: [bound method Addition.add of __main__.Addition object at 0x78db70]

In [27]: a()
Out[27]: 3

Score: +2 Class
Summary: You might be thinking "duh" with this example, but yes, that is exactly the point! Why nest things if you don't have to? Especially because of the fact that if someone looks at your highly nested closure they will go WTF as they should. Why make things more complex then they need to be? Just because you can use closures doesn't mean you should or it makes your code intuitive and readable.

Example 3: Delaying Execution of a Function
Delayed execution of a function using closures (3A)

In [29]: def outer():
   ....:     def inner(x):
   ....:         return x + 1
   ....:     return inner
   ....: 

In [30]: func = outer()

In [31]: func(3)
Out[31]: 4

Delayed execution of a function using class: Example (3B)

In [1]: class DelayCall(object):
   ...:     def __init__(self, x):
   ...:         self.x = x
   ...:     def logic(self):
   ...:         return self.x + 1
   ...:     def delay(self):
   ...:         return self.logic
   ...:     
   ...:     

In [3]: d = DelayCall(3)

In [4]: func = d.delay()

In [5]: func()
Out[5]: 4

Delayed execution of a function that takes args using a lambda in a class: Example (3C)

In [19]: class LambdaDelayCall(object):
   ....:     def logic(self, x):
   ....:         return x + 1
   ....:     def delay(self):
   ....:         return (lambda x: self.logic(x))
   ....:     
   ....:     

In [20]: l = LambdaDelayCall()

In [21]: func = l.delay()

In [22]: func(5)
Out[22]: 6

Delayed execution of a function that takes args using a lambda in a class w/ __call__: Example (3D)

n [19]: class LambdaDelayCall(object):
   ....:     def logic(self, x):
   ....:         return x + 1
   ....:     def __call__(self):
   ....:         return (lambda x: self.logic(x))
   ....:     
   ....:     

In [37]: l = LambdaDelayCall()

In [40]: func = l()

In [41]: func(3)
Out[41]: 4

Score: +3 Class
Summary: While 3A is shorter, I think it is much less clear then 3B, 3C or 3D, the class examples. I think of a closure in terms of an outer function that needs to operate on an inner function, such as in the case of a decorator. Using a closure just to delay execution of a function doesn't seem right to me, and I feel like it obscures the code.

Conclusion:

Closures are useful when you want to modify the state of inner function and return, as in the case of decorators. Lambdas, from Learning Python 3rd edition, " are often used as a way to inline a function definition, or to defer execution of a piece of code.". While closures are useful and powerful, with power comes responsibility. [1] Nested functions aren't that clear, try a class or maybe you don't even need a closure?. Make sure you Python for good, not evil. Stay tuned for the next installment.

[1] Added note about "nested functions" 04/29/2009


References:
Bruce Eckel on Decorators
Zen of Python
Functools Partial
Dive Into Python: Lambda

I am going to attempt to do weekly updates based on the newly formed Python AI SIG, http://groups.google.com/group/aipy . Sunday, New Zealand time, is going to be my day to summarize what is going on.

So far the hot topics are as follows:

1. Getting a code repository setup somewhere to share ideas.
2. Getting some domain setup so we can share and categorize what we learn. Jeff Rush mentioned possibly ai.python.org and using Sphinx. I kind of like that idea.
3. Filtered RSS reader: It seems like getting a bot to pre-filter RSS is low hanging fruit.
4. Continuous monitoring of the body: We are discussing the feasibility of continuous monitoring of the body, one gotcha so far is what device do we use?

Beaker 1.3 is out, actually, its been out for awhile and I’m just not getting around to blogging the fact. It’s a shame I’ve been a bit too busy lately to blog this earlier because in addition to some bug fixes it has some nice new features that make it even easier to use in any Python script/application/framework.

First, to air my dirty laundry, the important bug fixes in Beaker 1.3:

• Fixed bug with (non-cookie-only) sessions not being timed out
• Fixed bug with cookie-only sessions sending cookies when they weren’t supposed to
• Fixed bug with non-auto sessions not properly storing their last accessed time

The worst thing with the first two of these is that they were regressions that snuck in despite unit tests that exercised the code fairly decently. They’re fixed now along with more comprehensive tests to help prevent such regressions occurring again.

Beaker has always had session’s, and caching, but except for Pylons I’ve yet to see anyone actually use Beaker’s caching utility. I’ve seen the SessionMiddleware used in other WSGI based frameworks, but not the caching, which is kind of a shame since it:

• Supports various backends: database, file-based, dbm file-based, memcached
• Has locking code to ensure a single-writer, multiple reader model (This avoids the dreaded dog-pile effect that caching systems such as the one in Django experience!)

For clients that hit the cached function while its already being regenerated, Beaker serves the old copy until the new content is ready. This avoids the dog-pile effect, and keeps the site snappy for as many users as possible. Since the lock used is disk-based though, this does mean you only avoid the effect per machine (unless you’re locking against NFS or a SAN), so if you have 20 machines in a cluster, the worst the dog-pile effect can get is that you’ll have 20 new copies generated and stored.

Now, in Beaker 1.3, to try and encourage its use a bit more, I’ve added a few decorators to make it easier to cache function results. Also with Mike Bayer’s suggestion, there is now cache regions to make it easier to define various caching policy short-cuts.

Cache Regions

Cache regions are just pre-defined sets of cache instructions to make it easier to use with your code. For example many people have a few common types of cache parameters they want to use:

• Long-term, likely to a database back-end (if used in a cluster)
• Short-term, not cached as long, perhaps to memcached

To set these up, just tell Beaker that about the regions you’re going to define, and give them the normal Beaker cache parameters for each region. For example, in this Pylons app, I define 2 cache regions in the INI:

beaker.cache.regions = short_term, long_term
beaker.cache.short_term.type = ext:memcached
beaker.cache.short_term.url = 127.0.0.1:11211
beaker.cache.short_term.expire = 3600

beaker.cache.long_term.type = file
beaker.cache.long_term.expire = 86400

Note: For those wondering about multiple memcached servers, just put them in as the url with a semi-colon separating them.

If you want to use the caching outside of Pylons without middleware (ie, as a plain library), that’s a bit easier now as well:

from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options

cache_opts = {'cache.data_dir': './cache',
              'cache.type': 'file',
              'cache.regions': 'short_term', 'long_term',
              'cache.short_term.type': 'ext:memcached',
              'cache.short_term.url': '127.0.0.1:11211',
              'cache.short_term.expire': '3600',
              'cache.long_term.type': 'file',
              'cache.long_term.expire': '86400',
}

cache = CachManager(**parse_cache_config_options(cache_opts))

And your cache instance is now ready to use. Note that using this cache object is thread-safe already, so you just need to keep one around in your framework/app (Can someone using Django explain where you’d keep a reference to this object around so that you could get to it in a Django view?).

New Cache Decorators

To make it easier to use caching in your app, Beaker now includes decorators for use with the cache object. Given the above caching setup, lets assume you want to cache the output of an expensive operation:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.region('short_term', 'mysearch')
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

The second argument to the region decorator, ‘mysearch’. That isn’t required unless you have two function’s of the same name in the same module, since Beaker records the namespace of the cache using the function name + module + extra args. For those wondering what a Beaker namespace is, its a single cache ‘block’. That is, lets say you wanted to cache 4 versions of the same thing, but change them differently depending on the input parameter. Beaker considers the thing to be a namespace, and the things that change the thing being cached are the cache keys.

Only un-named arguments are allowed on the function being cached. These act as the cache keys so that if the arguments change, a new copy is cached to those arguments. This way you can have multiple versions of the function output cached depending on the argument it was called with.

If you want to use arbitrary cache parameters, use the other decorator:

# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache

def regular_function():
    # Do some boring stuff

    # Cache something
    @cache.cache('mysearch', type='file', expire=3600)
    def expensive_search(phrase):
        # Do lookup with the phrase variable
        return something
    return expensive_search('frogs')

This allows you to toggle the cache options per use as desired.

If there’s anything else I can do to make it easier to use Beaker in your application, be sure to let me know (Yes, I know more docs would help, this blog post was a first attempt to help out on that front, more docs on the way!).

I have been hacking on a Commandline tool and Web Service Generation framework called Adapt. Last night, Adam Shand, and I hacked on getting Phase 2 together, which is making a web service automatically serve out the same URLs as a commandline tool options, which are defined in a config file.

The not even Alpha quality code is here: http://bitbucket.org/noahgift/adapt/

The basic problems I am trying to solve is this:

1. Automatically convert existing Python command line tools into Web Services.
2. Create a plug and play WSGI application so I can talk to other WSGI goodies like LDAP auth, debugging, etc.
3. Create Yet Another Web Framework in Python (although it is pretty unconventional)
4. Automatically adapt existing shell scripts, aliases and Python scripts into a dynamically created command line tool.
5. Declarative mapping of actions via a simple config file.
6. Let people that are completely ignorant of Python, create command line tools and web services by running a tool.
7. Let people do BOTH, in one step.

Is this crazy...yes. But, it is also fun. Once I complete code coverage to 100% and move common code into a library, and make some "frameworkish" code, I will release it. Feel free to help if you think this sounds fun too.


I named it after my blog post "Adapt or Die". Because it was a direct outshoot of my philosophy on tools.

From the title of my blog,"Artificial Code", you can tell I am interested in Artificial Intelligence. I have noticed quite a few people in the Python community are interested in AI, so wondered if there was a mailing list, evidenced by viewing Raymond's AI talk at PyCon, and a recent blog post by Tennessee, and even Guido himself :). My google searches revealed there wasn't, so I started one:

http://groups.google.com/group/aipy

I think it would be fun to share ideas about artificial intelligence and then make real world implementations. We can also share how everything thinks that people interested in AI are crazy and that it is an unsolvable problem :) If you are interested in both AI and writing Python implementations, then join the list, and let's get to work.

Some of the things I am interested in are solving "low hanging fruit" problems that can be done this year, or sooner.

Recently I ran nosetests and it took 26 seconds to run. On looking at an strace and it generated about 53K lines of output in strace on ubuntu.

After some help from a couple of co-workers, I noticed that I could get the following performance boost, note this is one a massive NFS installation, not my laptop...

Total Elapsed Time: 2066 % speed improvement
Lines of strace output: 3050| 1695 % reduction in calls to file system

This was accomplished through a series of tests:

#1 Hijack setuptools by explicitly crafting sys.path to include only the eggs it needs. If you don't use --multi-version it will traverse EVERY egg and put them in your sys.path.
#2 unset my Python path eliminating all of the places that were there by default
#3 write a bootstrap script that calls python -S "myscript.py" so site-packages don't get looked at.

I was pretty shocked at how big of an improvement this made. The strace output shrunk to about 3K lines, which I might even be smaller if I only looked at open files.

There are a few downsides to this, obviously flexibility, but I need the speed. Has anyone done anything similar to hijack Python import to speed up time to fire up an interpreter. My hunch is I can go much, much farther.

On behalf of the Zope community, I am pleased to announce the creation of the "Zope 4.0" project. After extensive discussion with the Zope wizards in conclave at PyCon 2009, the new project's website has been launched!

(Note, sorry Planet Python for the previous, accidental cross post from my workout blog...)


One thing I had almost forgot to blog about, ReSTless
. One of my Atlanta area friends, Aaron Hilligass wrote a really sweet, and useful app to demo how to call Python from Cocoa in our book, Python For Unix and Linux System Administration. It turns to be a really, really useful application that I use all the time to preview Restructured text while I write it.

I have a link to the source code at the bottom of this post, as well as a link to the docutils source. One thing to watch out for, is to make sure you adjust the MyDocument.m source file to look at where you actually have rst2html.py. In my situation I used this location when I compiled:

    [task setLaunchPath:@"/usr/local/bin/rst2html.py"];

One of the other cool thing that it does is show you how to connect to a piped Python script. Nice work Aaron!









References:
Python For Unix and Linux System Administration Book
Source Code For Python For Unix and Linux System Administration
Latest Docutils Source Code

Monday/5k, Tuesday/6k, Wed/Off, Thursday/6k, Friday/6k, Sat/Off, Sun/25K+
Feeling: 5 days is pretty decent although I could probably do 6. I am very glad I took Saturday off as I had a fairly easy go at running 25k or 16 miles. Here is the map of my hella sweet 25k run around the tip of the North Island of New Zealand:
Notes: I got a tad thirsty, but stopped for water a public faucet. I also had some pretty killer death, water defying jumps over rocks. This old man still has the juice :)
http://mapmyrun.co.nz/view_my_run_127983

I was just talking to some people recently about running and how the human body adapts to ever increasing loads. Most people can run marathons if they just expose their body to the load of running on a regular basis, it isn't that tricky. Most of those people could probably run an ultramarathon, if they trained for it, I am training for one next year here. There are even some people like Dean Karnazes, that run 50 marathons in 50 states, in 50 consecutive days.

One of the reasons I like running is it continuously demonstrates, on a daily basis, no less, to me the simple axiom, "Adapt or Die". If I stop running I will get fat and out of shape, but if keep running I will keep adapting. One of the things that shocks first time runners is how they become aware that they can all of sudden run for an hour and a half and they forget they were even running. This is because our bodies are amazing things that adapt to what we expose it to. If you run enough, it becomes easy to run for hours on end with fairly little effort.

I want that same adaption in the tools I use. I have been using Unix and Linux for over a decade now, and I like its adaptability. You can reuse existing tools and creates pipes, or write your own tools and pipe them into existing tools. It is a surprisingly flexible and powerful system. I like Python because it continuously changes itself, incorporating new modules like the multiprocessing module in Python 2.6, or even completely reinventing itself like it is doing in Python 3k, converting to Unicode, and adding many other backwards incompatible changes.

To me WSGI is also an adaptable tool. To me it truly symbolizes "adapt or die", in both implementation and philosophy. Because WSGI is both a philosophy and a technology it draws a different crowd of developers than a pure web framework crowd. Developers that are drawn to WSGI want to use a constantly evolving best of breed component stack. For example, SQLAlchemy is developed by a very passionate developer, Michael Bayer, who obviously takes pride in knocking the shit out of one particular problem. Jonathan Ellis solved another problem when he wrote SQL Soup, which dynamically creates mapped objects from on the fly database queries with ZERO mapper code, this IS rocket science! Those are the kind of tools I want to use. They are flexible like unix pipes, and they work in lots of different contexts.

One of the downsides to WSGI, and WSGI inspired frameworks, like Pylons, Turbogears 2, BFG and Werkzeug is that they are a bit more difficult to deal with, partially, because they don't hide as much of the complexity as something like Django.

On the other hand, I feel that using something like Django or Ruby on Rails is a form of Technical Debt. Technical debt, can be an extremely useful concept, just like business debt, as long as you can get a bigger total payoff then the eventual interest you will need to pay on your "loan". One of the side effects of hiding complexity is some form of "lock-in" to the "Django" or "Rails" way, which is a bit too Alan Greenspan on an Ayn Rand Objectivism high for me. The Fountainhead was a cool book, but it was fiction after all. To me web frameworks often offer a false choice to a problem I don't have.

WSGI inspired frameworks, or just plain tools, like WebOb, urlrelay, repoze.who, are a different kind of philosophy. They remind me a lot of tools like ls, cat, rm, etc. They do a specific job, but they can also work in a vast array of combinations the authors never dreamed of. In the hands of experienced developers, WSGI tools and frameworks, just like unix tools, are quite handy. Many of these developers are using WSGI, but you just don't hear about it. Take SQLAlchemy book author Rick Copeland, who uses a tweaked out version of Turbogears 2.0 to write massively complex web apps for Predictix, or Jonathan, Metaclass, LaCour, who solves mind strangling problems in SQLAlchemy, Elixir, Pylons and Turbogears for ShootQ and major Airlines...yup, those kind of airlines. If your doing something crazy in WSGI, hopefully you comment on this post.

I am predicting 2009 will be the year of WSGI, where the WSGI frameworks and tools ultimately establish their philosophical and technical superiority over the walled garden development philosophy, or Ian Bicking's Solution 1 in his 2005 WSGI presentation. If your a Python developer, adapt....or die...

References

Pylons A Hacker's Framework: http://magazine.redhat.com/2008/11/05/introducing-pylons-a-hackers-web-framework/
Multiprocessing module: http://www.ibm.com/developerworks/aix/library/au-multiprocessing/
Intro to SQLAlchemy: http://www.ibm.com/developerworks/aix/library/au-sqlalchemy/
SQLAlchemy Book: http://oreilly.com/catalog/9780596516147/
Pylons Book Hard Copy: http://www.apress.com/book/view/9781590599341
Pylons Book Online: http://pylonsbook.com/
Ian Bicking WSGI Presentation 2005: http://ianbicking.org/docs/pycon2005/wsgi-presentation/slides.html

We are lookking forward to working with a number of folks at the sprints following Pycon this year.

There are seventeen folks signed up to date to work on Pylons, Repoze, TurboGears, and the other WSGI-centric Python web freamworks. Once exciting opportunity is to have a lot of cross-pollination between the frameworks, looking for ways to work together and share more code.

Some of the friction which comes up on the zope-dev list, apparently due to different goals, might in fact be due to some confusion in the terms we use to talk about our goals, and about the shared software we manage toward those goals. In the interests of reducing the friction, I would like to sketch out how I am using those terms.

Mikko Ohtamaa provides a blog entry about how to set up Plone 3.2 under repoze.zope2.

I was just reading, by pure chance, a post by Michele Simionato, in which he mentioned that twill, at least at the time of the article, packaged inside of it's package the two dependencies it used. I think this is a wise choice.

I think it is about time the Python community embraces the fact that dynamically configuring dependencies during network installation is a really, really bad idea for end users of that library. For the actual developers of the library, using something like easy_install is absolutely brilliant, as it allows you to automatically pull in packages that your library depends on. It should stop here though.

From an engineering standpoint, if you want to ensure something like 99.99999% reliability for your product, you don't perform network transactions with clients in an unknown state. You do that BEFORE you release the package to the general public, and then you release that bundle to the general public.

There is a downside to this, as it makes for more work for the developers of the library. They have to actually download each of the dependencies, test their library, and package it up as a big tar file. In addition, a user cannot easily swap out one of their dependencies, they would need to download the bundle again. I think this hassle far exceeds the "flexibility" that is currently offered. What makes it worse, is that many developers of their libraries have no idea that they turn away hundreds or thousands of users away from their framework or library, because they do a complex network transaction with clients in an unknown state and things blow up.

If your a developer of a web framework or library that depends on a huge pile of dependencies, consider offering a plain tar file prepopulated with all of your dependencies, or a plain tar file with a bootstrap.py of virtualenv that bootstraps the files inside of your tar file. It looks like Pip is a step in the right direction, but I still think the burden should be placed on the developers, NOT the end users of the library. If a package install blows up during a download or while downloading, the author of the package is the best person to troubleshoot this, not the thousands of people who try out the package for the first time.

Please, stop the insanity!

At the third ever Wellington, NZ Python User Group meeting, we talked about the multiprocessing module, command line tool tricks and visualization. It was a hoot, here are the notes.

First, me and a couple co-workers, Teijo Holzer and John van Leeuwen, gave a three part presentation on the multiprocessing module.

I gave an example of tool I wrote to asynchronously fork Net-SNMP called "multicore snmp", Teijo gave a presentation on shared data structures, which are available via a simple import of Process, Value, Array.

Finally, John gave a presentation about how generator expressions fit quite nicely with Process Pools. Dust up on the nested generator expressions, because functional programming is back with a bang. All in all the crowd was quite blown away with how "sick" the multiprocessing module was, in a good way.

Next, Stephen Judd, gave a talk on how he creates a powerful boilerplate template for command line tools using optparse, profile, doctests, and friends. He promised to post the code to the NZPUG wiki.

Finally, Richard Clark, gave a quick presentation on Event Timeline charting. He showed some pretty cool graphs that generated based on timeline's using I believe Chart Directory...but I could be wrong. Finally, he also mentioned Panda 3D, which graphs some killer 3D and it is has a simple Python API, check out the Hello World example!

Start slight diatribe:

We wrapped up the meeting with some impromptu talk on Web Frameworks, and I mentioned Pylons/Turbogears is the one to watch at the moment. For me WSGI and WSGI middleware like Repoze.who or Urlrelay + SQLAlchemy + a really experienced, mature, and smart developer community, makes Pylons enterprise ready from day one, for almost any task you could throw at it. Combine that with SQLSoup, which can do crazy stuff like doing a join on two different tables on two different databases living on the same server in MySQL, and it makes it a no brainer for almost anything I want a web framework for.

Here now, is some proof about Pylons being able to do incredible stuff. One slightly secret, skunkworks, Turbogears 2.0/Pylons project is ShootQ,which was created by Python mastermind, Jonathan LaCour. I totally agree with Adrian Holovaty, when he says, it is not about the tools you use, it is about the sites you create.

ShootQ rocks, and Jonathan took the enterprise quality toolset of Turbogears 2.0/Pylons, and created an awesome showcase site for anyone that wants to question if Turbogears 2.0/Pylons is the real deal. Banks, film studios, and airlines, as well as Web 2.0 companies are using it, and not only does it work well, but it solves complex problems that are not easily solvable in other web frameworks! I like to call it the hacker's framework, but you could also get a way with calling it Python's enterprise web framework too. I would even go so far as to say, if you work at a major corporation and you decided to use a Python webframework, you need to look at Pylons, or you haven't done due diligence. It is that good.

End slight diatribe:

Back on topic, if your on our side of the world in Wellington on the second Thursday of each month...drop by!

I must at first admit some bias with this review, as Tarek and I both had Shannon -jj Behrens as technical editors on our books: Expert Python Programming and Python For Unix and Linux System Administration. Packt Publishing also gave me a free copy, which was nice. This kind of cancelled itself out though, as I liked it so much I bought a PDF version so I could quickly refer to it over and over again.

I think the book rocks! I learned a lot of real world stuff that I had not been exposed to before reading his book. It is a much different book than say, the Python Cookbook, or Learning Python, but that is a good thing! It is good to read widely different types of books on Python. I read a lot of books, probably too many in fact, and I can't think of a Python book that I read that I didn't like in some way.

Having written a book, actually, I am on my second, I can attest it is tough. Python is a language that most, if not all, people taught themselves outside of work time. Python authors are perhaps some of the most abnormally motivated people you will ever meet as result, and really haven't read a bad Python book yet. This book in particular showcase Tarek's unique set of skills as a programmer. I gather this book is modeled after exactly how he writes code, and that is a good thing!

I was especially happy with how Tarek covered things tangentially associated with Python like buildbot, and mercurial, as it means I could drop this book off to someone in a company, and say, "take a look at a book that covers the latest developer techniques in Python", and be satisfied they could stick with just reading that book for a while, and learn a whole lot about real world Python programming.

If you are a serious Python programmer, or want to be a serious Python programmer, this is a book you need to have on your bookshelf. Thanks for writing it Tarek!

I did my civic duty and filled out the Python Packaging Survery setup by Tarek and Massimo. Here were my biggest issues:

1. End users of a library should not download a library, say a web framework, which then downloads a horde of components. This is just so, so wrong from an engineering standpoint. Imagine configuring the Space Shuttle parts as it began to launch. This should happen at some point, sure, but not EVERY SINGLE TIME you install a library. The failure rate for some applications installing from PyPi for me is over 70%, when it should be close to zero.

2. It is damn tricky to register a package on PyPI. Why?

3. CPAN has a like a billion mirrors. We have PyPI.

4. There is no real quality control, or feedback mechanism for a package. It would be AWESOME, if the install mechanism sent a traceback up on the same page as the package entry on PyPI, as a mark of shame, when a package failed to install. I bet there are some packages that would get buried in tracebacks.

5. Operations are not atomic. Often I will get partially through an install with easy_install, something blows up, and then I have a big pile of Python to sift through.

6. There should be some parity with operating system package management tools. Why not have the package manager also make RPM's automatically, and let them be aware of each other? I like RPM's and deb packages....they just work.

7. Put up a customer satisfaction survey up on the site. Ban anonymous comments, but allow registered users, under the real name, say what they thought. Did the package suck beyond all recognition, or was it manna from heaven?

Btw, I really enjoy easy_install most of the time, but when I have to deal with something that pulls down 50 other things, I want to scream...my God, what have you done, why, why.....

I have been meaning to write a review on Essential SQLAlchemy by Rick Copeland for a while now, so I am just going to do it.

Recently, I have been writing code that needs to talk to legacy databases. The kind of work I have to do, usually requires the creation of a bunch of command line tools, and so using a standalone ORM is really the only option. I pulled out my copy of Essential SQLAlchemy and got to work.

First, I read through the beginning of the book, and then skimmed through examples of some of the queries. Finally, I got into the SQLSoup section in Chapter 10, and really got down and dirty. If you haven't yet, used SQLSoup, and you have a legacy database, then you are in for a treat. Rick describes how powerful and easy SQLSoup is to use in Chapter 10, and gives it a great treatment.

Basically, you do something like this:

from sqlalchemy.ext.sqlsoup import SqlSoup
db = SqlSoup('mysql:///SarahPalinIsANutter.db')

You then simply query Tables as attributes, and you get back objects "magically"

for object in db.ShootingWolves.all():
   print object.FiftyCaliberAmmo

One topic I wish Rick would have covered more though were how to do joins on different databases on the same database server...especially if this is possible with SQLSoup. Jonathan Ellis if your out there how do I do this equivalent SQL with two MySQL Databases with SQLSoup:

SELECT p.ShotID, s.shot_ID FROM Production.Production p JOIN
Shots.Shots s on p.ShotID = s.shot_ID;

Update: You can do this. I just didn't know how:

db.schema = 'production'
p = db.production
db.schema = 'shots'
s = db.shots

Jonathan Ellis, has this new piece of SQLSoup goodness in the trunk now:

p = db.entity('production', 'production')
s = db.entity('shots', 'shots')

where entity is

def entity(self, tablename, schema=None):

Excellent! I love SQLSoup even more, if that was possible.



I also wrote an entry level article on SQLAlchemy here, for IBM Developerworks. I do cover the declarative syntax, so you can refer to this article, the book, and the excellent online docs for SQLAlchemy and be pretty much set.

Update: Oh, and you should buy this book. It is a must have for any SQLAlchemy developer!

I was showing someone python today who has a background in Perl, and got to the topic of how the property decorator was nice syntactic sugar to generate a read attribute that gets calculated when the attribute was accessed. He accidently thought of this example next:


In [1]: class EvilPython(object):
...: def __init__(self, x = 3):
...: self._x = x
...: @property
...: def x(self):
...: return self._x * random.random()

In [2]: import random

In [29]: val = EvilPython()

In [30]: val.x
Out[30]: 2

In [31]: val.x
Out[31]: 2

In [32]: val.x
Out[32]: 1

In [33]: val.x
Out[33]: 0

In [34]: val.x
Out[34]: 1



I had never thought to try to make "Evil" out of using properties, but creating random values for an attribute of an object could really frustrate someone for quite some time. Don't try this at home!

Hey, imagine, you want to run the setup.py of a Distutils-packaged package and your current directory is not the package's directory:

  [chrism@vitaminf env26]$ bin/python ../supervisor2/setup.py develop
  running develop
  error: error in 'egg_base' option: 'src' does not exist or is not a directory

Oh well of course. That makes sense. This is a really advanced use case. I mean, it's really hard to... you know..... not do the most idiotic thing on the planet. So, let's just cd to the package directory when we want to run its setup.py, over and over, forever, til the end of time.

Sometimes I think we might have been better off without any Python packaging solution.

I’m pleased to announce after a rather lengthy release candidate period, that Pylons 0.9.7 is finally out. Pylons 0.9.7 brings a good amount of changes to Pylons from 0.9.6 while still retaining a fairly hefty amount of backwards compatibility to ensure a mostly painless upgrade.

Some helpful documentation on the new release:

• Upgrading to Pylons 0.9.7
• Pylons 0.9.7 Documentation
• Pylons 0.9.7 Docs in PDF
• Pylons Book (Will be updated soon with the final copy from the printed book)

Major changes in 0.9.7:

• Switched to using WebOb for the request/response object
• Various performance improvements to object initialization
• Beaker and Routes updates
• Middleware improvements, and optimizations

This is a huge step forward for Pylons, and I’d like to thank all of the contributers who have helped make Pylons what it is today. We’ve knocked off more bugs for this release than any before, which shows just how far the Pylons community has come:

• 0.9.5 tickets: 45
• 0.9.6 tickets: 64
• 0.9.7 tickets: 160

And we have finally made a huge dent in the historical “lack of docs” problem that Pylons previously suffered from with the new Sphinx generated docs and a comprehensive Pylons book.

The full changelog which describes the major changes (Look for the bits marked with WARNING that might affect backwards compatibility).

0.9.7 (February 23, 2009)

• WARNING: A new option is available to determine whether or not an actions arguments should be automatically attached to ‘c’. To turn off this implicit behavior in environment.py: config[‘pylons.c_attach_args’] = False This is set to True by default.
• WARNING: Fixed a minor security hole in the default Pylons error page that could result in an XSS security hole.
• WARNING: Fixed a security hole in the default project template to use the StaticURLParser to ensure arbitrary files can’t be sent.
• WARNING: Refactored PylonsApp to remove legacy PylonsApp, moved session/cache and routes middleware into the project template. This will require projects to be updated to include those 3 middleware in the projects middleware.py.
• Changed to using WebTest instead of paste.fixture for app testing.
• Added render_mako_def to render def blocks within a mako template.
• Changes to cache_decorator and cached_template to support Beaker API changes in version 1.1. 1.0.3 is still supported.
• Fix HEAD requests causing an Exception as if no content was returned by the controller. Fixes #507. Thanks mvtellingen, Petr Kobalicek.
• Fix a crash when returning the result of ``etag_cache`` in a controller. Fixes #508.
• “response” flag has been removed from pylons.decorators.cache.beaker_cache, as it sends all headers along unconditionally including cookies; additionally, the flag was taking effect in all cases previously so prior versions of beaker_cache are not secure.

  In its place, a new option “cache_headers” is provided, which is a tuple of specific header names to be cached. It defaults to (‘content-type’,’content-length’).

• “invalidate_on_startup” flag added to beaker_cache, which provides a “starttime” to the cache such that when the application is started or restarted, the cache entry is invalidated.
• Updating host to use 127.0.0.1 for development binding.
• Added option to specify the controller name with a controller variable in the controller’s module. This name will be used for the controller class rather than the default naming scheme.
• setup.py egg_info now restores projects’ paster_plugins.txt, allowing paster shell to work again after the egg-info directory was lost. fixes #282. Thanks sevkin.
• The paste_deploy_config.ini_tmpl template is now located at package/config/deployment.ini_tmpl for new projects.
• Project’s default test fixtures no longer hardcode test.ini; the ini file used can now be specified via the nosetests—with-pylons argument (defaults to test.ini in setup.cfg). fixes #400.
• @validate now defaults to translating FormEncode error messages via Pylons’ gettext catalog, then falls back to FormEncode’s. fixes #296. Thanks Max Ischenko.
• Fixed SQLAlchemy logging not working in paster shell. Fixes #363. Thanks Christoph Haas.
• Added optionally engine initialization, to prevent Buffet from loading if there’s no ‘buffet.template_engines’ in the config.
• Updated minimal template to work with Tempita and other new templating changes.
• Fixed websetup to parse location config file properly when the section isn’t ‘main’. Fixes #399.
• Added default Mako filter of escape for all template rendering.
• Fixed template for Session.remove inclusion when using SA. Fixed render_genshi to properly use fragment/format options. Thanks Antonin Enfrun.
• Remove template engine from load_environment call.
• Removing template controller from projects. Fixes #383.
• Added signed_cookie method to WebOb Request/Response sub-classes.
• Updated project template to setup appropriate template loader and controller template to doc how to import render.
• Added documentation for render functions in pylons.templating.
• Adding specific render functions that don’t require Buffet.
• Added forward controller.util function for forwarding the request to WSGI apps. Fixes #355.
• Added default input encoding for Mako to utf-8. Suggested in #348.
• Fixed paster controller to raise an error if the controller for it already exists. Fixes #279.
• Added init.py to template dir in project template if the template engine is genshi or kid. Fixes #353.
• Fixed jsonify to use application/json as its the proper mime-type and now used all over the net.
• Fixed minimal template not replacing variables properly. Fixes #377.
• Fixed @validate decorator to no longer catch exceptions should they be raised in the action that is supposed to display a form. Fixes #374.
• Fixed paster shell command to no longer search for egg_info dir. Allows usage of paster shell with installed packages. Suggested by Gavin Carothers.
• Added mimetype function and MIMETypes class for registering mimetypes.
• WARNING: Usage of pylons.Response is now deprecated. Please use pylons.response instead.
• Removed use of WSGIRequest/WSGIResponse and replaced with WebOb subclasses that implement methods to make it backwards compatible with the Paste wsgiwrappers.
• Fixed missing import in template controller.
• Deprecated function uses string substitution to avoid Nonetype error when Python optimization is on. Fixes #334.
• E-tag cache no longer returns Content-Type in the headers. Fixes #323.
• XMLRPCController now properly includes the Content-Length of the response. Fixes #310, thanks Nicholas.
• Added SQLAlchemy option to template, which adds SQLAlchemy setup to the project template.
• Switched project templating to use Tempita.
• Updated abort/redirect_to to use appropriate Response object when WebOb is used.
• Updated so that 404’s properly return as Response objects when WebOb is in use instead of WSGIResponse.
• Added beaker_cache option to avoid caching/restoring global Response values that were present during the first cache operation.
• Adding StatusCodeRedirect to handle internal redirects based on the status code returned by the app. This replaces the use of ErrorDocuments in projects.
• Refactored error exceptions to use WebError.
• WSGIController now uses the environ references to response, request, and the c object for higher performance.
• Added optional use of WebOb instead of paste.wsgiwrapper objects.
• Fixed bug with beaker_cache defaulting to dbm rather than the beaker cache app-wide default.
• The—with-pylons nose plugin no longer requires a project to have been registered with setuptools to work.
• The config object is now included in the template namespace.
• StaticJavascripts now accepts keyword arguments for StaticURLParser. Suggested by Marcin Kasperski.
• Fix pylons.database.AutoConnectHub’s doInTransaction not automatically connecting when necessary. Fixes #327.

I have been meaning to wrap up Net-SNMP for quite some time with the fancy new multiprocessing library in Python 2.6, and I finally got around to making a prototype:

http://code.google.com/p/multicore-snmp/

This may be exciting to .0001 of Python programmers, but I think SNMP is pretty sweet. This current API is a bit rough, but I plan on polishing things up shortly, and making it into a PyPi release. If you want to give it a go just look at the main function, and pass in a list of hosts that are either hostnames, or a full instances of SnmpSession

One of the main motivations to wrapping it up in the multiprocessing library, is that the current Net-SNMP bindings are synchronous. By using the multiprocessing library it effectively makes Net-SNMP actually useable from Python, and opens it up to working on a box with say 8 cores. I based my initial API on one of the processing pool examples in the official docs.

If you have want to help or have ideas about how the API should work let me know.

Shane Hathaway has been working on a system that can be used to compose Zope a publisher configuration out of WSGI middleware implementations.

This is an implementation of an idea that I wanted to pursue last year but Shane beat me to it, and it looks pretty nifty!

I'm not certain that all the current Zope publisher behavior belongs in endware but it's interesting to see it take shape. One fallout from such an effort may be that more stuff that's currently "locked up" in the Zope publisher may fall out as reusable WSGI middleware, which could only mean good things.

I have been extremely impressed with Sphinx lately. I currently use it for most of my documentation and writing needs. I have a very simple workflow that publishes documentation in the form a sphinx web bundle and a PDF, all automatically. Here is my workflow:

1: Get Sphinx: http://sphinx.pocoo.org/
2. Download Latex: http://www.latex-project.org/
3. Start a new project, using the sphinx-quickstart command
4. Create a document in reStructuredText, and link to it from the index.rst file you created.
5. I then just type the ZSH alias:


buildcommit


This is what my .zshenv looks like:


#web dev documentation
alias buildweb="/usr/home/ngift/public_html/notes;sphinx-build source build"
alias webcheckin="/usr/home/ngift/public_html;hg ci -m 'checkin web'"
alias makepdf="/usr/home/ngift/public_html/notes;sphinx-build -b latex source build;build;make"
alias buildcommit="buildweb;webcheckin;makepdf"


6. I then get a nice PDF manual that matches my beautiful web site. The Sphinx developers have done an exceptional job, and the tool makes self-publishing trivial. In addition, my lazy alias is awesome because it checks in every changed file automatically to my hg repository, builds the webpages, and pdf in about 5-10 seconds.

For years I have hosted an updated copy of the Zope Book (2.7 edition) and from what I gather from my webserver logs it's still in pretty heavy usage. That said, as may be evidenced by the version number (Zope 2 is at version 2.12 now), I have ceased maintaining it. It needs a new home. Are you looking to adopt it?

By the way External Editor also still requires a maintainer too.

Sorry folks, it turns out that the discard_first_request flag in repoze.profile wasn't working correctly in my last round of tests, which skewed the absolute number of profile lines emitted by each profile run for all frameworks. The previous conclusion counted profile lines which were only meaningful for the first request. This was due to a bug in repoze.profile. Since they only happen on the first request, these lines are not indicative of reality, so I've redone the tests.

I've released a fixed version of repoze.profile (0.7), and I've updated the numbers using this version. This time, however, the first request is actually discarded, so its function calls don't show up in the profiler output. The resulting numbers, linked to detailed output, are below:

• repoze.bfg emitted 36 lines of profiler output
• Django emitted 57 lines of profiler output
• Pylons emitted 96 lines of profiler output
• Grok emitted 317 lines of profiler output

Links to those numbers lead to repaired profile output as well as enough information to reproduce the test on your own system. I've amended the previous blog post with a pointer to this one. Note that instead of using ab -n1000 -c4, I used ab -n1001 -c4 in order to show even numbers for most of the lines in each profile result.

The relative numbers between frameworks are not completely wildly different (previously they had been bfg 50, django 105, pylons 163, and grok 834; the relative "ranking" remained the same), Grok definitely got the short end of the stick on the last round of tests, they now fare better by a little less than 3X (apologies Grok folks). It should also be noted that the Pylons trunk fares better on this test, if you also use the Routes trunk and the WebOb trunk. Django fared better by half in this round as well.

NOTE : The numbers present in this blog post have been removed. They just weren't indicative of reality. New numbers are available from http://plope.com/whatsitdoing2 . However, I've retained the narrative here for context.

It can be a bit useless to benchmark web application frameworks. When you're commmitted to a particular framework, either it works or it doesn't for your particular application; often raw speed is not really a concern. You're probably not going to switch web frameworks in the middle of a project in order to get a 15% or even a 50% or 100% speed increase: you've got too much investment in the code that works under the framework to consider it. In my experience, very few people truly understand more than one web framework, and they tend to use that framework for everything even it it's slightly less optimal for any specific task; this is because the "switching cost" to go to another one is so high. So benchmarks aren't really all that interesting in the "real" web world; it all depends on context.

But if you haven't chosen a web framework yet (is there anyone?), or if you're falling out of love with your current web framework and you're considering using a different one, you might be able to learn something from profiling an application running under various frameworks nonetheless, even if you ignore the raw speed of the framework itself.

One measure of what you're going to be faced with with when your web application framework doesn't work as advertised is the complexity of what a it does to render a very simple page. If it does a lot of work to render a very simple page, you might need to understand a lot if it breaks or to extend it. If it does very little work, it's likely it will be easier to fix and/or extend than one that does a lot of work. Additionally, usually it's a corollary that the less work an application server does to render a response, the faster it will render that response. But that's not the point here, we're only concerned about work done.

I am the primary author of one of the frameworks shown here (repoze.bfg). As a result, "lies and benchmarks" applies, of course, more than it otherwise would. But I've done my level best to make each framework do as little work as possible to render the page. If I've not, I'm sure the various framework authors will tell me how to improve things.

I've done the following:

• I've created four "hello world" applications (one for each framework tested) These applications are available at http://svn.repoze.org/whatsitdoing in the directories named after their respective framework. The four frameworks that I wrote applications for were repoze.bfg, Grok, Pylons, and Django.
• I ensured I could run each using a WSGI server in order to be able to use http://pypi.python.org/pypi/repoze.profile.
• I placed the http://pypi.python.org/pypi/repoze.profile WSGI middleware into the pipeline in various ways within each application I tested.
• I ran "ab" against the "hello world" page of a running instance of each application (via "ab n1000 -c4") with the profiling middleware turned on within the WSGI pipeline.
• I scraped the output of the "__profile__" page provided by repoze.profile after the "ab" run for each framework was completed. This output gives a rough indicattion how much work was done during the run of "ab". repoze.profile uses the Python "profile" module to peek in to see what an application is doing under the hood.
• I counted the number of lines outputted by the profiler (not including header information). This is a rough estimate of how "broad" the software is. Each line represents a specific function called. So the more lines, the more functions called, and (to some extent) the more you'll need to understand when it doesn't work right or when you need to change what it does or just plain-old understand what it does. By this measure, fewer lines is probably better. Of course, some frameworks might defer doing "one-time" work until the first request that others don't, so this isn't a perfect metric. That said, why would they? Why not get it over with at startup time?

Here are the results:

EDITED (see amended numbers at http://plope.com/whatsitdoing2)

Each link above shows how the application was configured, what software versions were involved, how the application was invoked, as well as the SVN link to the source code for the application, and any config tweaks attempted to make it faster. You can run these yourself if you want to; the results files and code taken together should contain all the information needed to replicate the results.

My blog signup is broken, and people without accounts can't comment, so if you don't have an account, please mail me if you want an account here in order to respond here.

EDIT: shameless self-promotion: I'll be giving a repoze bfg.tutorial this year at PyCon. Please sign up if you're interested.

Once again, kudos to Carlos for his screencast on using Plone in conjunction with repoze.bfg.

Carlos writes that the BFG version, which publishes content mirrored to an RDBMS from the Plone CMS backend, renders pages at around 100x faster than Plone itself.

He is hoping to release the software, developed in conjunction with a big project for the Chilean Library of Congress, in time for PyCon, where he will be presenting about the topic. Way to go, Carlos!

Jonathan Saggue and I just completed an article for IBM Developerworks on writing an iPhone application that uses Google App Engine as a back end. The complete source for the application is available here, as well a framework to help deal with caching.

One additionally cool thing about Google App Engine, is that it is ridiculously easy to prototype applications, or extend development environments. Objective C is a reasonable language, but it is much slower to write code then Python. Using Google App Engine can become a way to prototype out parts of the application to speed up development time, and/or augment what something was intended to do.

The new PylonsHQ site has now launched!

The new site is running on the latest Pylons 0.9.7 code-base backed by the CouchDB database. New features that have been added:

• Snippet repository
• Traceback posting
• Easier to read buildbot status page
• Trac user integration, so your login to the main PylonsHQ site is carried into Trac seamlessly
• Feeds for the various sections, including the new blog
• All new documentation using Sphinx
• OpenID registration and sign-up
• All PylonsHQ code open-sourced and available on BitBucket

Unfortunately, we were unable to integrate the Wiki’s auth, so that will still require a separate login for now.

Comments are through-out the site, to ensure that feedback isn’t missed and of course there’s many more features planned that are coming soon. The site isn’t quite 100% complete, as a few links here and there are likely broken yet (like the tutorial links on the front page). I’ll be putting out frequent updates to remedy this and any other little bits that need more polish.

Enjoy!

Repozecast 3 (mp3), a podcast about Repoze, is very short; less than 10 minutes. It talks about repoze.bfg. The reason it's so short is that we talked for another forty-five minutes after the Handy Zoom recorder had run out of space. We'll try to recover the ground we lost in subsequent epsisodes. Have fun!

- Chris McDonough

Pycon US 2009 is in Chicago this year again. I love going there. This year at the conference, I'll be presenting a tutorial on the repoze.bfg web framework along with Chris Perkins. The tutorial is on Thursday, March 26. If you're interested in repoze.bfg, this is a great way to learn how it works.

Chris Perkins will also be presenting a tutorial on ToscaWidgets: Test Driven Modular Ajax on the same day.

- Chris McDonough

Pycon US 2009 is in Chicago this year again. I love going there. This year at the conference, I'll be presenting a tutorial on repoze.bfg web framework along with Chris Perkins. The tutorial is on Thursday, March 26. If you're interested in repoze.bfg, this is a great way to learn how it works.

Chris Perkins will also be presenting a tutorial on ToscaWidgets: Test Driven Modular Ajax the same day.

The repoze.what authorization framework has its first stable release!

repoze.what, which is the default authorization framework in TurboGears 2, was initially a TurboGears-specific repoze.who plugin (tg.ext.repoze.who) to support authorization based on the groups the authenticated user belongs to and the permissions granted to such groups, written by Chris McDonough, Florent Aide and Christopher Perkins.

The plugin evolved as an framework for arbitrary WSGI applications which allows developers to store the groups and permissions of the application in other source types (not only databases), just to name a few of the features implemented as a TurboGears-independent project.

The code sample below illustrates how this fully documented and tested framework (yes, its code coverage is at 100%) can be used:

# Sample use in TurboGears 2; pay attention to the line with the "@require"
class RootController(BaseController):
    # ...

    @expose('algo.templates.index')
    @require(predicates.has_permission('manage', msg=_('Only for managers')))
    def manage_permission_only(self):
        return dict(page='managers stuff')

In the example above, only people with the "manage" permission will be granted access to the "manage_permission_only" action. Also, if access is denied (i.e., user doesn't have the "manage" permission), she will be redirected to the login form and the message "Only for managers" will be displayed; a behavior that is fully customizable.

This groups/permissions-based authorization pattern is just the default pattern supported in repoze.what, and you can extend it to support your own pattern by creating so-called "predicates".

Planning for the upcoming major release of the package has already started, so please don't hesitate to report the features you want to see in this release!

Special thanks go to Chris McDonough for his support throughout the development of repoze.what.

-- Gustavo Narea.

Kudos to Carlos for his writeup on implementing the "songlist" app, now circulating in the Python blogosphere, using repoze.bfg.

Carlos writes that the BFG version, implemented in ~50 lines of Python and ZCML, benchmarks at 1060 requests per second, compared to the 1780 r/s of the "raw" WSGI version on the same machine.

I have a project that is coming up that involves extending version control and moving data around. I am a Python developer who loves writing Python code, so I was curious to consider not just Subversion and pysvn, but also Mercurial, Bazaar, and Git.

It is one thing to buy into the benefits of distributed version control, but when you want to actually extend a system, it does make me look long and hard at the tools made in Python. Bazaar, in particular, seems to have a nice plugin architecture. Does anyone have an experience they would like to share as a Python developer?

I started playing around with Zine recently and have to say I am quite impressed with Armin, and the whole Zine. Every time I look at pocoo.org I think, WTF, you guys are INSANE. Keep up the incredible work with all of the stuff you guys are working on.