Jan Ulrich Hasecke reviews the repoze.bfg book via the Zope.de website (English translation).

Darryl has put a screencast online of the demo session he is doing this evening at the Christchurch branch meeting of the New Zealand Python Users' Group:

Nice demo, Darryl

The PyCon 2010 audio-video team has done a really bang-up job getting the conference talks recorded, edited, and online quickly, The two BFG-relate talks are now available:

Tres Seaver's "Evolving Your Framework Under Fire" talk:

Carlos de la Guardia's "Pay Only For What You Eat" talk:

Folks have been sprinting here at PyCon this week on a number of repoze-related projects:

Sprint Humor

In reaction to the perhaps overly cute mascots adopted by a certain other nameless framework, the BFG sprint team has adopted the "flying" or "war" pig as a mascot (not the cute Porky type, but a real tusker!).

After hearing Brandon Rhodes' talk on deploying applications to WebFaction, and talking with one of the WebFaction staff here at the PyCon 2010 sprints, I sprinted today on creating an installer script for creating a BFG webapp inside a WebFaction account.

To use it in your WebFaction account, create an application in the control panel using the "Custom Installer Script" option, and paste the URL of the script into the form. Once the installer finishes, you will have a "starter" application running in the webapp directory, You can start hacking on the code there, or upload your own app to it and tweak it there.

For the KARL project, the development team is primarily involved in operations.  Along with Six Feet Up as the hosting provider, we are responsible for many activities in “SaaS”.  Bugs are reported to us, we do the software updates, we help monitor the site, we do staging and testing of customer instances.

I really enjoy this aspect.  It’s different from past involvement in open source projects, where involvement in the software is somewhat de-coupled from living with your mistakes. [wink]  Stated differently, we have a direct interest in stability, performance, quality, and even in things like making KARL easy to monitor by Zenoss.

We periodically update the software.  Which means restarting the app server.  Which usually means, downtime.  Over time, we’ve whittled that down.

First, KARL restarts fast.  Like, two seconds or so.  Thus, the impact is minimal.  Next, we use mod_wsgi, which lets us do “graceful” restarts in Apache.  Serve all your current requests and restart your processes.  These combine for providing very fast updates.

There’s one aspect that’s harder though.  Sometimes our updates require “evolve” scripts to update data.  For example, adding an index, or fixing a value that requires waking up lots of objects.

We used to do this live in a ZEO client, but when the evolve script takes more than a few minutes, we get prone to conflict errors with the running site.  Which means, shutting down the main site.  Which, sucks.  (I’ve become quite obsessive about performance and uptime.)

We have some ideas we think can mitigate this:

• SSD.  Six Feet Up has the solid-state disks installed.  Because of RAID and cabinet issues, the SSDs are going in the spare box in the rack.  We’ll then move the site over.  The hope is that evolve scripts get faster.  If the evolve scripts are bottlenecked elsewhere (e.g. ZEO single-threading), then that’s a different issue.
• Read-only mode.  Perhaps we could leave the site in read-only mode during the update, with a little banner informing the user.  Preferably we could put the site in read-only mode without a restart.

Any other ideas on minimizing downtime on such applications without major changes in architecture?

Tres will be giving a talk at PyCon 2010 in Atlanta this Saturday at 2:15 PM. The talk, entitled Evolving Your Framework Under Fire covers the evolution of BFG as it was updated and extended to support the needs of KARL, a mission-critical knowledge management system commissioned by the Open Society Institute .

The previous version of KARL was built on top of Plone, with more than four hundred communities and 75,000+ content pages. The application had evolved over several years, and had become increasingly difficult to scale and extend. In addition, OSI was actively recruiting participation in KARL from other, simliar organizations, whose own unique requirements would have to be folded into the mix.

Over the course of six months, a small team built the new version of KARL on top of the BFG framework, which we were simultaneously extending and elaborating to support the needs of KARL. During that period, we made more than forty releases of BFG, as well as releasing new versions of many supporting packages. The 1.0 release of BFG followed shortly on the successful launch of KARL in May, with subsequent versions continuing to be informed by the needs of OSI and the other KARL users.

This talk discusses the process of evolving the BFG framework, with particular emphasis on learning from the changes to the assumptions made by the framework at its inception. We discuss also the tradeoffs in the choice to include features in the framework versus leaving outside, either in the application or in supporting libraries. Finally, the talk proposes some general "lessons learned" which should be useful to Python programmers building any sort of framework.

This has been the week of major KARL updates.  KARL is the open source collaboration and knowledge system published by the Open Society Institute.  It is an end-user product atop the BFG web framework.

Some recent happenings in KARL-land:

• We are wrapping up a major improvement to the calendar tool. We introduced a concept of sub-calendar “layers” that can aggregate events from other communities.  More visibly, we completely re-did the UI with a new Weekly view and lots of Ajax sprinkling.
• We also did most of the work to re-implement the entire form system using io.formish.  (repoze.bfg.formish to be more precise.)  Big reduction in code and test code.  Still have a lot to think about on form controller patterns.  Those going to PyCon can hear Chris’s recounting of form controller torture in a panel he’s on about form frameworks.
• Along with our friends at Six Feet Up as hosting partners, the KARL team is now operating KARL sites for five organizations.  KARL has a unique approach to customization, the inverse of the traditional Zope approach.  In KARL, a customization package is the starting point, and that pulls in the main software.  Or, doesn’t.  We just rolled out changes to thin out the size of each customization package.
• Chris Rossi has been working on a web interfaces to a number of admin activities, including integration with Six Feet Up’s Zenoss monitoring.
• KARL has a number of periodic admin jobs, for things such as processing incoming/outgoing mail, pulling in feed content, etc.  To date we had been running these as cron jobs.  However, we had cases where hundreds of crons got piled up.  In the latest updates, we converted these to Supervisor-managed jobs.  Risk involved, so we’ll see how it goes.
• And finally, we are going to work with Six Feet Up to install solid-state disks.  Our initial test shows that an SSD alone, with no code refactoring, will completely eliminate our performance concerns on LiveSearch.  As well as benefit other catalog-constrained screens, possibly.  We’ll report back once we live with them for a while.

All in all, KARL (like BFG) is humming along nicely.  Just to emphasize: KARL isn’t a framework.  It is an out-of-the-box product with a strong opinion.  By making such a deliberate choice to not boil multiple oceans, KARL gets to be very compact, very fast, and very stable.

That’s the key takeaway for people working on larger projects that have dynamic performance needs.  Unless your needs fit into Product X’s bulls-eye, you’re probably better off not beating Product X into submission.  Instead, we need an approach where assembling your own custom application, leaving out the parts you don’t need, is more feasible.

Not only is this a win for custom apps, IMO it’s actually a win for Product X.  Instead of having reputation beatdown when it doesn’t excel at Every Possible Thing, it can just say: “We’re good at X. If you want X, you want us.”  Then, focus scarce resources and reputation on being the best possible X.

Big new happenings for the BFG web framework.  First, Chris has wrapped up the writing on a BFG book which you can get via Createspace or Amazon.  The book content is also available as open source.  However, buying via Createspace helps the author a bit more, so please feed the animals by grabbing a copy.

Chris recently documented his experience writing the book.  I’ll add to it a bit.  Chris worked on this book.  And worked.  And worked hard.  Laboriously, fanatically, during the entire course of creating BFG.  In fact, the docs were released before the software, and every one of his million software releases included doc updates. In double fact, he used the book writing as a reason for improving and refactoring BFG.

If your idea of a good web framework includes one with scrupulous documentation (as well as 100% test coverage) then BFG is the right cultural fit for you.  Chris really is nuts, certifiably.

In more BFG news, Chris released BFG 1.2 recently with a What’s New in 1.2 document.

The first highlight is an “imperative mode” that let’s you write almost nothing, import almost nothing, and postpone understanding big new concepts, but still get a real application going.  Follow the link and look for the first code example.

The other major highlight, from my perspective, is Jython support.

BFG has been experimenting with alternative approaches to “configuration”.  The imperative mode is part of the result, but the entire configuration story is worth looking at.

Congrats, BFG and Chris.  The good story keeps getting better, with more people deploying sites, contributing, and giving tutorials at PyCon.

Recently I authored a book about software . My company self-published the book. I chose to self-publish the book rather than shop the book around to professional publishing companies for a number of reasons:

1) Control. The book contains the entirety of the documentation for a piece of open source software . In my mind, this documentation must always be freely available; otherwise the platform isn't truly open source. I neither wanted to offer another for-profit company a copyright on the material, nor did I want to lose any future flexibility, such as the ability to grant copyright to a foundation during a code transfer, etc. Retaining maximum control was a non-negotiable requirement.

2) Simplicity. I didn't need to talk to anyone, or negotiate with anyone, or rationalize choices with anyone in order to get this book published. I could just do it.

3) Quality. I have heard some horror stories about publishers having low quality standards when it comes to layout and organization. I felt I could do as good a job, if not a better one, than many publishers with respect to book quality and layout.

Now that the book has been published, I think I think the choice to self-publish was a good one, and I'm extremely comfortable with the result.

I've never published a book through a traditional publishing house. But I know a good number of people who have. Although self-publishing a book is a pain, from what I hear from other authors who did not self-publish, it's not that much more of a pain than publishing a book via a professional publishing company.

I chose to use Createspace to help me with self-publishing. Createspace has a really good system for document review and publication. Basically you fill in a bunch of book info, provide them two PDFs (one for the cover and one for the actual book content), and they guide you through the process of publication. You receive a proof copy of the printed book before you need to make the book available to the world. Their proof and author copy prices are excellent, their integration with Amazon and other book outlets is very good, they provide an "eStore" presence for you, they handle payment processing, and they give generous royalties.

I chose Createspace over Lulu solely because I thought Createspace would have better Amazon.com integration than Lulu; Createspace is an Amazon subsidiary. I have no way of actually knowing this for a fact, because I've never used Lulu; they may have Amazon.com integration that is just as good. Subsidiary relationships can be poor indicators: large company organization often means that subsidiaries are effectively separate companies. But the book showed up on Amazon.com fairly quickly, and without any effort on my part, so it appears that Createspace and Amazon.com at least don't have an adversarial relationship.

After being available for a week, the book has sold exactly 24 copies. Most of the sales were via the eStore provided by Createspace, presumably due to my own marketing efforts. This is obviously a modest number, but because we kept overhead reasonably low, we've almost broken even on the hard cost of its publication already. Book sales alone will never recompense the time that went into writing it, but of course I never expected them to. Its availability is more of a "presence" thing, and as long as we lose no money on it, and people find the book worth its price, I'm happy.

One thing I didn't expect: it was extremely fun to be able to send out gift copies of the book to major project contributors (internal and upstream). Although it's a poor gift in comparison to the time these folks have spent making things that are useful to me, being able to send some physical thing to people you usually communicate almost purely electronically with is a novelty. We should all do that more often.

Some suggestions to potential technical book authors (the "if I had it to do all over again" list):

• Don't underestimate the time it will take to do typesetting.

  While I wouldn't trade away the flexibility of self-publishing just in order to avoid doing my own typesetting, it is an extremely time-consuming, tedious effort.

  I used Sphinx to generate LaTeX source. I had to make a number of local modifications to Sphinx (which I later "contributed upstream" minimally via a maillist post) to make it write LaTeX suitable for a book. This worked extremely well. On the other hand, I became more familiar with LaTeX than I ever wanted to become while doing typestting for the book. This was no fun, but I'm certain it would have been worse if I had used, say, OpenOffice, or Word or something.

• Remember that most technical books have an international audience, and that shipping cost is a factor.

  I didn't really know how this worked when I published my book. I knew enough that I made sure it would get an Amazon.com listing, but I really didn't understand that this didn't really mean reduced shipping rates for people in Europe. I just sort of assumed it would show up in Amazon's localized sites (like Amazon.co.uk, Amazon.fr, etc). But in order for the book to show up in country-specific sites, you need to take explicit steps; it doesn't just happen. When people from Europe buy from Amazon.com as opposed to their localized Amazon site, shipping can cost $50, and understandably, as great as your book may be, most reasonable people just won't pay this much extra. It will also take a long time to arrive.

  I have since taken steps to make sure that the book will show up on Amazon's localized sites eventually (it still doesn't yet), but I didn't find any way to make sure it all happened as a "big bang" where it was available everywhere all at once. I don't really mind that it didn't happen as a big bang, but I would have preferred to know this beforehand, so I had a better response than "homina homina" when responding to complaints from people about localized availability and shipping rates. My answer is now still lame ("please wait"), but at least I have one.

• Secure a set of reviewers before you are ready to publish, and make sure you allot enough time for book review.

  I wanted this book to come out before PyCon . By the time I had finalized the book copy and done the typesetting, I was in a place where giving reviewers a reasonable amount of time to review would have delayed the book publication beyond that time. Andrew Sawyers bailed me out by reviewing much of the book before publication, but if I hadn't had his help, I would have been out of luck.

  On finding reviewers. Although I did send out a general request for reviewers to a related maillist, I wasn't brave enough to email individuals asking each if they would do technical book review. I get requests every so often to do technical book review, and I almost always have to decline, because the compensation (usually a free book) usually can't justify the amount of work to do a good review unless the book topic aligns very closely with my own work. Due to my own reluctance to review tangential books, and because the topic is tangential to most (BFG isn't topping the popularity charts yet), I didn't want to put other folks in a place where they felt boxed in by such a request. I also didn't really have a budget to offer potential reviewers anything beyond a free book. I'm not sure how I'd do this differently in the future, but the book would only have been better with more reviewers.

I'd definitely recommend that potential technical book authors consider self-publishing rather than licensing their work to a professional publisher. There are tradeoffs, of course; if you go with a publishing house, more copies of your book are likely to be sold than if you self-publish and therefore also self-market. But often it just doesn't matter ; it certainly doesn't to me, anyway.

The availability of good physical-medium self-publishing facilities like Createspace is maybe a bellwether for the future of book publishing. I suspect professional book publishers will need to specialize even further, and offer better service, licensing terms, and royalty structures. I think this will be an overall improvement for writers and consumers, although it may imply an extremely painful adjustment for book publishing companies.

We're proud to announce the latest major release of the repoze.bfg web application framework, version 1.2.

It may be installed via:

easy_install -i http://dist.repoze.org/bfg/1.2/simple repoze.bfg

The most important feature addition in 1.2 is an "imperative" configuration mode. This implies:

• A repoze.bfg application can now be contained within a single Python file.

• Developers are no longer required to use or understand ZCML to create a

  repoze.bfg application (ZCML is, however still supported).

See the "What's New in 1.2" document at http://docs.repoze.org/bfg/1.2/whatsnew-1.2.html for a detailed list of new features, changes, and backwards incompatibilities.

The docs at http://docs.repoze.org/bfg/1.2 have been updated. Additionally, a printed book which includes the documentation has been published (see http://bfg.repoze.org/book). The documentation has been extensively reworked and restructured for the purposes of printing the book.

The 1.2 version of BFG replaces 1.1 as the "current" version, which means that the index URL http://dist.repoze.org/bfg/current/simple and the documentation URL http://docs.repoze.org/bfg/current both now point at the 1.2 versions of things. The older 1.1 index is available via http://dist.repoze.org/bfg/1.1/simple and the older 1.1 docs are available via http://docs.repoze.org/bfg/1.1.

Enjoy!

The repoze.bfg Web Application Framework, Version 1.2" paperback is now available for purchase:

https://www.createspace.com/3422488

This is a high-quality printing of the along with extras such as a foreword from Paul Everitt, an author introduction, and cool cover artwork from the guys at Electrosoup (http://www.electrosoup.co.uk/). It is slightly over 500 pages long.

If you like repoze and BFG, please show your support by buying a copy!

Book abstract:

repoze.bfg is a small, fast, down-to-earth, open source Python web development framework. It makes real-world web application development and deployment more fun, more predictable, and more productive.

This book will show you how to develop web applications using repoze.bfg and Python step-by-step, including:

• Using SQLAlchemy and ZODB along with repoze.bfg.
• Using built-in templating facilities to render HTML and XML.
• Using repoze.bfg security functionality to protect your application.
• Exending an existing repoze.bfg application without changing its source code.
• Using sessions within repoze.bfg.
• Running repoze.bfg under mod_wsgi.
• Deploying your repoze.bfg application to Google's App Engine.

This book is written by the primary author of repoze.bfg. It has developed along with framework itself, and therefore contains the most accurate and up to date information.

The repoze.bfg Web Application Framework, Version 1.2 paperback book has been published and is now available for purchase via Createspace .

This is a high-quality printing of the BFG docs along with extras such as a foreword from Paul Everitt, an author introduction, and nifty cover artwork from the guys at Electrosoup . It is slightly over 500 pages long.

If you like the Repoze project and BFG, please show your support by buying a copy.

Book abstract

repoze.bfg is a small, fast, down-to-earth, open source Python web development framework. It makes real-world web application development and deployment more fun, more predictable, and more productive.

This book will show you how to develop web applications using repoze.bfg and Python step-by-step, including:

• Using SQLAlchemy and ZODB along with repoze.bfg.
• Using built-in templating facilities to render HTML and XML.
• Using repoze.bfg security functionality to protect your application.
• Exending an existing repoze.bfg application without changing its source code.
• Using sessions within repoze.bfg.
• Running repoze.bfg under mod_wsgi.
• Deploying your repoze.bfg application to Google's App Engine.

This book is written by the primary author of repoze.bfg. It has developed along with framework itself, and therefore contains the most accurate and up to date information.

About the Author

Chris McDonough is the primary author of the repoze.bfg web application framework. He is a Python developer and consultant for Agendaless Consulting, a company based in Fredericksburg, VA, USA. His other major projects include Supervisor, a Python process management system.

People hate systems that allow them to succeed poorly. They hate such systems even more than systems that don't allow them to succeed at all. People will forgive systems that don't allow them to succeed; they will never forgive a system that allowed them to succeed poorly.

This is because, by allowing a person to succeed poorly, you have tricked them into believing a problem is solved and that they may move on to the next problem. The problem, however, is not solved. The problem remains, papered over by something that only makes it appear solved. Because the problem was not solved, the implementor will eventually need to revisit the problem. Except now instead of just having one problem to solve, he'll have two: he'll have to re-solve the original problem, then he'll have to figure out how to make the code that implemented the previous (poor) solution work on top of new code for backwards compatibility purposes.

It's hard enough to solve a problem in "green-fields" coding; needing to carry along the burden of backwards compatibility with older code and possibly rebasing old code on top of the new solution makes it at least twice as hard. Sometimes ten times as hard.

This is why people hate software that allows them to succeed poorly. It makes 2-10 times as much work for them in the long run. It would be far better if the software they chose either a) allowed them to succeed optimally or b) allowed them to fail very quickly.

The very most potent vitriol is reserved for systems which allow people to succeed poorly. Consider the hatred unleashed on public forums, even today, ten years later for Zope 2 features like TTW coding, large-ZODB-objects-as-blobs and implicit acquisition. All of these technologies provided solutions (albeit not full ones) for real problems at the time and they became widely used. Each one of these things allowed people to do things that were previously very difficult with very little effort. But ultimately, solutions based on these systems needed to be revisited due to performance problems, testability, and code comprehension. All have been largely sunsetted, because it was recognized that the development effort required to allow people to truly succeed when using them would be enormous.

But still, people actively hate them and will go out of their way to tell others so (usually in a brutal form like "Zope sucks"). Ten years later. Do you see vitriol like this for technologies that people failed entirely with? No. This is because these people felt fooled and betrayed in the long run.

What's the takeaway? If you're going to introduce a feature to code that other people rely on, start by documenting it. If you cannot document it adequately, or you find that you may need to spend an inordinate amount of time creating code that makes the feature not suck, don't add the feature. Only add the feature when you can both document the feature itself and its failure modes. Be conservative about what you offer. You can always add features; you can never take them away. If you do add a feature, make sure that it doesn't pretend to do more than it actually does. Allow its common usage to have a "fail quickly" path.

Darryl Cousins gives us a blog post explaining BFG on Google App Engine including test coverage support.

Srikanth Suri gives us a blog post showing how he has integrated BFG with repoze.catalog.

I’ve enjoyed seeing some writeups on requests/second and memory usage for upcoming versions of Plone.  It’s great to see things trending in that direction.  Hopefully with some tough choices and deprecation, more gains can be made (just my personal opinion.)

I thought I’d give a primitive try at the same numbers for KARL, the collaboration application atop BFG that we’ve been working on and deploying to customers.

Using the ‘ab -n 100 -c 2′ on my first gen MacBook 2 GHz, 2 Gb of RAM, I leveled off at just over 134 requests per second.  Memory usage was 31 Mb.

Obviously it’s not an apples-apples comparison.  The feature set is smaller.  Although we do have cataloging, text search, workflow, security, and the like, there’s a ton of stuff we don’t do.  We’re an end-user application with specific features, versus a framework.

On the other hand, all requests in KARL are authenticated and fully-dynamic.  So the 137 rps above?  That’s our slow number: authenticated, personalized, security-aware, fully dynamic.

For more fun, we recently built an ugly, cheap Core i5 box in the Agendaless office for $600, with 4 Gb of RAM.  In production we deploy under modwsgi, so we fired it up to have 3 processes (for 3 of the four cores).  We also have a script that lets us bulk load 300 sample communities, each containing a bunch of content.

That’s a bit more realistic of a test, since we start paying the price of having content in the catalog.

In that “with content” test, we got 349 requests/second.

Sometime soon we’re going to think a bit harder about a more realistic test.  Pounding the same URL over and over as the same user just doesn’t mean squat.  Well, it’s valuable in so much as it is a veto: if your numbers are pathetically low on the fastest-possible “test”, it’s only going to get worse.  We are slowing building up some Funkload scripts that cover a scenario which includes different users, different activities, and some writes as well as reads.

We need this as we are evaluating various KARL ideas in 2010.  First and foremost, we bought a solid-state disk for the test box.  We had a query (prefix match on text search, where only one letter was entered) which blew up our system previously.  Think, 60+ seconds.  That time fell down to 2 with the SSD.

Next, we’d like to see some before/after on RelStorage using some real-world scenarios.  Finally, I’d like to see some before/after on repoze.pgtextindex, where we swap out just one of our catalog index types (the text one) with transactional text indexing in Postgresql.

Yawn. Boring. I'll post it anyway, though.

The Zope "multiadapter" machinery lookup algorithm isn't defined or documented in any sort of human-consumable way anywhere. So you'll have to trust me on the following. It's almost incomprehensible anyway, so if you have a low tolerance for details, don't bother trying to understand this.

Zope views are multiadapters on ``(context, request)``. When you say please adapt (context, request) to IView, it does more or less this:

• Give me every interface provided by context.
• From most specific to least specific interface provided by context, search the "first" slot in the registry for further lookup information.
• If something is found in the first slot that is registered for any of the interfaces of context, continue on to look for an adapter based on the request object.

In terms of a registry represented by a dictionary and a pseudocode implementation of getMultiAdapter, it looks a bit like this:

  def getMultiAdapter(objects, target=Interface):
      num = len(objects)
      info = registry[num][target]
      for object in objects:
          items = providedBy(object)
          for item in items:
             rest = info.get(item)
             if rest is not None:
                break
          if rest is None:
              raise ComponentLookupError
          info = rest
      return info(*objects)

This isn't quite right, because some backtracking happens when a secondary lookup fails, and we're disregarding the adapter name, but for our purposes it's close enough.

Let's also take for granted we have some interfaces:

  from zope.interface import Interface

  class IRequest(Interface): pass
  class ISpecializedRequest(IRequest): pass
  class IContext(Interface): pass

In such a setup, when you then do this:

  registry = {2:{IView:{IContext:{IRequest:adapter}}}}

  directlyProvides(context, IContext)
  directlyProvides(request, IRequest)

  getMultiAdapter((context, request), IView)

You will get back the result of "adapter" in our faux registry. This is well and good. However, there is a corner case lurking here that is not well and good.

For view lookups, I actually think the historical Zope ordering of (context, request) is not correct. It should actually likely be (request, context). If you take this to its logical "worldview" conclusion, I think this means that Zope view class constructors should have been made to accept (request, context) rather than (context, request).

Why? Well, as a general rule, due to the lookup algorithm above, you almost always want to create multiadapters with arguments ordered in such a way that "provides" registration arguments more likely to be for specific interfaces come before "provides" registration arguments that are less likely to be for specific interfaces.

View registrations are always made for some very specific request (e.g. IRequest) interface, but often they are made for a very weakly binding context interface, sometimes just "Interface". "Interface" is implemented by everything.

For example, the (context, request) ordering breaks down a bit when you have a registry populated as follows:

  registry = {2:
                {IView:
                  {IContext:{IRequest:adapter}},
                  {Interface:{ISpecializedRequest:anotheradapter}},
                }
             }

And you do this:

  directlyProvides(context, IContext)
  directlyProvides(request, ISpecializedRequest)

  getMultiAdapter((context, request), IView)

You will still get back the result of "adapter" even though it's likely that you really wanted to get back the result of "anotheradapter".

Why would you expect to get back the result of "anotheradapter"? It's an extremely uncommon thing to do to put a specialized interface on the request and to make registrations for this specialized request type. You almost always want to find adapters registered for the request interface first, even if the registration context interface is "more binding".

Jarn has built Tribaspace using BFG and a collection of other technologies. The site is still in beta, but Tribaspace is a promotion site for fashion-related events. Looks good. ;-)

Fabian Reinhard from Seantis gmbh wrote to let us know that his company has released a BFG-based invoicing application:

First, thanks so much for your hard work on repoze!!
We are coming the long way from Zope/Plone and it's just amazing to play
with this new toy!

We just released a small app for invoicing: http://invoice.seantis.ch

Might be useful for other people out there since invoicing seems to be a
quite common use case! :-)

Check it out! It looks like a really nice application!

Once a programmer latches on to some process or idea that makes him more sucessful, he tends to treat it like a subroutine, calling in to it over and over again to get his job done. It becomes part of his fundamental toolkit.

However, over time, for some reason, programmers also seem to forget that the helpful process or idea they found is just a mechanism, and not some sort of natural law. As a result, sometimes the helpful process or concept itself becomes dogma. Dogma is a form of extremism: the belief that an idea is superior even when there's scant proof of its superiority, or when there is existence proof to the contrary.

Using dogma is almost never useful except in one very specific circumstance. Your can use dogma successfully to box someone very inexperienced into a particular worldview for the purpose of getting output from them: inexperienced programmers are willing to accept a limited worldview, filled with all sorts of dogma, because they often lack the experience to make otherwise reasoned decisions. Dogma actually helps inexperienced programmers produce things, because they don't get wrapped around as many axles when trying to make decisions about how to get something done: to them, dogma acts as a bridge between a problem and a solution. Without the dogma, they might never reach a solution.

But sometimes, it can hurt. Let's take, for example, treatises on testing. I see lots of blog posts with this theme: "test-driven development helps me, because, at the end of the process, I wind up with a codebase that has tests." Such posts might mention other benefits, such as "test-first" as a design aid, but the central theme of the blog post usually marvels at the idea that the author can later refactor his codebase without unintentionally breaking it. To the author of the blog post "test-driven-development" is testing; there is no other kind. To suggest that you might write tests after you write a bit of code would be an anathema to such a person, because they believe that the process of testing simply cannot be accomplished without test-first. And it takes a lot of time to convince them otherwise; time that could have been used to actually do something useful.

Lots of programmers do indeed test-last (or at least test-during) without any noticeable difference in code quality. I personally don't mind if people use test-first (aka test-driven) development or test-last, as long as they wind up with good tests at the end. As long as the design works out OK, and the code is tested, the path you take to reach "good tests" should be negotiable. Dictating a particular process to reach that goal is often not helpful.

The same criticism applies to any insistence that some particular technology or library or approach is always better than another, no matter what the circumstance. Questioning a dogma in such circumstances can be fraught with peril, both interpersonal and political, but to reach an optimum technical solution, questioning dogma almost always has to be done. But sometimes its better to optimize for the best interpersonal solution rather than the best technical one: "pick your battles". Deciding when to question dogma and when to leave well enough alone is an art, I think.

I think the phenomena of dogma might be explained by the nature of the job. Programmers need to tell a really, really dumb and unforgiving box "do this, then do this" over and over and over. It's quite easy to forget that humans can operate with more incomplete data, and can produce the same meaning from a set of inputs using radically different thought processes and activities.

BFG platform support is really very good at this point. The latest release of BFG (1.2a11) supports the following platforms:

• UNIX (CPython)
• Windows (CPython)
• Google App Engine
• Jython

I haven't tried it on PyPy. If someone has the gumption to try it out and help out a bit when it breaks, I'll try to make it work on PyPy too.

The reason we can support these platforms: BFG's dependency set is tiny. It's reasonably easy to make them all work on arbitrary platforms. Less software really is an asset.

Eduardo Diaz blogs about running repoze.bfg on Jython. Given Eduardo did a lot of work to get it mostly-running, we'll try to make it a supported platform.

Yesterday I blogged about the view predicates feature of repoze.bfg 1.1 . BFG 1.1 was released several months ago, and since that time, we've had a good number of alpha releases of the subsequent major release, 1.2. The most recent version in the 1.2 line as of this writing is 1.2a9. 1.2 final is due out within the next few weeks.

The most important new feature in the BFG 1.2 line is "imperative configuration". "Imperative configuration" is a funny, loaded term so let's take it one word at a time.

Imperative: this term contrasts a requirement in previous releases that configuration be at least partially declarative. In BFG releases prior to 1.2, it was necessary to have at least two files for any given application: a Python file representing the code and a ZCML file.

Configuration: "configuration" in terms of BFG means the stuff that wires up specific URLs to specific views, or that wires up up event subscribers to event emissions, etc. The stuff that turns the BFG framework into a particular application deployment.

The declarative configuration "story" provided by BFG is stolen almost entirely from Zope. The zope.configuration package is used to process declarations made in ZCML files. These declarations mutate an "application registry", adding various registrations as the ZCML files are processed. At the end of processing, the application registry represents all the stuff needed to run a particular application.

In BFG 1.0 and 1.1, you were required to have at least one ZCML file per application. This ZCML file, in turn, needed to contain at least one statement in it: one which kicked off a "scan", which is a process that scans a package or module for view configuration decorators. Alternately, you were permitted to disuse view decorators entirely and configure view mappings via <view> declarations.

The requirement that a ZCML file exist in 1.0 and 1.1 was a holdover from BFG's Zope heritage. In BFG 1.2+, it is possible to create an application entirely in Python without any ZCML (or even any decorators) in sight. For example:

   from webob import Response
   from paste.httpserver import serve
   from repoze.bfg.configuration import Configurator

   def hello_world(request):
       return Response('Hello world!')

   def goodbye_world(request):
       return Response('Goodbye world!')

   if __name__ == '__main__':
       config = Configurator()
       config.begin()
       config.add_view(hello_world)
       config.add_view(goodbye_world, name='goodbye')
       config.end()
       app = config.make_wsgi_app()
       serve(app)

If you put this stuff in a Python file, and invoke it with an interpreter that has the repoze.bfg software installed, you'll get a running helloword application. The add_view method of the configurator does effectively the same thing as a ZCML <view> declaration; in fact the ZCML view directive code calls into this method when a ZCML <view> declaration is found.

While it's sort of neat to be able to run an application as a single file, it is sort of a "gee whiz" feature that isn't very useful in practice. Every sizeable application will eventually need to split itself across multiple files. But the act of making this possible really helped me clean up the code a lot; rather than coding for configurability via ZCML, and then adding weird sort of stubs for testability, the Configurator can be used to configure both the application and test setup for an application in exactly the same way. Likewise, lots and lots of code got centralized into the configuration module (a lot removed from code that drove various ZCML directives), and we were able to document the resulting API much more effectively and cleanly.

The end result? Well, now, truly, if you don't want to use ZCML, you needn't. Ever. But even so we didn't wind up with a system that leads you towards making a false choice between "convention over configuration" and ZCML: both the declarative and the imperative configuration modes are still extremely explicit (they are really mirror copies of each other, as the declarative code uses the imperative API). The API used by imperative configurators is not "bare" ZCA registrations: it's just more domain specific, like helper ZCML directives in Zope. And hopefully we've brought down to earth the kinds of configuration tasks you can potentially perform, because they're now all enumerated in the Configurator API documentation and via various narrative documentation chapters. All of the older declarative/ZCML configuration still works as it did in 1.0/1.1, so there's no backwards compatibility concerns either. ZCML will always be a reasonable way to configure a BFG app.

I think 1.2 is the best of the bunch as far as BFG releases go, because, although BFG still has an obvious Zope heritage, 1.2 unglosses over some of the things that Zope has been glossing over for a long time, such as the relationship from the code in a ZCML directive to code that needs to be invoked imperatively. BFG also eschews some dogma-driven assumptions Zope has been making for a long time. When I say this I am reminded of this passage from the ZCML chapter of Stephan Richter's Zope 3 Developer's Handbook :

  While the developer is certainly the one that writes the initial cut of the 
  configuration, this user is not the real target audience. Once the product
  is written, you would expect a system administrator to interact much more 
  frequently with the configuration, adding and removing functionality or 
  adjust the configuration of the server setup. System administrators are 
  often not developers, so that it would be unfortunate to write the 
  configuration in the programming language, here Python. But an administrator 
  is familiar with configuration scripts, shell code and XML to some extend. 
  Therefore an easy to read syntax that is similar to other configuration 
  files is of advantage.

I think it's safe to say that this assumption has turned out to be just false. Sysadmins never change ZCML files. I have never, ever, not once in the last seven years, had a sysadmin with enough context that allowed him to change a ZCML file in any meaningful way. Only developers change ZCML files. So let's just un-dogma-ify this assumption, and put programmers back in charge to whatever extent they'd like to be. Demystification of this stuff is the only reasonable way forward, even if it means admitting we made really big design mistakes in the past.

In a podcast we recorded yesterday, I held forth about the new features of the last two repoze.bfg releases. One of these, new in 1.1, was the idea of "view predicates".

Now, the idea of a view predicate was not mine, it was Malthe Borch's. At the time, I was still wrapped up in the Zope-inspired worldview that a view invocation is (by god) an invocation of a ZCA multiadapter, QED, full-stop. Malthe figured out that it is useful to look outside the idea of adaptation to resolve a request to a view, and as a result I implemented view lookup in terms of view predicates in 1.1; 1.0 lacked this feature. The epiphany is somewhat obvious in hindsight, but at the time, it was not. I was well and truly wrapped around the adaptation axle.

For those who don't know, an "adapter" is a bit of code that converts a set of input objects [I] to an output object O. The simplest physical example of an adapter is the one we know and love: a power plug adapter. Maybe one that converts a US-style two-pronged power plug to a "proper" mains plug like they use in the UK. Other stilted and trivial adapter analogies exist, I'll skip them here. Google for "Zope adapter".

In Zope (and BFG, and Django) terms, a "view" is a bit of code invoked as the result of a particular request. It's called a "controller action" in other framework religions.

In terms of view dispatch, Zope uses a combination of graph traversal and an adapter lookup to find a view. Rather than using some ordered set of URL match tests like many other web frameworks, Zope encodes all knowledge about which view should be invoked in all particular circumstances within a given application as a set of adapter registrations. When a request enters the system, traversal is responsible for finding a "context" object and a "view name". Using this context object, this view name, and the request object it does a "multiadapter" lookup something like this: please find me a view adapter for this kind of context and this kind of request with this name. Ex.: ((context, request), view name) -> view object.

While I'm a big fan of this style of dispatch, real-world requirements stretch its applicability. It was always pretty odd to need to attach an interface object to a request in order to convince the view machinery to do something different in some circumstance; it got truly weird when Zope started to support persistent registries that allowed you to make different view registrations in different contexts. Entire subframeworks of various Zope subsystems and offshoots exist just to manage the results for adapter and utility lookups related to requests and contexts. I fear it may all be for naught.

What Malthe figured out is that view dispatch is not just an adapter lookup. While adapter lookup can help speed things up, there's just not enough value in the adapter lookup machinery to spell all the query axes in terms of interfaces, as required by zope.component. For example, it makes a hell of a lot more sense to register a view callable that will be invoked in a circumstance like so:

  request.method == 'POST'
  'application/json' in request.accept

Than it ever would to register something in terms of interfaces, like Zope makes you do, ala:

  providedBy(request) -> IPostRequest, IAcceptApplicationJSON

I mean, literally, it's just not really possible to anticipate all the interfaces you might need to attach to a request in order to provide a vocabulary that allowed you to compose interfaces in enough combinations for such a system based on interface lookup to work. Even if you could, who would understand it after you created it?

With view predicates in BFG 1.1+, it becomes quite easy to register view callables for very specific circumstances that would be extremely difficult (or maybe impossible) to spell if you treated view lookup as only an adaptation problem. For example, using the containment view predicate in BFG, you can do this:

  <view
    for=".models.Entry"
    name="edit.html"
    containment=".models.Blog"
    />

The containment=".models.Blog" attribute is the important bit. It says "only invoke this view when the context object or any of its parents is a Blog object". This solves a whole raft of problems for UI, where you want to use the same kind of object in two places (an "Entry" object) but you want its edit view to be slightly different in two different "sections" of a site: maybe one view when we're in the Blog section, and another when we're in a Calendar section. This sort of registration is not possible in bare Zope, and adaptation alone, at least in the context of using a single adapter registry, cannot do it.

I don't think we can continue to treat view dispatch in the Zope world as a bare adapter lookup; it's just too limiting. In BFG 1.1+ we do not. Using a pattern much like Zope's we use a multiadapter lookup to find a "view" object. However, the view object that is found is often a "multiview". A multiview is a collection of views with associated predicates. Once a multiview is found and called, it evaluates the predicates associated with each of its constituent view callables (in a reasonably easy to understand order) to find the most specific view callable; then it invokes that to obtain a response.

Essentially we use the adapter lookup machinery only as a speed enhancement in this circumstance. The adapter lookup gets us "in a ballpark" where we can perform fewer predicate evaluations to find "the best" view callable. It would be possible to implement it without doing any adaptation at all, it would just be a lot slower.

Other predicate attributes exist for use in this way: route_name, request_method, request_param, xhr, accept, header, path_info. In the very latest release (1.2a9), there is even a "custom predicate" predicate, which allows you to provide a callable that returns true or false for an arbitrary circumstance so you can create your own predicate logic. These can be combined arbitrarily to specify an extremely granular set of circumstances without any interface foolery.

Personally, I think view predicates are a pure win, and I think their existence demonstrates why bare adaptation is not ideal for view lookup. I might even take that a little further: it would also be useful to be able to look up other things (such as utilities, and other code) using the same set of predicates. If this becomes true, I think the adaptation worldview as "from interface X to interface Y" as a fundamental assumption begins to look pretty anemic.

Get it at http://static.repoze.org/casts/repozecast-5-20091230.mp3

Tres and I talk about BFG, the ZTK, and repoze.who.

Happy holidays!

Repozecast #5 released

http://static.repoze.org/casts/repozecast-5-20091230.mp3

Repozecast is an every-so-often podcast. This is the latest one, wherein Tres and I talk about BFG, the ZTK, and repoze.who.

Happy holidays!

• Chris

In response to Tarek's meme

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

   Sphinx. Sphinx is really super. It makes writing and publishing technical documentation less painful by an order of magnitude than it used to be. I'm actually not sure this wasn't a 2008 discovery, but I really started using it "in anger" on my own projects in 2009.

2. What new programming technique did you learn in 2009 ?

   Testing using statement coverage (via Ned Batchelder's coverage). Maintaining 100% statement coverage gives me a lot more confidence about refactoring old code.

3. What’s the name of the open source project you contributed the most in 2009 ? What did you do ?

   Repoze (including BFG). I wrote the lion's share of BFG and contributed to various other Repoze-related pieces of software. I also did a good bit of work on Karl (a collaboration system based on BFG and other Repoze software). I also did some work on Formish/Schemaish, Pylons/Routes, WebOb, and Sphinx.

4. What was the Python blog or website you read the most in 2009 ?

   Reddit's Python section, and the blog entries tossed at me by Planet Python.

5. What are the three top things you want to learn in 2010 ?

   No idea. I just take it as it comes.

I think the most important thing we can do in the open source world (far more important than releasing new cool software) is to improve the quality of the documentation of existing software. Software without documentation is pretty much useless. Software with bad documentation is slightly better than useless, but still nearing uselessness. Software with some good documentation but which has incomplete documentation is better than useless, but only for some period of time: you can't keep telling yourself "I'll do it later" or outsource the job of documentation to some mythical "community": it's a chicken and egg scenario. Writing documentation is part of your job description as a programmer.

As the general quality of software improves due to widespread use of good software engineering techniques such as unit and integration testing, documentation quality is becoming the differentiator between project success and failure.

The most valuable descriptions of highly technical things:

• get to the point quickly without endless theory or architectural navel-gazing.
• start with an example.
• are written in an informal style.
• aren't optimized for brevity, nor are they optimized for testability; instead they are optimized for readability and imparting knowledge.
• can be read more than once; the first time you read one, you may gain different knowledge from it than the fifth time, but you still gain knowledge.
• aren't thrown over the wall as an afterthought, never to be consulted or changed again, they evolve over time as the software evolves.
• are complete
• are written using proper spelling and grammar

Here are some examples:

• The XML-RPC Specification
• AMK's Regex Howto
• Sphinx Documentation
• The Django Book
• Unix Power Tools

If you can create documentation of this quality or better, your project has no chance but to succeed on some level, even if your code is terrible.

Note also that I think that a choice between good docs and better software is a false choice. The act of continually writing documentation always makes software better, because if it's hard to explain, it's probably not very good. Complexity becomes clear very quickly when you need to explain it away; it's usually easier to change the software to be less complex than it is to document something complex. It's frighteningly easy to write undocumentable software.

I have an admission to make. I not a fan of systems that are composed of an integration of heterogenous cooperating service processes.

When I say "service", I mean that the task provides some stovepiped service to a caller (perhaps a "web service" or a database network API). When I say "heterogenous", I mean that the process performs some task that it's "good at", implicitly leaving other processes to perform other different tasks that they are "good at."

When I say that these service processes are "cooperating", what I really mean is that some other process integrates the communications between all these service processes together to form some end-user-facing system. A corollary is that the end-user-facing system probably won't work (at least in any way that is explicable to the end user himself) if any of the service processes dies or gets wedged, even if the remainder of the service processes survive.

Such a setup makes sense sometimes. It makes sense particularly when each service process is a stable, well-tested, widely-used, well-supported, well-defined, well-documented subsystem written in a style or language widely divergent from any other service process. For example, using a Postgres database as a "service process" in a system otherwise composed of Python is not often very controversial.

But even when you use a non-controversial "service processes", the number of service processes in any integration matters. A lot. Each service process a system takes on implies the following:

• Maintenance (e.g. Postgres "vacuum").
• Crash monitoring / email notification.
• Performance monitoring and amelioration.
• Cognitive load related to setup and API.
• Code integration.
• Build automation.

It may be fun to think about tying Postgres, CouchDB, Solr, and some web service together via a frontend that integrates them all. Some folks may consider this highly practical because the result is termed an integration of "best of breed" components. The perceived benefit of such an integration is that the whole becomes better than the parts. The assumption is that a system that uses more off-the-shelf service components and less custom code will be easier to manage, or it will be faster, or it will be prettier, etc.

My experience says that such an assumption is usually just wrong, and that it's usually a mistake in any small or moderately-sized system to rely on more than a single service process when other potentially useful service processes overlap functionality of the single existing service process. When a system doesn't actually strictly need the functionality of extra processes running, I think a high number of service processes is symptom of either optimism about other system capabilities or complexity fascination.

Sometimes the amount of work to add a new service process is actually a net win, if when you introduce a new service process you remove more complexity from the system than you add. But often, for various reasons, you can't. Although you've added some amount of complexity by adding a new service process, but for various practical reasons, you find yourself unable to retire the code or process that the new service process was meant to replace, so you don't remove any complexity from the system. Instead, you just add some.

Let's take a concrete example. Let's say you've developed a system that uses Postgres as a persistence mechanism. Now let's say you have a new requirement: you need to add full-text indexing to the system. Postgres has fairly good full-text indexing capabilities. But you see that Solr has some shiny feature that promises to make your life much better if you'd just be willing to take the time to learn and integrate it.

I say don't fall for it. Just use the Postgres full-text indexer, and fill in any missing functionality with custom code. Your system will only gain complexity by adding Solr, it's almost guaranteed. This is because you're still going to need to manage Postgres; you can't replace it with Solr. Even though Solr's feature set might be better, and its promises may be shinier, the full-text indexing service which Postgres offers is probably good enough unless you're truly reaching to invent requirements.

The addition of a new service process should be a monumental event if you want your system to be as complexity-free and stable as possible. Even if it means the system is slightly slower, or some edge requirement becomes impossible to satisfy, a system that works all the time, every time, and which people can understand is usually more valuable than any individual outlying feature. A system with many moving parts is just naturally harder to understand and manage than one with fewer.

It's bad enough to pile on existing "best-of-breed" services to some integration. But there is also a degenerate case of adding service processes: writing service process code which is only useful in the context of a single integration. For example, some folks believe that composing an application out of many highly focused process-bounded services for every project is a good idea. It's not as if they're reusing some existing service process, they're inventing each service process for purpose of a single integration. Personally, I think this is just not sane. It may be easier to determine responsibilities by separating services between processes, but I think mostly this is best used as a mind game. Once you've figured out the responsibilities of each subsystem, if the potential that you're going to document some particular subsystem well enough for other people to use well is vanishingly small, just put all the subsystems into a single process. If it's not documented for use outside of a particular integration, it doesn't rate being its own service process.

Install via:

$ easy_install -U http://dist.repoze.org/bfg/1.2/simple repoze.bfg

The docs at http://docs.repoze.org/bfg/1.2 have been updated.

The changelog follows:

A couple of days ago I posted a long blog entry about how we (or at least I) tend to use the Zope Component Archictecture. I personally often use the ZCA to introduce dependency injection specifically for unit tests.

The types of dependency injection the ZCA allows for can be used far more generally than just for making unit testing easier. The ZCA works pretty well generally for unit testing dependency injection patterns. But in some cases ZCA assumptions may make the resulting tested code less understandable. In particular, if you put this code under test:

  from zope.component import queryUtility
  from zope.interface import implements
  from mypackage.interfaces import ICatalogQuery

  class CatalogQuery(object):
      implements(ICatalogQuery)
      def __call__(self, context, **kw):
          """ Does something complicated """

  query = CatalogQuery()

  def unit_under_test(context):
      query = queryUtility(ICatalogQuery, default=query)
      return query(context, a=1)

And then in the code doing the testing, you do this:

  import unittest
  from zope.component import getSiteManager

  class Test(unittest.TestCase):

      def setUp(self):
          sm = getSiteManager()
          sm.__init__()

      def test_it(self):
          from mypackage import unit_under_test
          from zope.component import getUtility
          from mypackage.interfaces import ICatalogQuery

          class DummyQuery(object):
              def __call__(self, *arg, **kw):
                  return [1]

          class DummyContext(object):
              pass

          query = DummyQuery()
          context = DummyContext()

          sm = getSiteManager()
          sm.registerUtility(query, ICatalogQuery)
          self.assertEqual(unit_under_test(context), [1])

You note above that the code under test uses a utility lookup to obtain the catalog query API. This is done purely for testing purposes; not for pluggability purposes. However, a casual reader of the above code may assume that the "ICatalogQuery" API is a ZCA "plug point", as advertised by its retrieval as a ZCA "utility". While it may be useful to have an API definition for ICatalogQuery, in most systems such an API is explicitly not pluggable. There's no expectation in most systems with a low-level API like this that an arbitrary user should be expected to be able to plug in a different implementation of ICatalogQuery at all. The getUtility code in the unit under test should not need to exist; it's a misdirection indicating that this is some sort of plug point that is meant to take alternate implementations.

For testing situations such as this one, I've created a package named repoze.depinj (see also http://svn.repoze.org/repoze.depinj/trunk/).

This package is meant to be used instead of the ZCA for unit testing purposes when there is exactly one implementation of the dependency being stubbed out, and the code just needs to be able to find an alternate testing implementation when the system is under test.

Using repoze.depinj, the above code under test becomes:

  from repoze.depinj import lookup

  class CatalogQuery(object):
      def __call__(self, context, **kw):
          """ Does something complicated """

  query = CatalogQuery()

  def unit_under_test(context):
      query = lookup(query)
      return query(context, a=1)

And the code doing the testing becomes:

  import unittest
  from repoze import depinj

  class Test(unittest.TestCase):

      def setUp(self):
          depinj.clear()

      def test_it(self):
          from mypackage import unit_under_test
          from mypackage import query

          class DummyQuery(object):
              def __call__(self, *arg, **kw):
                  return [1]

          class DummyContext(object):
              pass

          dummy_query = DummyQuery()
          context = DummyContext()
          depinj.inject(dummy_query, query)
          self.assertEqual(unit_under_test(context), [1])

While the code is really no more readable in isolation, we have avoided needing to define an ICatalogQuery Zope interface for this interaction. We don't actually use a Zope interface at all.

Note that while we don't need Zope interfaces to document this behavior, we can still document the CatalogQuery API with a Zope interface if we want to. But because we wouldn't use that interface as an adapter marker, it would be impossible for someone to misunderstand as a ZCA plugpoint.

By the way, the answer to "why not monkeypatch instead?" is because I don't own the module-scope codepath . The answer to "why not supply an optional unit_under_test keyword argument supplying the query implementation?" is because it makes generating documentation from source harder. It's also kinda fugly and doesn't account for callables that want to accept truly arbitrary **kw args.

The Zope community currently asserts that there is a benefit to using the current set of conventions and APIs that forms the Zope Component Architecture as a mechanism: use of its mechanisms and concepts can produce a system of higher quality and one which is more understandable than a system built without these mechanisms and concepts.

The promised benefits of obeying the conventions and constraints of the ZCA are:

• A system based on use of the component architecture machinery is "pluggable", meaning that implementations can be swapped out for one another as necessary without changing consumer code.
• If you've paid enough attention to detail over time that the contract spelled by an interface still matches the concrete implementations of the interface, the interface forms a natural set of obviously-correct documentation for that component. For a large system where there is an existing body of interface documentation, it is on average easier for someone new to the system to quickly get productive.
• The concept of adaptation, where one object can be converted to another "type" through introspection of its interfaces, allows the average developer to understand the interaction between discrete components in a large system more easily than in a comparable system which doesn't use adaptation.

I hope to explain via this blog entry that the first promise is completely true, the second is at least half-true, and the third one is just plain false.

The constraints and conventions required by the ZCA are these:

• "Interfaces" are type markers that have metadata.
• An object may declare that it "implements" one or more interfaces.
• An interface may optionally specify a contract: objects which implement the interface are presumed to offer this contract.
• "Adaptation" is the act of converting an object that implements one interface to an object that implements another interface.

Let's compare the promises of following these constraints and conventions to their actual benefits.

Components as Plugpoints

Use of a ZCA "component" as a plugpoint works extremely well. These uses include: testing dependency injection, view overrides, and other extremely high-level policy plugpoints. The ZCA really shines here, as long as you use it in small doses and take care to document the resulting system narratively in terms that don't necessarily require an understanding of component architecture concepts.

I am obviously biased, but I believe repoze.bfg is such a system. However, it actually does not use any documented API defined within in the zope.component package itself. It just uses a ZCA registry as machinery and disuses much of the worldview surrounding the traditional definition of adaptation as documented in various tutorials about the ZCA. It doesn't try to use interfaces as documentation, either. The API docs are generated from the implementation classes rather than from interfaces. The presence of interfaces at all is largely just an implementation detail.

Interfaces As Documentation and Component Creation

Interfaces can act as great documentation. If you're conscientious about maintaining the interface definition for some implementation (or set of implementations), there is incontrovertible benefit to interface documentation.

However, designing a good interface is hard. Can anyone really be expected to sit down and design a formal interface that represents the contract for some component before he even codes up the first implementation of that component and some tests for it? No, of course not. Just can't happen, sorry, unless you've done this particular thing ten times before. API design is a highly iterative process: you throw ten revisions of an API away before you get one that is mostly right.

Therefore, you almost never want to promise that a particular component even has any interface until you've got that interface right in the context of some larger system full of consumers. Until you've got the interface right, the interaction between code and components that might implement some notional interface implied by a component is just an implementation detail. It might be an important implementation detail, but it's often OK if it stays one forever. Some components don't need their interfaces spelled out because they are purely an implementation detail. If there's only one or two consumers of a particular "component", it's probably OK for that component to not have a formal interface. A system that has fewer, more meaningful and highly formalized contracts is usually easier to understand than a system that has many small contracts that all have a high probability of changing radically over time, because the overhead of maintaining the interface definitions for smaller contracts makes them less likely to be true. Wrong definitions are worse than none.

So here we have this system, the ZCA, that uses interfaces as a primary mechanism to perform higher-level operations such as adaptation. You are required to declare an interface for some component to use those higher-level operations. This is required by the worldview that states that adaptation is the conversion of an object implementing one interface to another, period, full stop. Worse, once you define an interface, you are expected to maintain its definition over the lifetime of the project; ideally it should never fall out of sync with the true interface expected by the consumers of an object which implements it. This is usually unreasonable in systems with dozens or hundreds of interface definitions. On projects with many contributors over time, and many interfaces, it's not easy to ensure that interface definitions are both correct and comprehensive. You can define an interface as a plain "marker" without any attribute or method definitions. This essentially means that you promise no contract of an adaptation to this interface. But this is frowned upon in a worldview where the point of adaptation is to convert an object implementing one interface to an object implementing another. It's considered bad form, an "abuse", a cast to a void pointer.

Meanwhile, the adaptation machinery provided by the ZCA is useful in many contexts: particularly for dependency injection in unit tests, so it's extremely tempting to want to make use of it for this purpose alone. Almost all Zope-related projects use the ZCA for unit testing dependency injection. Defining "interfaces" for all your "components" is required there because it's the only way to get what you want for testing dependency injection purposes.

There should actually be very few true plugpoints in any system. Far, far, far fewer, by maybe even an order of magnitude, than those implied by use of adaptation in the typical Zope-related project. Probably nine out of ten uses of adaptation in ZCA-using application source code is there only to provide a place to hang some testing dependency injection point. These are extremely hard to tell apart from the "real" plugpoints in any given system, because they quack exactly like the plugpoint duck. Another questionable use for an interface include using one to look up a "utility" (especially an "unnamed" utility) where the utility is something that has an "obvious" API such as a dictionary. Documenting the API of this obvious thing in the interface definition is just silly. Interfaces tend to be added to any system of consequence for similarly questionable reasons.

So now at this point, if you've been following along, we've built a system that probably has incomplete documentation for its "components", because, really, there aren't any yet. Or maybe very few "true" components. We're not really smart enough to define any "components" yet. We may never be smart enough to define them properly. Our project just uses adaptation for several discrete purposes, none of them actually related to increasing comprehensibility or providing plugpoints: we've just hacked in some dependency injection points for unit tests that use the adapter machinery to good purpose. But we didn't do this to increase comprehensibility; we did it because it makes unit testing easier. We've thus effectively misdirected the casual code reader into believing that there are lots and lots of "components" with well-thought-out "interfaces" when really all we've been trying to do is to test the system more easily. The system is still in major flux, and its "interfaces" only represent some half-baked transitional state of affairs, probably not even correct. The contracts aren't truly formalized yet at all. But it looks pretty impressive.

A developer new to the project needs to be able to detect and discount the adaptations in the code done purely for unit testing dependency injection and convenience and the adaptations that imply true component plugpoints. That task is understandably difficult because we're using the same machinery for entirely different purposes.

As a result, we've actively subverted the original goal: to make the system more understandable.

Use of Adaptation Makes A System More Understandable

The idea that mere use of adaptation as a mechanism to improve large system comprehensibility for a new team member is utterly ludicrous. Mere use of the adaptation pattern adds no additional comprehensibility whatsoever to any system, small or large. Seeing "getMultiAdapter((foo, bar), IFrobnozz)" in code lends no particular insights as to the intent of the developer who wrote it. I'm not talking about syntax here either; the adaptation could be spelled "config['frobnozz](foo, bar)" and it would have the same level of comprehensibility in isolation.

Believing that mere use of any particular adaptation syntax in the code itself helps a new developer understand the intent of the adaptation is a complete fantasy. The only way to explain the intent of any particularly important adaptation is via hand-crafted narrative documentation explaining why some particular adaption takes place in a particular spot. The component architecture doesn't help at all there; you're on your own.

Writing software is hard. Writing software documentation is even harder. But it needs to be done. No tool is going to remove this requirement. Mere use of adaptation in code is no substitute whatsoever for writing high-level design documentation that explains how the system components work together. Using adaptation without explaining the "why" just hurts comprehensibility. Folks who pretend that they've made their system comprehensible simply by using adaptation are fooling themselves in the same way that folks who write doctests but don't test all their code and don't write narrative docs have fooled themselves into thinking they've written both good tests and good docs.

Worse, when we do use adaptation without explanation, a new developer must first go understand the machinery that implements adaptation to get a sense of the intent. That machinery is not particularly well-documented. There is no explanation of the lookup ordering that takes place when components are adapted at all, in particular.

As a result, we've actively subverted the original stated goal: to make the system more understandable.

Conclusions

I believe that following the current trajectory of "the ZCA" is unwise. I am happy to continue using the "zope.component" package, because its machinery works really, really well and it is useful in many scenarios, particularly for unit testing dependency injection and for building systems that can be considered "pluggable" after the fact (systems with additional documentation that explains the how and why of each true plugpoint). However, I no longer buy in to the idea that extremely casual use of the concept of adaptation defined formally as converting one interface to another laid on top of this machinery is of any benefit whatsoever when producing a large system in isolation.

I think we can improve this state of affairs via the following:

• Create a package with only machinery which can be used to implement an adaptation pattern in the package. Document the shit out of it, like it was for your grandma. Don't impose any particular "worldview" on consumers of this machinery: in particular, do not require the use of Zope interfaces as inputs. Maybe provide extra behavior (such as inheritance) when the inputs are interfaces, but allow for registration and lookup markers that are not interfaces.
• Base the API of what is currently zope.component on the machinery in the package I've suggested above. zope.component would be a consumer of this package, and would be free to impose the "an adaptation is the conversion of one interface to another" worldview on its consumers.
• Provide some tools that make it easier to do unit testing dependency injection without needing to declare formal interfaces (this might be the package in the first bullet point). Or at least think up a convention where it's possible for a casual source reader to immediately know that some adaptation is present only for test dependency purposes, as opposed to a true plugpoint.

I'd be willing to work on projects that go in these directions. I'm not willing to work on projects that take the current zope.component and add APIs to it without factoring out the underlying machinery bits, though.

repoze.bfg 1.2a1 is now released. It can be installed via:

easy_install -i http://dist.repoze.org/bfg/1.2/simple repoze.bfg

Or via PyPI.

Documentation for this release exists at http://docs.repoze.org/bfg/1.2/.

This is a major feature release. The new features of the release are detailed in a What's New document.

In particular, this release has a new "imperative configuration" mode that makes it possible to create the simplest repoze.bfg application as a single Python file.

There's a hand-wringing coversation today on the Python list where someone suggested making 2.7 "the last Python in the 2.X series", giving the rationale that people really should be using 3.X and that dropping maintenance for 2.X will give them a reason to move.

I say hooray! But not for the reason you think.

Now I'm pretty sure there will be a Python 2.8, 2.9, and 2.10, 2.11 and so on as sure as the sun will rise. These releases may not be prepared by the same people who have the responsibility of creating releases today, but other folks will probably pop up to fill in for them, even if only unofficially.

But even if there isn't another blessed Python 2 release from the "core team", that's OK. Worse is better.

I get the sense that the folks who are currently pushing new Python releases aren't really that interested in making sure old code runs "forever". It was a tipoff when version 3 started out with minor backwards incompatiblities, of course. This isn't very surprising. Preserving backwards compatibility in complex systems is really just no fun. It's an ugly, thankless slog. Your mistakes stare you in the face daily. It's a lot more fun to do greenfields projects like Python 3000. We all love green-fielding things. (BTW, I think PEP 3003 might declare playtime over though).

In the meantime, though, old Python code never dies, and must continue to run, even when nobody understands it fully any more. With a Python 2 that is very conservatively maintained (or even un-maintained), this is won't be much of a problem. Because there won't be new features, there just won't be any backwards compatibility problems. If Python 2.8 was released as just a set of patches that made Python 2 compile against the latest and greatest set of operating system updates from the major vendors, and had no other changes, fantastic. For legacy systems I maintain, that's change I can believe in. Give me more of that!

I fear that in the actual world, code stability provides more benefit than new syntactical candy and included batteries and obsessive reorganization. It's going to take a long time for people to switch to Python 3. You can try to speed up the process by shepherding people along some decision tree, but they're going to do things at their own pace no matter what decisions you try to make for them. You may want them to arrive at a decision like "I'm going to rewrite this code I barely understand anymore in a backwards incompatible variant of the programming language it was originally written in". That's a reasonable desire. But if the only other choice you provide to them other than that one is "maintain the old programming language version myself", you are only implicitly leaving out the choice "or just rewrite the system in a different language altogether". This implicit choice is often the one people make when they are led along some artificial decision tree.

If a new version of something is not 100% backwards compatible, I think you just need to let the old version and the new version compete on the field of mindshare instead of declaring one the royal blessed successor to the other, and artificially mandating that releases of the old version won't be made. There's no guarantee that today's Python 2 code will become tomorrow's Python 3 code. And that's OK. Python 2 is a fantastic language.

This post has the jaded volume turned to eleven but even so, I’m impressed.

I watched the video of Rob Gietema’s presentation on Deco, the “new way to manage page layout, composite pages and rich content in Plone 4″.  I approach the entire topic with cynicism, dread, and exhaustion on a number of fronts.

Well, color me intrigued.  I’ll probably clutch tightly to my bitter, curmudgeonly outlook.  But I must confess to being impressed, in a number of ways.  I think they’ve done a good job thinking about the problem.  It looks like they’re going slow and being unafraid to refactor ideas and implementations.  It also has, already, a nice visual appeal along with some clever simplifying assumptions to keep the congitive overload under control.  Finally, this is a hard space to work in, technically, and it seems like they have some mad skillz.

There’s still a lot of challenge ahead.  The biggest is what I view as the chief paradox confronting Plone.  On one hand, a significant portion of people are fed up with how some of the features are implemented.  But on the other hand, they’re burned out by a legacy of undead overhauls and don’t have much patience for revolution.  (Sidebar: my opinion is, I sympathize but The Time Has Come.)

Additionally, while it’s a lot of work to do the product, it’s a lot more work to do the “whole” product: documentation, bug fixing, ongoing compatibility, performance, and other stuff over the long haul.  Letting Deco have a long gestation period outside the core would be advisable.  The more baked it is, the more legitimate it will feel when added to Plone, and the less resistance from the undead-overhaul-worriers.

So good luck Deco.  If even 2/3 of the features are implemented, but implemented superbly over the long haul, then Plone will have something sweet to hang its hat on.  (All of the above applies equally to Dexterity, the new content type system.)

Per the announcement, congrats to the new board of directors for the Plone Foundation.  I really, really like that lineup of people. It’s amazing, though, that there were at least five others in the election that would also have been strong to have on the board.

A particular thanks to Hanno Schlichting for once again conducting the election.  I have utter confidence in the things he does, be it Plone Foundation process stuff or gradually untangling years of technical cruft as part of his Plone 5 release manager duties.

I saw the minutes of the Plone Foundation annual member’s meeting (thanks Maurits for the writeup.)  Toby reported that, after spending half of the original CA donation of $100k, the PF in one year has its bank account back to the original amount.  This is primarily due to the sponsorship plan that Jon Stahl helped kick off, tied to the excellent work the Plone.net team at Pilot Sytems (plus Reinout, plus others) have done over the years.

The price point for sponsorship makes it easy to get companies in, and the allocation of funds (e.g. release managers, promotion) makes it very easy to justify.  Since the price point is a yearly fee, I suspect that you’ll see another $50k come in during the next half year, meaning…

…the Plone Foundation is a very successful community-managed outfit.  Congrats to the previous board for orchestrating this turnaround on finances, and here’s to a good 2010 for the Plone Foundation.

I don't like reading sanctimonious treatises on testing either. This is only partly one of those.

First of all, I hereby to resolve to buy Ned Batchelder a case of his favorite for all his excellent work on coverage . This tool measures statement coverage while some program runs. nose has helpful bindings to this tool, which Chris Perkins pointed me at one day, so I probably owe the nose guys and Chris at least one beer apiece too.

Because statement coverage is so easy to measure using Ned's coverage tool, and because it integrates so well with unittest via nose, I have resolved that the open source code I release will all have 100% statement coverage when its tests are run.

As far as I'm concerned, 100% statement coverage is the "least you can do" to make sure the code is of a particular quality; it doesn't mean the code does what it's supposed to, but it does mean that the author has probably grokked most of the code, and hopefully, by extension, the problem domain, in order to figure out how to test all of its statements.

I promised that this wouldn't be entirely cheerleading for testing or a chiding wag of the finger pointed at people who do not test. You can decide for yourself the direct quality benefits of 100% statement coverage via tests; I can only say it works for me. Let's instead consider a big side effect benefit.

With a 100% test coverage invariant, you can reject poor-quality patches with less subjectivity.

A common sort of patch received by open source project maintainers is the "paper towel roll" patch. It's a patch that was coded while its author looked through a paper towel roll at some very specific bit of code in your larger system. The patch is wrong, but its author does not have enough context to know why: it patches some subsystem in a way doesn't make any sense when the entirety of the system is considered. It works, but applying it as-is would be disastrous on some level (documentation requirements / conceptual integrity / code cleanliness, additional unwanted software dependencies, etc).

Paper-towel-roll patches are tricky to deal with as an open source maintainer. Do you throw it away? Do you add the feature implied by the patch in "the right way"? How do you deal with the original submitter? How pissed off is he going to be if you reject it out-of-hand without trying to help him implement it in the right way? Do you even want the feature? Even if you're +0 or +1 on the feature, do you have enough time to deal with doing it properly?

One handy attribute of paper-towel-roll patches is this: they usually come without any tests. We can turn this obvious deficiency into at least one immediate advantage if we require 100% statement coverage.

Without a 100% coverage invariant, your only recourse to reject a poor paper-towel-roll patch will be to say "hey, this patch is pretty terrible", which is obviously subjective and can lead to pointless, time-consuming arguments. It may also require that you explain in great detail the system from end-to-end to the submitter, just so he or she will understand why it's a bad patch. This takes a long time, and the payoff rate is low, because often the submitter is a one-timer: he won't be back to submit patch #2.

With the 100% coverage invariant, you can instead use the less provocative request "this patch doesn't maintain the 100% statement test coverage invariant, could you fix this?" The requirement is unambiguous: either 100% of the statements in the package are executed when the tests run or they're not. You can't argue with a percentage. We've taken the subjectivity out of the initial contact with the patch submitter.

Insisting that a patch maintains the invariant is usually enough to scare off extremely casual paper-towel-roll patch submitters who've submitted something insane which in reality you just don't really want to even think about. This ability to reject a patch without wasting any time is the 99% benefit. Nobody gets their ego bruised, no time is wasted, and your code is still of high quality.

But the submitter might have enough tenacity to write the tests after you ask him to. This will have one of two outcomes:

• the act of needing to write the tests will give the patch submitter enough context to do it correctly, so the second time around, you just will receive a patch that is both correct and has tests.
• the patch submitter may come back with an equally awful solution that just happens to have 100% statement coverage.

This is risky too: he's gone through all the hoop-jumping work and you still have to have "the talk" with him. I haven't really figured out how to deal with this. On the positive side, though, it does mean you've identified somebody who has the tenacity to do something right, and has a high likelihood of submitting more patches in the future. This is extremely good.

I have blogged before about this topic, but what the hell, I'll do it again.

Lots of people want to "use Python" for everything. They'd like to use Python for configuration files, for populating external registries as the result of some import statement, and other things. They inevitably define "use Python" as "put lots of code at module scope in a Python module". I don't really mind this so much, but please, if you do, don't make me use the result. In particular, don't make me use a framework or library in which this behavior is relied upon.

There are reasons that every credible book about Python, in the first chapter, when describing an executable Python script, places the following kind of clause in the script, which allows execution when a module is a script, but prevents it when it is imported:

  if __name__ == '__main__':
      .. do stuff ..

Since the script may (at some point) be imported as a module by another module, and the script author does not want the code inside the "if" block to be executed at import time (in the case where the script is imported by another, __name__ will not be __main__), it makes sense to code "defensively" like this.

Many people seem to forget this "my first Python" lesson when developing larger systems. It's usually insane for the import of a module to have side effects other than the following:

• An import of another module or global.
• Assignment of a variable name in the module to some constant value.
• The addition of a function via a def statement.
• The addition of a class via a class statement.
• Control flow which may handles conditionals for platform-specific handling or failure handling of the above.

Any other sort of logic inside the top level execution path of a Python module (any code that would be executed during "import") should be regarded with great suspicion and perhaps even loathing.

The codepath represented by top-level statements in modules is owned by Python, not by you. Python is in charge of its execution; you are not. You can never predict when it will be executed.

Here's an example of such an antipattern (code is at module scope):

   import atexit
   import logging

   def _exit_function():
       logging.info('process shutting down')

   atexit.register(_exit_function)

This is an example I've taken (in spirit) from a module in the Python standard library.

First of all, any library code that thinks its smart enough to register an atexit function is mistaken. Only an application (and not a library) can possibly be smart enough to register an atexit function, because only an application is responsible for controlling the horizontal and vertical of application startup and shutdown, not any of the libraries employed by the application.

That's a bit besides the point, though. The real problem here is that the code which performs the atexit.register registration executes at module scope. This means that when this module is imported, it has the side effect of registering an atexit function. But what if this module just happened to be imported by another "innocently", but its functionality never used?

It's totally absurd to try to build a system that configures or starts itself based on import time side effects. When you build a system that relies on import time side-effects, you have no control over when the code gets executed: it might get imported much earlier, or much later than you expect it to be. You may not even have any control over why the code is executed: some code scanner, such as a testing tool, may import it, and may fail as a result, possibly at shutdown.

If you do this sort of thing, great. Just keep it to yourself and wash your hands afterwards. And maybe don't do it. OK. Just don't do it; that's the best solution.

You are smart. No one else knows much. In fact, most people are pretty stupid. Everything except what you're doing right now is stupid. Django is stupid. Zope is stupid. Twisted is stupid. Setuptools is stupid. The GIL is stupid. Pylons is stupid. TurboGears is stupid. Guido is stupid. PJE is stupid. Ruby on Rails is a total hack. And oh my god, all these people in IRC are clearly mentally challenged. Why do they keep arguing with you?

I get it: it's not easy being a genius. So, if you don't mind, I have a request. Given that it would certainly not tax you professionally, because it's all so simple and obvious, do you think that you could contribute something to Python or some Python-related project that demonstrates your immense base of knowledge and helps other people?

Ah but wait! You have. I've seen that one package you keep talking about that you wrote and released six months ago. It has a couple of users, even. But look... uh... oh dear.. this is awkward. I don't mean to be negative here. But there's a couple problems. You may be a genius, but at 33% test coverage, you better be almost omniscient. I'm personally not a genius, so I need to rely on something so banal as package test coverage metrics to make decisions about what to use.

Look. Let's be frank. I know you were the smartest guy in your high school class. I realize that in your circle of peers, you are the one who most often actually knows what he's doing. I get the fact that you like puzzles, and you're good at solving them. I realize you believe you are hot shit, and a few other people might too.

But if I may be so bold, here are some suggestions:

• Shut the fuck up. I mean this in the kindest, and gentlest of ways, as maybe a friendly uncle might tell you to "shut the fuck up".
• Work on your capacity to talk with other people without being a complete, utter cock. We've already adjusted our expectations, with you being a programmer and all, we realize you're constitutionally straightforward. But there's a difference between being straightforward and being a dick. Your profession and your history as "the smartest guy in the room" doesn't excuse you from displaying basic courtesy.
• You don't need to prove anything to me or anybody else. I could care less. It's not always about you.
• Write some code that works all the time, every time that lots of people find useful. Maintain that code for 5 years. At this point, you will have something to be proud of.

Thank you.

Thanks to ElectroSoup, the BFG website has a new design. The design is excellent; ElectroSoup does fantastic work should you need some design help.

Florian Schulze has created a buildout that compiles Python 2.4, 2.5, and 2.6 on Snow Leopard. By habit, I generally compile my Python interpreters "by hand" so I have a clue what's going on. I really just looked at Florian's buildout to see what it did and did that by hand.

Here's what I had to do to get Python 2.4 compiled on a fresh Snow Leopard install (not an upgrade) on a 32-bit Mac Book Pro (a Core Duo, not a Core 2 Duo; those have other issues).:

• Install Fink
• Install the readline5-dev library using "fink install" (for readline support in Python). If you don't compile against a "real" readline, Python will probably compile but it will probably buserror in an interactive shell or you'll not have any readline support in the interactive shell.
• Download Python 2.4.6
• Apply Florian's patch to the unpacked source of the Python 2.4.6 tarball. Note that things mostly sorta work without this patch, but (in particular) supervisor won't run without this patch (can't import the "resource" module and other POSIX-related things).
• Use the following command to configure Python 2.4:

      CC="gcc -I/sw/include -L/sw/lib" ./configure MACOSX_DEPLOYMENT_TARGET=10.6 --disable-tk --prefix=$HOME/opt/Python-2.4.6
  

• Use the following command to install Python 2.4:

      CC="gcc -I/sw/include -L/sw/lib" MACOSX_DEPLOYMENT_TARGET=10.6 make install
  

From August 21 - August 24 in Arlington Virginia, 12 people collected for a sprint, nomimally on the topic of creating a new "CMS" (content management system).

Chris Shenton was nice enough to lend his home office for the occasion. In attendance were Chris Shenton, Reed O'Brien, Tres Seaver, Chris McDonough, Malthe Borch, Carlos de la Guardia, Hanno Schlichting, Robert Marianski, Nathaniel Wingfield, Kapil Thangavelu, and Alex Clark.

On the first day, Friday, we had a long discussion about the components that might make up a not-very-simple CMS. We decided that the most important top-level components were:

• A "content repository"
• A workflow system
• An indexing system
• Observability (events)

We spent a good long while talking about the plus's and minuses of various content repository implementations. We defined a content repository as "the place where content goes", separate from any retail application which uses it. We finally decided that to get everyone's buyin that content repositories would be effectively "stupid", managed by a top-level "content manager" node that would handle indexing.

Lots of whiteboarding happened on this day.

On the second day, Malthe and Robert worked on a blog application that used "repoze.filecat" as a backend. Carlos and Chris M. and several other folks helped create a ZODB implementation of a content repository and manager system that had the (very temporary) name of "plite".

On the third day of the sprint, Malthe and Robert had finished their blog application, which used files on the filesytem to represent assets. Chris and Carlos and others had made a content system using plite that was able to add files and page objects.

Things sort of trailed off at this point, as we weren't really able to create an application that used the concepts we had come up with. Any application we could create in the time allotted seemed too simple to effectively use a content repository. So we sort of worked on infrastructure and drank beer (a keg had been strategically placed for the occasion). Malthe added a "static" directive to repoze.bfg.

Things we did well:

• Discuss and agree on components.
• Had fun.

Things we didn't do so well:

• Concentrate on coming up with any demonstration of the technology components we had dreamed up.

In future sprints, I suspect I'd like someone to come armed with an implementation of some of the components we dreamed up, with some idea about how to wire them up into a sample app that would be useful to lots of folks. We sort of bit off more than we could effectively chew by not having a more focused task at hand. But we had fun all the same. This is truly a huge task, so I wasn't particularly surprised (or even disappointed) that we didn't walk away with any truly useful software.

Many thanks to all who participated! Many thanks also to Chris' significant other, Irene, who put up with geeks in her basement and even fed us.

Pictures of the event are here .

if "string" in variable:

#!/usr/bin/env python
print "My Script is running"
import os
if os.path.exists("/tmp"):
    def myfunc():
        x = 4
        return x
class Foo(object):
    y = 1

f = Foo()
print f.y + myfunc()

count = 0
for i in range(5):
    if os.path.exists("/tmp"):
        print "bingo"

def foo():
    print "stuff"

if os.path.exists("/tmp"):
    import sys
    sys.exit(1)

def api_entry_point():
    try:
        obj.method()
    except:
       #lets not bother the developer with this exception.  He should quit out immediately!
       import sys
       sys.exit(0)

Dear Lazyweb,

Marius (and others) reported that Planet Python screws up the links on my articles.  Instead of a full link to the article, it instead does a href="".  Looking at my RSS feed, I see that it correctly has link>http://pauleveritt.wordpress.com/2009/07/15/kudos-to-malthe-for-chameleon/ in the item entries. My feed URL passes the validator test as well.

As you well know, Lazyweb, I switched to WordPress (hosted!) so I wouldn’t have to think, as I’m not very good at that particular trait.  Any ideas on the cause of this?

At the end of May we made the KARL cutover from KARL2 to KARL3.  We have now had almost 8 weeks in production, so we can form impressions about some of the decisions.

For example, we’re using modwsgi as a WSGI server.  In a way, this was a surprising decision.  Chris and I were both a bit skeptical about whether it was a risk worth taking, as we hadn’t used it “in anger” beyond the repoze.org site.  As somewhat a throwaway test, I set it up on the test site we used for OSI to do the user acceptance testing for the 15 or so milestone deliverables on KARL, but with the idea that it wasn’t a permanent decision.

By the time we setup the deployment server, we had months of living with modwsgi under our belt, somewhat by accident.  Far more important, we added Shane Hathaway to the project.  Like Chris Rossi, Shane has been a boon for KARL in many ways.  In this case, Shane had experience with modwsgi and said he would make sure we could stand behind it.

So we put it into production.  We’re running on an 8-core box setup as a Xen server, with the OSI instance getting 3 CPUs.  We started with modwsgi setup to run 2 BFG application processes (and thus ZEO clients), each with 2 threads.  modwsgi gave us some simplification: we didn’t need Apache/mod_proxy + BFG/Paster, with the latter managed by supervisor, possibly with a load balancer in between.  Instead, we let modwsgi in daemon mode handle that.

We also found that we could upgrade the software on the server, send a -GRACEFUL restart to Apache, and have a zero-downtime update to the server.  Which was nice.

Later, though, we found a really useful benefit to having modwsgi in the equation.  We then found that our LiveSearch implementation (and another part of the application) were slower (up to a second for a request) than other parts.  So we put in an Apache alias that matched on those URL types and sent them to a separately-configured BFG instance we call the “ghetto”.  This instance is primarily for catalog requests, so we changed the ZEO cache to be *very* high, but we also flush from the cache on each request any object that isn’t in the catalog.  This has been a boon.

I have an interest in looking later at some more options we gain from modwsgi.  For example, file delivery using wsgi.file_wrapper (which modwsgi fixed its implementation bugs in later releases.)  Also, modwsgi has some facilities for setting limits on the life of a request.

Still, regarding just the basics, we haven’t had much of a hitch at all.  So far, so good.

While I was on vacation, Chris McDonough wrapped up work on BFG 1.0.  The BFG elevator speech hits the nail on the head:

I’ve used BFG on KARL for the last 9 or so months and have found almost everything about it to be a relief. With this 1.0 release, BFG has achieved a sweet spot of stability and maturity combined with ongoing vitality. While BFG is particularly attractive to Zope developers looking for a modern alternative, it is also attractive more generally to Python web developers.  It’s hard to imagine there still being unmet needs, but BFG stands out for people who take the points in the elevator speech above seriously.

All of this due to the massive effort Chris put into it. Not just making it, but mega-documenting it, 100% coverage on tests, answering questions, sample applications, keeping everything up to date on every change, etc. There are quite a few people participating in BFG, but these efforts are generously given because Chris is there to integrate them.

So congrats Chris, and thanks.

After being the guinea pig to iron out bugs and documentation issues for the pairing of repoze.urispace and repoze.dvselect, Eric Brown went above and beyond the call of duty by writing up how to use the pair to theme multiple Plone sites using Deliverance.