Planet Pypefitters

Application Configuration for Frameworks

2008-12-30T09:44:16Z

Idea

It would be useful to find or create a configuration system for use in repoze.bfg (and other frameworks) that had the same purpose as zope.configuration (aka ZCML), but rids itself of the hard dependency on zope.component.

Rationale

Most frameworks need some form of explicit configuration step. This usually happens in two phases. The first phase is to load the configuration data from some file(s). The second phase is an "execution" phase which operates against the configuration data. The execution phase is not passive, it's active. The return value of the execution phase is not terribly important, it is only executed in order to perform arbitrary setup.

It is often useful to "box this step in" to some known time period. Instead of relying on import ordering to do ad-hoc configuration, making the execution step explicit ensures that the configuration happens in a known order as a result of a single function call. It also forms a convention which can help load "the right" configuration for functional testing and other purposes.

The specific code that is executed depends on what the user puts in the configuration data. The configuration data is typically explicitly not Python: Python is often too powerful for this task. Instead, there is some controlled set of directives in a structured format that are available for users to inject into a configuration file.

Most frameworks also need somewhere to stash and retrieve application-specific settings (as opposed to module-scope settings), as more than one instance of an application using the framework may exist in the same process space.

zope.configuration provides such a system but it assumes that the framework developer is willing to "bite off" the entire Zope component architecture. Also, zope.configuration makes the assumption that there is a single global configuration; this is a limitation that makes it hard to run multiple instances of an application using the configuration system in the same process.

Requirements

The pattern for loading and executing directives from configuration is similar to the pattern exposed by the design of zope.configuration. E.g. there is a "load" phase which reads data from a config file, and an execute phase which executes the configuration; these can be invoked independently as necessary.
The input format should be overrideable (e.g. YAML vs. XML)
No automatic dependency on zope.component (ie. the configuration registry itself is not a component architecture registry). It should be possible to use a zope.component registry within the system, but the configuration system should not assume it is populating a component architecture registry.
The configuration system should allow developers to extend the system with new directives; for example, someone should be able to plug in an implementation of a directive that, when spelled in the config file by a user, could cause a set of Routes .connect statements to be run, or a ZCA utility to be registered.
The configuration system should be able to raise an error when an unknown directive is specified or the options to a particular directive are somehow wrong.
Configuration should be able to span multiple files. There should be a way to include configuration from another Python package or filesystem location.
The configuration system should provide an application-specific data structure to both retrieve and temporarily store data at runtime.

Anyway, that's enough for now. Just thinking out loud.

Avoiding Silos: “link” as a first-class object

2008-12-27T19:09:18Z

One of the constant annoyances to me in web applications is the self-proclaimed need for those applications to know about everything and do everything, and only spotty ad hoc techniques for including things from other applications.

An example might be blog navigation or search, where you can only include data from the application itself. Or "Recent Posts" which can only show locally-produce posts. What if I post something elsewhere? I have to create some shoddy placeholder post to refer to it. Bah! Underlying this the data is usually structured in a specific way, with the HTML being a sort of artifact of the database, the markup transient and a slave to the database’s structure.

An example of this might be a recent post listing like:


  for post in recent_posts:
    
      
        {{post.title}}

There’s clearly no room for exceptions in this code. I am thus proposing that any system like this should have the notion of a "link" as a first-class object. The code should look like this:


  for post in recent_posts:
    
      {{post.link()}}

Just like with changing IDs to links in service documents, the template doesn’t actually look any more complicated than it did before (simpler, even). But now we can use simple object-oriented techniques to create first-class links. The code might look like:

class Post(SomeORM):
    def url(self):
        if self.type == 'link':
            return self.body
        else:
            base = get_request().application_url
            return '%s/%s/%s/%s' % (
                base, self.year, self.month, self.slug)

    def link(self):
        return html('%s') % (
            self.url(), self.title)

The addition of the .url() method has the obvious effect of making these offsite links work. Using a .link() method has the added advantage of allowing things like HTML snippets to be inserted into the system (even though that is not implemented here). By allowing arbitrary HTML in certain places you make it possible for people to extend the site in little ways — possibly adding markup to a title, or allowing an item in the list that actually contains two URLs (e.g., href="url1">Some Item ( href="url2">via)).

In the context of Python I recommend making these into methods, not properties, because it allows you to later add keyword arguments to specialize the markup (like post.link(abbreviated=True)).

One negative aspect of this is that you cannot affect all the markup through the template alone, you may have to go into the Python code to change things. Anyone have ideas for handling this problem?

"I Am Not A Programmer"

2008-12-21T17:59:20Z

So... you're developing or modifying an existing web or GUI application. You send an email to the maillist or inject into IRC asking for help:

  I need help!  I just need to do .  I am not a programmer.

Hey... guess what! Congratulations, you just became a programmer by deciding to do what you're trying to do! So you can't hide behind that particular "I'm not a programmer" skirt anymore, you gave that right up when you opened the first file in your editor.

None of us are really "programmers"... there's no cabal or test we all take. We're all trying to figure it out just like you. So save the excuses, use Google and the powers of logic to ask a reasonable question that pertains to your problem and maybe someone will help.

Plugin-Izing an Application

2008-12-19T03:53:16Z

I just tried to create a plugin system as per André Roberge's specification using the Zope Component Architecture. The program which is being made pluggable is a program written by the effbot. I'm sure the effbot will be thrilled. ;-)

This is not exactly the kind of problem domain where you'd expect to need plugins. It's a parser and needs to operate very quickly, and most plugin architectures (including the ZCA) are based on indirections that aren't really optimized to be inside the inner loop of programs that require very high speed. But that said, I suppose it's as good an application as any to introduce plugins into as a demonstration, as long as you're not judging on speed difference between the pluggable version and the original. I didn't measure speed, as I'd never use the component architecture "for real" in this particular program.

After completing pluginizing the application, a few things strike me. First of all, in order to get the ZCA to parse ZCML, it needs a lot of dependencies. We knew this, of course, but it's pretty striking exactly how many are required when you're dealing with this simple of a problem. Here they are:

  pytz-2008i-py2.4.egg
  zope.component-3.4.0-py2.4.egg
  zope.configuration-3.4.0-py2.4.egg
  zope.deferredimport-3.4.0-py2.4.egg
  zope.deprecation-3.4.0-py2.4.egg
  zope.event-3.4.0-py2.4.egg
  zope.exceptions-3.5.2-py2.4.egg
  zope.i18n-3.6.0-py2.4.egg
  zope.i18nmessageid-3.4.3-py2.4-macosx-10.5-i386.egg
  zope.interface-3.4.1-py2.4-macosx-10.5-i386.egg
  zope.location-3.4.0-py2.4.egg
  zope.proxy-3.4.1-py2.4-macosx-10.5-i386.egg
  zope.proxy-3.4.2-py2.4-macosx-10.5-i386.egg
  zope.publisher-3.5.4-py2.4.egg
  zope.schema-3.4.0-py2.4.egg
  zope.security-3.5.2-py2.4-macosx-10.5-i386.egg
  zope.testing-3.5.1-py2.4.egg
  zope.traversing-3.5.0a4-py2.4.egg

That's just absurd. The publisher? zope.security? zope.location? We really need to detangle these dependencies this at some point to make it reasonable for very small applications to use zope.configuration (ZCML). Most of these dependencies are actually dependencies of ZCML (zope.configuration), rather than the ZCA "proper" (zope.component).

In any case, on to the actual plugin-ization. To no one's surprise, the resulting plugin-ized version of the application is much more complex.

To actually plugin-ize the app, I made each operator into a named utility using the ZCA, configured via ZCML. The name of the utility is the operator itself. Each utility is registered via ZCML, ala:

One utility is registered for each operator required. Accordingly, the application's tokenize() function now looks up each operator via a utility lookup:

  def tokenize(program):
      for number, operator in re.findall("\s*(?:(\d+)|(\*\*|.))", program):
          if number:
              yield literal_token(int(number))
          elif operator:
              utility = queryUtility(IOperator, name=operator)
              if utility is None:
                  raise SyntaxError("unknown operator: %r" % operator)
              yield utility()
          else:
              raise SyntaxError("unknown operator: %r" % operator)
      yield end_token()

When an operator is encountered, queryUtility is run; it will try to find a named utility (or it won't, and will raise a syntax error). The utilities themselves are classes. For example, the operator_add_token utility is defined as:

    class operator_add_token(object):
        """ plugin """
        lbp = 10
        def nud(self, context):
            return context.expression(100)

        def led(self, context, left):
            return left + context.expression(10)

Note that I changed the application to use a "context" object rather than module-scope globals to find the token and expression callable, so this definition isn't exactly like the one defined by the original application, but it still does the same thing.

In any case, all of the operators are defined in the same file (calc.operators). This is just for convenience; they could be defined all over hell and gone if you liked (you'd just change the ZCML to refer to a utility component at a different dotted name). And of course if you included more ZCML (which can cross files too), you'd could add another operator or override the implementation of an existing operator. I don't have very much imagination, so I did neither. You get the point, hopefully.

I suppose this is about the simplest possible example of using the Zope Component Architecture to create a pluggable application. You can also define your own ZCML directives (e.g. I could have made the ZCML read something like . You also don't really need to use ZCML, it's just a shell around the actual component architecture that makes clear the difference between code and configuration.

The result of my toying around exists at http://plope.com/static/misc/calc-0.2.tar.gz . Run "setup.py install" (in a virtualenv, to prevent polluting your system Python with the above libraries) to install it. To run it subsequently, run "bin/calctest" (a setuptools console script).

Javascript Status Message Display

2008-12-17T18:03:15Z

In a little wiki I’ve been playing with I’ve been trying out little ideas that I’ve had but haven’t had a place to actually implement them. One is how notification messages work. I’m sure other people have done the same thing, but I thought I’d describe it anyway.

A common pattern is to accept a POST request and then redirect the user to some page, setting a status message. Typically the status message is either set in a cookie or in the session, then the standard template for the application has some code to check for a message and display it.

The problem with this is that this breaks all caching — at any time any page can have some message injected into it, basically for no reason at all. So I thought: why not do the whole thing in Javascript? The server will set a cookie, but only Javascript will read it.

The code goes like this; on the server (easily translated into any framework):

resp.set_cookie('flash_message', urllib.quote(msg))

I quote the message because it can contain characters unsafe for cookies, and URL quoting is a particularly easy quoting to apply.

Then I have this Javascript (using jQuery):

$(function () {
    // Anything in $(function...) is run on page load
    var flashMsg = readCookie('flash_message');
    if (flashMsg) {
        flashMsg = unescape(flashMsg);
        var el = $(''+
          ''+
          'X'+
          flashMsg + '');
        $('a#flash-message-button', el).bind(
          'click', function () {
            $(this.parentNode.parentNode).remove();
        });
        $('#body').prepend(el);
        eraseCookie('flash_message');
    }
});

Note that I’ve decided to treat the flash message as HTML. I don’t see a strong risk of injection attack in this case, though I must admit I’m a little unclear about what the normal policies are for cross-domain cookie setting.

I use these cookie functions because oddly I can’t find cookie handling functions in jQuery. It’s always weird to me how primitive document.cookie is. Anyway, CSS looks like this:

#flash-message {
  margin: 0.5em;
  border: 2px solid #000;
  background-color: #9f9;
  -moz-border-radius: 4px;
  text-align: center;
}

#flash-message-close {
  float: right;
  font-size: 70%;
  margin: 2px;
}

a#flash-message-button {
  text-decoration: none;
  color: #000;
  border: 1px solid #9f9;
}

a#flash-message-button:hover {
  border: 1px solid #000;
  background-color: #009;
  color: #fff;
}

This doesn’t have non-Javascript fallback, but I think that’s okay. This isn’t something that a spider would ever see (since spiders shouldn’t be submitting forms that result in update messages). Accessible browsers generally implement Javascript so that’s also not particularly a problem, though there may be additional hints I could give in CSS or Javascript to help make this more readable (if there’s a message, it should probably be the first thing read on the page).

Another common component of pages that varies separate from the page itself is logged-in status, but that’s more heavily connected to your application. Get both into Javascript and you might be able to turn caching way up on a lot of your pages.

Using pip Requirements

2008-12-17T01:30:21Z

Following onto a set of recent posts (from James, me, then James again), Martijn Faassen wrote a description of Grok’s version management. Our ideas are pretty close, but he’s using buildout, and I’ll describe out to do the same things with pip.

Here’s a kind of development workflow that I think works well:

A framework release is prepared. Ideally there’s a buildbot that has been running (as Pylons has, for example), so the integration has been running for a while.
People make sure there are released versions of all the important components. If there are known conflicts between pieces, libraries and the framework update their install_requires in their setup.py files to make sure people don’t use conflicting pieces together.
Once everything has been released, there is a known set of packages that work together. Using a buildbot maybe future versions will also work together, but they won’t necessarily work together with applications built on the framework. And breakage can also occur regardless of a buildbot.
Also, people may have versions of libraries already installed, but just because they’ve installed something doesn’t mean they really mean to stay with an old version. While known conflicts have been noted, there’s going to be lots of unknown conflicts and future conflicts.
When starting development with a framework, the developer would like to start with some known-good set, which is a set that can be developed by the framework developers, or potentially by any person. For instance, if you extend a public framework with an internal framework (or even a public sub-framework like Pinax) then the known-good set will be developed by a different set of people.
As an application is developed, the developer will add on other libraries, or use some of their own libraries. Development will probably occur at the trunk/tip of several libraries as they are developed together.
A developer might upgrade the entire framework, or just upgrade one piece (for instance, to get a bug fix they are interested in, or follow a branch that has functionality they care about). The developer doesn’t necessarily have the same notion of "stable" and "released" as the core framework developers have.
At the time of deployment the developer wants to make sure all the pieces are deployed together as they’ve tested them, and how they know them to work. At any time, another developer may want to clone the same set of libraries.
After initial deployment, the developer may want to upgrade a single component, if only to test that an upgrade works, or if it resolves a bug. They may test out combinations only to throw them away, and they don’t want to bump versions of libraries in order to deploy new combinations.

This is the kind of development pattern that requirement files are meant to assist with. They can provide a known-good set of packages. Or they can provide a starting point for an active line of development. Or they can provide a historical record of how something was put together.

The easy way to start a requirement file for pip is just to put the packages you know you want to work with. For instance, we’ll call this project-start.txt:

Pylons
-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

You can plug away for a while, and maybe you decide you want to freeze the file. So you do:

$ pip freeze -r project-start.txt project-frozen.txt

By using -r project-start.txt you give pip freeze a template for it to start with. From that, you’ll get project-frozen.txt that will look like:

Pylons==0.9.7
-e svn+http://mycompany/svn/MyApp/trunk@1045#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk@1058#egg=MyLibrary

## The following requirements were added by pip --freeze:
Beaker==0.2.1
WebHelpers==0.9.1
nose==1.4
# Installing as editable to satisfy requirement INITools==0.2.1dev-r3488:
-e svn+http://svn.colorstudy.com/INITools/trunk@3488#egg=INITools-0.2.1dev_r3488

At that point you might decide that you don’t care about the nose version, or you might have installed something from trunk when you could have used the last release. So you go and adjust some things.

Martijn also asks: how do you have framework developers maintain one file, and then also have developers maintain their own lists for their projects?

You could start with a file like this for the framework itself. Pylons for instance could ship with something like this. To install Pylons you could then do:

$ pip -E MyProject install \
>    -r http://pylonshq.com/0.9.7-requirements.txt

You can also download that file yourself, add some comments, rename the file and add your project to it, and use that. When you freeze the order of the packages and any comments will be preserved, so you can keep track of what changed. Also it should be ameniable to source control, and diffs would be sensible.

You could also use indirection, creating a file like this for your project:

-r http://pylonshq.com/0.9.7-requirements.txt
-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

That is, requirements files can refer to each other. So if you want to maintain your own requirements file alongside the development of an upstream requirements file, you could do that.

A Few Corrections To “On Packaging”

2008-12-14T21:53:25Z

James Bennett recently wrote an article on Python packaging and installation, and Setuptools. There’s a lot of issues, and writing up my thoughts could take a long time, but I thought at least I should correct some errors, specifically category errors. Figuring out where all the pieces in Setuptools (and pip and virtualenv) fit is difficult, so I don’t blame James for making some mistakes, but in the interest of clarifying the discussion…

I will start with a kind of glossary:

Distribution:: This is something-with-a-setup.py. A tarball, zip, a checkout, etc. Distributions have names; this is the name in setup(name="…") in the setup.py file. They have some other metadata too (description, version, etc), and Setuptools adds to that metadata some. Distutils doesn’t make it very easy to add to the metadata — it’ll whine a little about things it doesn’t know, but won’t do anything with that extra data. Fixing this problem in Distutils is an important aspect of Setuptools, and part of what Distutils itself unsuitable as a basis for good library management.
package/module:: This is something you import. It is not the same as a distribution, though usually a distribution will have the same name as a package. In my own libraries I try to name the distribution with mixed case (like Paste) and the package with lower case (like paste). Keeping the terminology straight here is very difficult; and usually it doesn’t matter, but sometimes it does.
Setuptools The Distribution:: This is what you install when you install Setuptools. It includes several pieces that Phillip Eby wrote, that work together but are not strictly a single thing.
setuptools The Package:: This is what you get when you do import setuptools. Setuptools largely works by monkeypatching distutils, so simply importing setuptools activates its functionality from then on. This package is entirely focused on installation and package management, it is not something you should use at runtime (unless you are installing packages as your runtime, of course).
pkg_resources The Module:: This is also included in Setuptools The Distribution, and is for use at runtime. This is a single module that provides the ability to query what distributions are installed, metadata about those distributions, information about the location where they are installed. It also allows distributions to be "activated". A distribution can be available but not activated. Activating a distribution means adding its location to sys.path, and probably you’ve noticed how long sys.path is when you use easy_install. Almost everything that allows different libraries to be installed, or allows different versions of libraries, does it through some management of sys.path. pkg_resources also allows for generic access to "resources" (i.e., non-code files), and let’s those resources be in zip files. pkg_resources is safe to use, it doesn’t do any of the funny stuff that people get annoyed with.
easy_install:: This is also in Setuptools The Distribution. The basic functionality it provides is that given a name, it can search for package with that distribution name, and also satisfying a version requirement. It then downloads the package, installs it (using setup.py install, but with the setuptools monkeypatches in place). After that, it checks the newly installed distribution to see if it requires any other libraries that aren’t yet installed, and if so it installs them.
Eggs the Distribution Format:: These are zip files that Setuptools creates when you run python setup.py bdist_egg. Unlike a tarball, these can be binary packages, containing compiled modules, and generally contain .pyc files (which are portable across platforms, but not Python versions). This format only includes files that will actually be installed; as a result it does not include doc files or setup.py itself. All the metadata from setup.py that is needed for installation is put in files in a directory EGG-INFO.
Eggs the Installation Format:: Eggs the Distribution Format are a subset of the Installation Format. That is, if you put an Egg zip file on the path, it is installed, no other process is necessary. But the Installation Format is more general. To have an egg installed, you either need something like DistroName-X.Y.egg/ on the path, and then an EGG-INFO/ directory under that with the metadata, or a path like DistroName.egg-info/ with the metadata directly in that directory. This metadata can exist anywhere, and doesn’t have to be directly alongside the actual Python code. Egg directories are required for pkg_resources to activate and deactivate distributions, but otherwise they aren’t necessary.
pip:: This is an alternative to easy_install. It works somewhat differently than easy_install, but not much. Mostly it is better than easy_install, in that it has some extra features and is easier to use. Unlike easy_install, it downloads all distributions up-front, and generates the metadata to read distribution and version requirements. It uses Setuptools to generate this metadata from a setup.py file, and uses pkg_resources to parse this metadata. It then installs packages with the setuptools monkeypatches applied. It just happens to use an option python setup.py –single-version-externally-managed, which gets Setuptools to install packages in a more flat manner, with Distro.egg-info/ directories alongside the package. Pip installs eggs! I’ve heard the many complaints about easy_install (and I’ve had many myself), but ultimately I think pip does well by just fixing a few small issues. Pip is not a repudiation of Setuptools or the basic mechanisms that easy_install uses.
PoachEggs:: This is a defunct package that had some of the features of pip (particularly requirement files) but used easy_install for installation. Don’t bother with this, it was just a bridge to get to pip.
virtualenv:: This is a little hack that creates isolated Python environments. It’s based on virtual-python.py, which is something I wrote based on some documentation notes PJE wrote for Setuptools. Basically virtualenv just creates a bin/python interpreter that has its own value of sys.prefix, but uses the system Python and standard library. It also installs Setuptools to make it easier to bootstrap the environment (because bootstrapping Setuptools is itself a bit tedious). I’ll add pip to it too sometime. Using virtualenv you don’t have to worry about different library versions, because for any one environment you will probably only need one version of a library. On any one machine you probably need different versions, which is why installing packages system-wide is problematic for most libraries. (I’ve been meaning to write a post on why I think using system packaging for libraries is counter-productive, but that’ll wait for another time.)

So… there’s the pieces involved, at least the ones I can remember now. And I haven’t really discussed .pth files, entry points, sys.path trickery, site.py, distutils.cfg… sadly this is a complex state of affairs, but it was also complex before Setuptools.

There are a few things that I think people really dislike about Setuptools.

First, zip files. Setuptools prefers zip files, for reasons that won’t mean much to you, and maybe are more historical than anything. When a distribution doesn’t indicate if it is zip-safe, Setuptools looks at the code and sees if it uses __file__, an if not it presumes that the code is probably zip-safe. The specific problem James cites is what appears to be a bug in Django, that Django looks for code and can’t traverse into zip files in the same way that Python itself can. Setuptools didn’t itself add anything to Python to make it import zip files, that functionality was added to Python some time before. The zipped eggs that Setuptools installs are using existing (standard!) Python functionality.

That said, I don’t think zipping libraries up is all that useful, and while it should work, it doesn’t always, and it makes code harder to inspect and understand. So since it’s not that useful, I’ve disabled it when pip installs packages. I also have had it disabled on my own system for years now, by creating a distutils.cfg file with [easy_install] zip_ok = False in it. Sadly App Engine is forcing me to use zip files again, because of its absurdly small file limits… but that’s a different topic. (There is an experimental pip zip command mostly intended for App Engine.)

Another pain point is version management with setup.py and Setuptools. Indeed it is easy to get things messed up, and it is easy to piss people off by overspecifying, and sometimes things can get in a weird state for no good reason (often because of easy_install’s rather naive leap-before-you-look installation order). Pip fixes that last point, but it also tries to suggest more constructive and less painful ways to manage other pieces.

Pip requirement files are an assertion of versions that work together. setup.py requirements (the Setuptools requirements) should contain two things: 1: all the libraries used by the distribution (without which there’s no way it’ll work) and 2: exclusions of the versions of those libraries that are known not to work. setup.py requirements should not be viewed as an assertion that by satisfying those requirements everything will work, just that it might work. Only the end developer, testing the system together, can figure out if it really works. Then pip gives you a way to record that working set (using pip freeze), separate from any single distribution or library.

There’s also a lot of conflicts between Setuptools and package maintainers. This is kind of a proxy war between developers and sysadmins, who have very different motivations. It deserves a post of its own, but the conflicts are about more than just how Setuptools is implemented.

I’d love if there was a language-neutral library installation and management tool that really worked. Linux system package managers are absolutely not that tool; frankly it is absurd to even consider them as an alternative. So for now we do our best in our respective language communities. If we’re going to move forward, we’ll have to acknowledge what’s come before, and the reasoning for it.

Bounty Solved

2008-12-14T09:01:49Z

Brandon spake verily:

 I have solved our problem with triple-quotes.

 The problem is that Emacs expects PyFlakes to only output error
 messages, but on a syntax error, PyFlakes prints out an error message,
 then the *entire* contents of the module that it cannot import, and
 *finally* a line that contains a number of spaces equal to the offset
 into the file of the syntax error (in the case of my real-world file,
 the triple-quote was 3,896 characters into the file, so PyFlake's line 
 of spaces was that long as well).  The offending code is in the
 "pyflakes" command-line program and looks like this:

    print >> sys.stderr, 'could not compile %r:%d:' % (filename, lineno)
    print >> sys.stderr, line
    print >> sys.stderr, " " * (offset-2), "^"

 By removing or commenting out those last two lines, so that PyFlakes 
 only outputs its error message, you will stop flooding the Emacs
 regular-expression engine with data.  It's actually the long line of
 spaces that causes the problem, and it's one regular expression that's
 really sensitive to it (this is from the Emacs 22 flymake.el):

     ;; ant/javac
     (" *\\(\\[javac\\] *\\)?\\(\\([a-zA-Z]:\\)?[^:(\t\n]+\\)\:\\([0-9]+\\)\:[ \t\n]*\\(.+\\)"
      2 4 nil 5))

 For some reason (I'm not an RE engine guru), something about the way
 it's matching spaces takes exponential time when given several thousand
 spaces.  Go figure.  Anyway, I see no point in throwing anything but 
 error messages at Flymake, so I comment out both the printing of the
 module and the spaces from PyFlakes.

And sure nuff, that solves it. Brandon gets a well-deserved $500.00.

Bounty: Make Emacs Flymake + PyFlakes Setup Not Spin On Triple-Quote Entry

2008-12-13T18:11:53Z

Several months ago, I created a blog entry where I documented an integration of PyFlakes with Emacs flymake-mode that is insanely useful when writing Python code.

This setup has been working fantastically, except for one thing. When you type a single set of triple-quotes ( """ ), and stop typing, the Emacs process will begin to "spin", the CPU usage hits 100%. If you type the ending set of triple-quotes, although they don't immediately appear on the screen, Emacs will return editing control to you perhaps several minutes later with your keystrokes evident; the process continues to spin during the waiting period. If you never type the ending triple-quotes, the process will spin "forever".

I have worked around this so far: every time I need some triple-quoted string, I type two sets of triple quotes very quickly ( """ """ ), which presumably prevents the flymake-mode syntax checker from kicking in. Then I just back up and insert the text I want. However, sometimes I forget to type the ending triple-quotes, and it sucks to wait for the process to stop tripping over itself and it sucks worse to have to kill it and start it again. It kills momentum during coding something awful.

I am not interested in debugging this problem personally (I have no experience writing Emacs Lisp). But I'd really, really like to get it fixed. Therefore, I hereby offer a $500.00 bounty to anyone who has the know-how to reproduce and remediate this problem (I use Carbon Emacs on Mac OS X although I believe the problem is not OS-platform-specific, it will probably be evident in any version of GNU Emacs). I will pay the first person that comes up with a solution via PayPal or check or however it works for you really after you've demonstrated that the problem is solved and provided instructions on how others can fix it themselves (by applying a patch, or whatever needs be done). Send me a mail (chrism@plope.com) if you've decided to work on it.

If you take a look at the problem, and you think it's going to take longer than $500.00 is worth, let me know, and I'll see if I can't find some other folks to cough up some more dough.

lxml: an underappreciated web scraping library

2008-12-11T03:19:44Z

When people think about web scraping in Python, they usually think BeautifulSoup. That’s okay, but I would encourage you to also consider lxml.

First, people think BeautifulSoup is better at parsing broken HTML. This is not correct. lxml parses broken HTML quite nicely. I haven’t done any thorough testing, but at least the BeautifulSoup broken HTML example is parsed better by lxml (which knows that elements should go inside

elements).

Second, people feel lxml is harder to install. This is correct. BUT, lxml 2.2alpha1 includes an option to compile static versions of the underlying C libraries, which should improve the installation experience, especially on Macs. To install this new way, try:

$ STATIC_DEPS=true easy_install 'lxml>=2.2alpha1'

One you have lxml installed, you have a great parser (which happens to be super-fast and that is not a tradeoff). You get a fairly familiar API based on ElementTree, which though a little strange feeling at first, offers a compact and canonical representation of a document tree, compared to more traditional representations. But there’s more…

One of the features that should be appealing to many people doing screen scraping is that you get CSS selectors. You can use XPath as well, but usually that’s more complicated (for example). Here’s an example I found getting links from a menu in a page in BeautifulSoup:

from BeautifulSoup import BeautifulSoup
import urllib2
soup = BeautifulSoup(urllib2.urlopen('http://java.sun.com').read())
menu = soup.findAll('div',attrs={'class':'pad'})
for subMenu in menu:
    links = subMenu.findAll('a')
    for link in links:
        print "%s : %s" % (link.string, link['href'])

Here’s the same example in lxml:

from lxml.html import parse
doc = parse('http://java.sun.com').getroot()
for link in doc.cssselect('div.pad a'):
    print '%s: %s' % (link.text_content(), link.get('href'))

lxml generally knows more about HTML than BeautifulSoup. Also I think it does well with the small details; for instance, the lxml example will match elements in class="pad menu"> (space-separated classes), which the BeautifulSoup example does not do (obviously there are other ways to search, but the obvious and documented technique doesn’t pay attention to HTML semantics).

One feature that I think is really useful is .make_links_absolute(). This takes the base URL of the page (doc.base) and uses it to make all the links absolute. This makes it possible to relocate snippets of HTML or whole sets of documents (as with this program). This isn’t just href> links, but stylesheets, inline CSS with @import statements, background attributes, etc. It doesn’t see quite all links (for instance, links in Javascript) but it sees most of them, and works well for most sites. So if you want to make a local copy of a site:

from lxml.html import parse, open_in_browser
doc = parse('http://wiki.python.org/moin/').getroot()
doc.make_links_absolute()
open_in_browser(doc)

open_in_browser serializes the document to a temporary file and then opens a web browser (using webbrowser).

Here’s an example that compares two pages using lxml.html.diff:

from lxml.html.diff import htmldiff
from lxml.html import parse, tostring, open_in_browser, fromstring

def get_page(url):
    doc = parse(url).getroot()
    doc.make_links_absolute()
    return tostring(doc)

def compare_pages(url1, url2, selector='body div'):
    basis = parse(url1).getroot()
    basis.make_links_absolute()
    other = parse(url2).getroot()
    other.make_links_absolute()
    el1 = basis.cssselect(selector)[0]
    el2 = other.cssselect(selector)[0]
    diff_content = htmldiff(tostring(el1), tostring(el2))
    diff_el = fromstring(diff_content)
    el1.getparent().insert(el1.getparent().index(el1), diff_el)
    el1.getparent().remove(el1)
    return basis

if __name__ == '__main__':
    import sys
    doc = compare_pages(sys.argv[1], sys.argv[2], sys.argv[3])
    open_in_browser(doc)

You can use it like:

$ python lxmldiff.py 
'http://wiki.python.org/moin/BeginnersGuide?action=recall&rev=70' 
'http://wiki.python.org/moin/BeginnersGuide?action=recall&rev=81' 
'div#content'

Another feature lxml has is form handling. All the cool sexy new sites use minimal forms, but searching for "registration forms" I get this nice complex form. Let’s look at it:

>>> from lxml.html import parse, tostring
>>> doc = parse('http://www.actuaryjobs.com/cform.html').getroot()
>>> doc.forms
[]
>>> form = doc.forms[0]
>>> form.inputs.keys()
['thank_you_title', 'City', 'Zip', ... ]

Now we have a form object. There’s two ways to get to the fields: form.inputs, which gives us a dictionary of all the actual elements (and textarea and select). There’s also form.fields, which is a dictionary-like object. The dictionary-like object is convenient, for instance:

>>> form.fields['cEmail'] = 'me@example.com'

This actually updates the input element itself:

>>> tostring(form.inputs['cEmail'])
''

I think it’s actually a nicer API than htmlfill and can serve the same purpose on the server side.

But then you can also use the same interface for scraping, by filling fields and getting the submission. That looks like:

>>> import urllib
>>> action = form.action
>>> data = urllib.urlencode(form.form_values())
>>> if form.method == 'GET':
...     if '?' in action:
...         action += '&' + data
...     else:
...         action += '?' + data
...     data = None
>>> resp = urllib.urlopen(action, data)
>>> resp_doc = parse(resp).getroot()

Lastly, there’s HTML cleaning. I think all these features work together well, do useful things, and it’s based on an actual understanding HTML instead of just treating tags and attributes as arbitrary. (Also if you really like jQuery, you might want to look at pyquery, which is a jQuery-like API on top of lxml).

Public launch of Stanford Intellectual Property Litigation Clearinghouse

2008-12-09T01:04:48Z

At long last, we’ve finally launched the Stanford IPLC website that I’ve been working on for the past year. It’s quite nice to finally have something out there that I can show people, though I know its definitely more of a niche area of interest, as not everyone is probably as interested in intellectual property litigation as I am. :)

This site is running Pylons of course, with various other technologies I’m unable to disclose powering the back-end.

Note that signing up requires a valid e-mail address as e-mail confirmations are sent out to them. For those that are keen to keep up on what’s going on with patent litigation, hopefully our website can help out.

Note To Bloggers

2008-12-08T03:59:27Z

Nope, I don't remember you releasing that software last week. That means it's useless to tell me that some bugs were found and fixed in it this week without a brief re-explanation of what the software actually does. Also, I haven't been following along closely, which means I have no idea what you're referring to when you talk about your software by name without explaining a little bit about what it does first. Note that it's also useless to start a blog entry with "the next thing I'm going to cover is..."; I don't remember the first thing you did, and since you didn't provide a link to the old thing, I'm already lost. Be kind: think about your audience a bit.

The Magic Sentinel

2008-12-04T04:59:27Z

In an effort to get back on the blogging saddle, here’s a little note on default values in Python.

In Python there are often default values. The most typical default value is None — None is a object of vague meaning that almost screams "I’m a default". But sometimes None is a valid value, and sometimes you want to detect the case of "no value given" and None can hardly be called no value.

Here’s an example:

def getuser(username, default=None):
    if not user_exists(username):
        return default
    ...

In this case there is always a default, and so anytime you call getuser() you have to check for a None result. But maybe you have code where you’d really just like to get an exception if the user isn’t found. To get this you can use a sentinel. A sentinel is an object that has no particular meaning except to signal the end (like a NULL byte in a C string), or a special condition (like no default user).

Sometimes people do it like this:

_no_default = ()
def getuser(username, default=_no_default):
    if not user_exists(username):
        if default is _no_default:
            raise LookupError("No user with the username %r" % username)
        return default
    ...

This works because that zero-item tuple () is a unique object, and since we are using the comparison default is _no_default only that exact object will trigger that LookupError.

Once you understand the pattern, this is easy enough to read. But when you use help() or other automatic generation it is a little confusing, because the default value just appears as (). You could also use object() or [] or anything else, but the automatically generated documentation still won’t look that nice. So for a bit more polish I suggest:

class _NoDefault(object):
    def __repr__(self):
        return '(no default)'
NoDefault = _NoDefault()
del _NoDefault

def getuser(username, default=NoDefault):
    ...

You might then think "hey, why isn’t there one NoDefault that everyone can share?" If you do share that sentinel you run the risk of accidentally passing in that value even though you didn’t intend to. The value "NoDefault" will become overloaded with meaning, just as None is. By having a more private sentinel object you avoid that. A single nice sentinal factory (like _NoDefault in this example) would be nice, though. Though really PEP 3102 will probably make sentinals like this unnecessary for Python 3.0.

Note that you can also implement arguments with no default via *args and **kwargs, e.g.:

def getuser(username, *args):
    if not user_exists(username):
        if not args:
            raise LookupError(...)
        else:
            return args[0]

But to do this right you should test that len(args)<=1, raise appropriate errors, maybe consider keyword arguments, and so one. It’s a pain in the butt, and when you’re finished the signature displayed by help() will be wrong anyway.

WSGI/Pylons Talk in San Francisco tonight (Dec 3rd)

2008-12-03T18:03:09Z

I’ll be giving a talk tonight about WSGI, making apps using it, and Pylons tonight in San Francisco. If you live in the Bay Area and have been wanting to learn more about some of the packages utilizing WSGI, as well as Pylons, RSVP soon as they’d like to get some numbers for food.

Here’s the meetup event page for the talk. This is one of the longer talks I’ve given, so I should actually have enough time to cover the topics (WSGI, WSGI Middleware, making low-level WSGI apps with WebOb, making small WSGI stacks, and Pylons) in good detail, as well as some demonstrations.

Wherefore CouchDB for ZODB Users

2008-12-01T03:06:45Z

After reading Eric Florenzano's pair of posts Why CouchDB Sucks and Why CouchDB Rocks and being generally sick of not understanding why folks were excited about this system, I put my spelunkin boots on to find out more about CouchDB .

Before I set about trying to understand it, I didn't understand why folks were excited about CouchDB given that a good number of its features (append-only storage and "schemaless design" in particular) have been present in ZODB for a little under ten years now. Even more in particular, I was really baffled as to why Python developers were excited about such a system given the availability of ZODB.

I think I understand a bit better now. ZODB and CouchDB are quite similar in a lot of respects, but CouchDB beats ZODB on a narrow set of goals that seem to be becoming more important these days.

First of all, availability is everything. Since the documented way to talk to CouchDB is over HTTP, and because you send it JSON primitives containing data structures, it can be used from just about any program written in just about any language. And given that most folks are not accustomed to being able to access any database without dark-magic C bindings that speak strange connection-oriented TCP protocols (or the equivalent embedded usually-crashy C bindings and awkward APIs), this is probably a novelty for many users. For Zope users, however, it's not really that nifty, as we've been writing applications that use HTTP to modify ZODB structures for quite a long time now.

Second of all, Damien Katz, CouchDB's author used to work on Lotus Notes. Though I've never developed under (or even used) Lotus Notes, I do know that it is widely respected for its replication facilities. CouchDB has offline replay replication that I imagine smells a lot like Notes' built-in facilities for the same. This is a big deal if you're creating offline applications that need to synchronize to one or more other databases. ZODB itself has no such facility.

Third of all (and probably most importantly), CouchDB has a built-in indexing and querying facility, in the form of Views. This is something that ZODB does not share. Instead, ZODB relies on applications that are built on top of it to provide indexing capabilities. Moreover, creating indexes in applications that use ZODB is historically a static kind of thing that the application developer does "up front", or at least as a "software release" sort of thing. In CouchDB, creating an index is not really an exceptional sort of event. You tell the server, over HTTP, to create the index by PUT-ing a view. The first time any view is queried, CouchDB does the indexing. The following times the view is queried, it uses the index you've created via the view to return the results faster. No application that I know of written on top of ZODB allows you to do such a thing so casually.

There is a lot of Interweb talk about the stuff that Erlang provides to CouchDB "for free", usually discussed in terms of stability and "crash-only" design. I don't know anything about this topic, so I can't comment. But CouchDB views are implemented in terms of "map/reduce", which seems to imply that it will be possible to create a farm of servers, each which can operate on only a slice of the entire data structure in order to return a result. However, I'm not sure that this feature is actually implemented in any release of CouchDB (I think it's just implied by its design).

All in all, CouchDB is a neat piece of software. If I ever have to build an application that needs to store data that needs to be accessible from programs written in langauges other than Python across HTTP and I don't need to use a relational database, it seems like a great solution.

ZODB is probably still a better choice if your application is 100% Python and you want arbitrary application database write logic to be bounded within a single transaction. CouchDB's transaction semantics default to one-request-one-transaction, although they do have some batching facilities. Likewise, if you need to store large amounts of data (like arbitrary numbers of multimegabyte or multigigabyte files), ZODB is also probably a better choice, as it has blobs built in, and the blobs don't need to be base64 encoded in memory as "attachments" in order to transmit over a wire protocol. Likewise, if you need to store data structures that cannot be represented as JSON (like complex object instances), ZODB really can't be beat.

REVISION: Jan Lehnardt from the Apache projects sent this via email wrt to blobs

Thanks for the excellent overview. You certainly understand CouchDB and this helps a lot removing FUD. Good work, thanks for taking the time.

There's one addition I'd like to make: CouchDB no longer requires encoding binary data in base64. Attachments for documents (that work just like email attachments) have their own REST API since this summer where you can send the raw binary data to create and retrieve documents. So storing loads of data is no longer a bad idea.

It's sort of obvious that most of what makes up CouchDB (besides the currently unquantifiable-by-me benefits of it being written in Erlang, anyway) could really be done in a reasonably straightforward ZODB web application. I actually started such an animal (just to learn, not to use, I doubt I will play with it much more) called loveseat . The hardest thing to get right would be the indexing and the replication, of course. It already does database creation and document creation and retrieval, but not views or replication. That would probably take several months to get right, and I only had today. ;-)

I'll also note that ZODB has pretty terrible marketing compared to CouchDB. So I suppose I should try to do some. For you Python developers that don't know about ZODB, and whom are excited about CouchDB, you might check out ZODB. You can think of a ZODB database as a place to hang a graph of arbitrary Python objects that becomes persistent. It's sort of an "uber-pickle"; it actually makes heavy use of the pickle module under the hood, but it breaks the object graph up into separate pickles, and so can be used for high volume applications. Changes take place transactionally. Multiple clients can make use of the same ZODB database over a protocol called "ZEO" (Zope Enterprise Objects), which isn't nearly as scary as it sounds; basically, you just set up a ZEO server and point the clients at it. You can use packages like repoze.catalog to do indexing and querying of object data that is inserted into the graph. You can use packages like repoze.folder to hold large collections of objects. It's fast, and other than a few C extensions, completely written in Python. It's been around, like I said for about ten years, and it's in production in tens (hundreds?) of thousands of Zope deployments today. The "Zope" in ZODB is a "brand-only" name; it does not require Zope; it can be used in any Python application. It has limited deployment outside a Zope context, but I think this is mostly cultural: I personally use it in non-Zope applications frequently. It works on all major platforms. A good place to start would be to do:

  easy_install ZODB3

Then maybe take a gander at this tutorial . And for a more verbose look, try this howto .

Repoze Status Report

2008-11-23T20:50:43Z

I suppose it's about time to give a status report about Repoze.

Much of the action since the last report in March has revolved around the repoze.bfg web framework. It's been about four months since we made the first release of that package. repoze.bfg is a web framework that could be most effectively compared to Pylons inasmuch as it makes very few decisions for you, and it's based on WSGI technologies and many Ian Bicking creations such as Paste and WebOb. However, it also relies heavily on Zope technologies and concepts (such as object publishing), and as a result, should be familar to Zope developers. It's a bit different than Grok or Zope 3, because it doesn't attempt to expose all the features of Zope 3 (particularly it doesn't expose much of the Zope Component Architecture); instead it uses those technoloigies and exposes a more minimal API.

In the time since we released 0.1, we've made 26 minor releases of the repoze.bfg package (the most current version is 0.5.0). During that time, we've maintained backwards compatibility with the original release. The docs have been kept pretty much in sync with the codebase. Also over that time, we've added the following features:

URL-based dispatch via Routes
Allow views to be customized by a request type (meaning that you can dispatch to a view based on attributes of the incoming request, such as whether it's a GET or a POST, or which hostname was used to request the page).
Created demo applications such as repoze.shootout (courtesty of Carlos de la Guardia) and repoze.cluegun which can be used as "jumping-off-points" to develop your own bfg applications.
Added a repoze.bfg.convention module to allow people to configure bfg views without using ZCML.
Integrated Chameleon as the default templating implementation. This templating implementation allows for text, ZPT-style, and Genshi-style templates. Support for the more obscure nooks and crannies of each of these languages has been improved by Malthe Borch, and it's still much faster than most templating languages.
Added the capability to render views programmatically via the repoze.bfg.view module.
Added documentation about the repoze.bfg startup process
Added documentation about the environment and configuration of a repoze.bfg application.
Added a helper module that makes unit testing easier
Many bugfixes.

My company (Agendaless Consulting) is developing two customer projects on top of BFG now. I also gave a presentation about repoze.bfg at the most recent Plone conference. The slides are available.

Of course, lots of stuff has been happening outside BFG too.

The Repoze project has been able to give about 25 new developers SVN commit access over the same amount of time, which seems to bode well for the technologies that share the Repoze brand. Many folks have been contributing a lot of stuff. If I were to try to account for all the stuff added to the repoze SVN repository in this message, it would be essentially unreadable due to length. Some highlights:

repoze.what is a WSGI authorization system managed by Gustavo Narea, which builds on top of the repoze.who . This is a package which aims to provide generic WSGI apps with authorization, and will be used by TurboGears 2.
repoze.catalog is an indexing and searching system which provides Zope-catalog-like features.
repoze.lemonade is a package that brings Zope-CMF-like content registrations to WSGI apps.
repoze.workflow is a library that provides an implementation of a state machine usable for workflow-like tasks.
repoze.monty is a form field marshalling library which contains code stolen from Zope which converts form elements to various data types when they are posted.
repoze.bitblt is image resizing WSGI middleware written by Malthe Borch and Stefan Eletzhofer. repoze.squeeze is automagical JavaScript and CSS merging in WSGI middleware, also by Malthe and Stefan.
repoze.urispace is a WSGI application that implements the Akamai urispace specification.
repoze.accelerator is an implementation of an HTTP caching reverse proxy in WSGI middleware.

We hope to be able to continue normalizing both Zope and non-Zope technologies for use in a WSGI toolchain. Things are looking pretty good so far. If you're interested in participating, or just asking questions, don't hesitate to visit the #repoze IRC channel or the mailing list .

Touch Engine: An iPhone to AppEngine Bridge

2008-11-22T01:44:55Z

Jonathan Saggau and I just released a nice iPhone to Google App Engine bridge project.

http://code.google.com/p/touchengine/

He has a bunch more than I do about it on his blog here.

Religion: An API To The Brain?

2008-11-19T05:54:04Z

I have been thinking a lot about publicly exposed API's to the brain, in the context of AI programming. It seems like one of the more interesting APIs to the brain is religion. At PyWorks, there was a talk on AI, in which, hunter/prey behavior was simulated. I wonder if it would be possible to simulate religious behavior in much the same way? The specific behavior would be a pastor/flock behavior.

It seems like if we could simulate religious behavior, then it might allow us to gain insights into how to create artificial life that follows that same API. How could that look in Python?


from brain.religion import PublicApi
from brain import emotion, rationality, memory

p = PublicApi(denomination="MyChurch", scale=10, **kw)

One of the more fascinating parts would be to study the actual brain scans of deeply religious and non religious people while they talk about politics, and other hot issues. This research could help to identify what regions of the brain are firing, and why, and thus help software engineers create a similar API with a future artificial life form.

Using repoze in a Plone buildout

2008-11-14T15:42:11Z

Props to Tom Gross, who has tackled the job of figuring out and documenting the minimal set of changes needed to make use of repoze and WSGI from within a stock Plone buildout.

Correction: Tom Gross sent me the link, but it was to a blog post by Tim Terlegård. Apologies for the mistake!

Book Meme

2008-11-13T19:03:59Z

Via :

Grab the nearest book.
Open it to page 56.
Find the fifth sentence.
Post the text of the sentence in your journal along with these instructions.
Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

From The Omnivore’s Deilemma by Michael Pollan:

And then of course there’s the corn itself, which if corn could form an opinion would surely marvel at the absurdity of it all—and at its great good fortune.

I don’t usually do the meme thing, but I happened to have a book on my desk so it took so little effort… only one chapter to go!

Using repoze in a Plone buildout

2008-11-13T16:20:16Z

Props to Tom Gross, who has tackled the job of figuring out and documenting the minimal set of changes needed to make use of repoze and WSGI from within a stock Plone buildout.

PyWorks Tutorial on Google App Engine

2008-11-13T14:37:56Z

Here is the website for the 3 Hour Tutorial I taught at PyWorks on Google App Engine, GData API, and Google's AJAX API's. There is some good stuff in there, especially the tutorial on how to programatically control Blogger. If you write a command line tool to write to blogger, let me know!

http://row.appspot.com/

Most bizarre Git service and other stupid Rails powered "businesses"

2008-11-12T19:18:42Z

I can’t help but get totally baffled when I see a business model like this.

Yes, that’s right, you can pay for the privilege of keeping a copy of your distributed version control system (DVCS) private repositories on someone else’s machines. You also get to pay depending on how many people you want to allow to collaborate on it.

Nevermind that one of the entire points of a DVCS is that you do NOT need a central repository. Does anyone actually work at a “Large Company” (as the page indicates) that would be stupid enough to pay $100/month so they can put all their proprietary and very personal code repositories on a third party web service?

So what are you paying for? Well, to start with, they have awesome integration with Lighthouse, since we all know there’s no decent free open-source issue tracking system… cough trac cough roundup cough. Oh wait, since there’s absolutely no simple web-based issue tracking systems, let’s have another slick business model to get people to pay for a stripped down Trac (but this time with a really pretty UI)!

What do these sites have in common? Rails, “look ma, I can copy-paste the business plan too” pricing models, and some good graphic designers at the helm. There also seems to be an interesting amount of promotion between these sites, as well as a nice blog post from the Rails creator himself promoting GitHub. I’m sure no one who has read this rant should be surprised though.

I only hope that no one starts to believe that a DVCS actually requires these “please pay” copies of their DVCS repo.

Update (11/12/2008): This post is apparently popular enough to come up on occasion several times now, so I thought I’d clarify a bit more.

Many people have suggested the obvious benefits of services like GitHUB, and I’ve used one just like it myself, BitBucket. These sites are great for open-source projects as many have rightfully pointed out, they make it easy to collaborate and fork projects, and easy for maintainers to pull patches from forks after looking them over.

Most of their social-network features become moot though when working on company code thats not open-source, (note that this rant is directed entirely at the paid service options which are for private repos). None of the companies I’ve worked at would ever let their private source code leave their own servers. Since you need to deploy a site anyways (many times to a remote computer), which will generally require ssh access, its trivial to use the modern DVCS’s over ssh…. which makes it seem very silly to me to be paying so much to another company for a bunch of useless social features for a private repo.

Part of the original humor intended in this rant was that a centralized repo hub has become one of the stronger selling points for a distributed VCS system. Unfortunately many seemed to have missed that point.