mathematism

Feb 15

Lessons Learned from django-cumulus

While building this reusable application for Django I gained some valuable experience. Below is a history of the short life of django-cumulus and insight into what I learned while preparing it for release.

History

In April of 2009 I needed a custom file storage backend for Rackspace’s Cloud Files within Django. I used the Googlewebs and couldn’t find anything so I decided to build my own, which Django makes pretty darn easy. Within a few hours I had a working application and released version 0.1 of the code.

Over the next week I received some messages about possibly integrating my custom storage into django-storages, an application that pulls together multiple custom storage backends including Amazon S3 and MogileFS among others. I thought about it and it made sense at the time to add my custom storage. In May of 2009 I added the Cloud Files storage backend to django-storages.

Over the next few months a few forks of django-storages popped up and I pulled in some solid tweaks to the applicaiton. I had a pretty busy second half of 2009 at work and admittedly neglected maintenance of my part of django-storages. When 2010 rolled around I decided that one of my resolutions would be to create, contribute to, and maintain several open-source applications. Among them was django-storages.

With my newfound focus I started to dive into the open tickets on django-storages. There were (and remain) quite a few open tickets that posed pretty big issues for users (including a broken setup.py). Over the next week or so I began to start tackling some of these tickets and it quickly became aware that things were at a standstill.

There seemed to be no interaction between the application and its users. I took on and updated several tickets, aiming to prepare django-storages for a 1.1 release. Not one of the ticket creators responded to me. At that point I made a few observations about django-storages:

  1. The interaction was weak between the community and the application authors
  2. There were, in my opinion, too many things going on with django-storages; it seemed to violate Django’s reusable app mantra of “do one thing and do it well”
  3. The application was missing a proper setup.py file, documentation and tests

At this point I decided to focus my efforts on pulling my custom storage out of django-storages and back into my own application, django-cumulus. I took some time in late January and early February to get up to speed on the latest Python packaging practices, documentation and testing. In early February I released django-cumulus 0.2, complete with documentation, tests, and a proper setup.py.

Packaging

I’ll admit that before django-cumulus I considered myself borderline idiot when it came to packaging. I didn’t understand why everyone had different setup.py files and felt like there was no good resource for learning how to properly build a setup.py file. Then I took a look at Distribute because that’s what the cool kids told me to do.

Distribute is very well documented and provided me with the first setup.py overview that actually helped me better understand packaging. I used Distribute to build my setup.py for django-cumulus and it worked like a charm. It’s what I’ll be using for any of my Django apps moving forward. I’ll also be updating all of my currently released Django apps to use it as well.

Documentation

I knew of Sphinx and had used it a few times in the past but never really took the time to dig in. This was a good opportunity for me to get all buddy-buddy with properly documenting my work.

Sphinx is insanely easy to learn and use. The documentation is solid. I went through the tutorial once and had a good starting point. From there on I just referred back to their docs for detailed information.

Another tool I’d like to mention is Jannis Leidel’s sphinx-pypi-upload. This allows you to easily upload your package’s documentation to packages.python.org using the following commands:

python setup.py build_sphinx
python setup.py upload_sphinx

The documentation for django-cumulus was created using Sphinx and uploaded using sphinx-pypi-upload to http://packages.python.org/django-cumulus/.

Thumbnailing

One last item I had to look into was the ability to use popular Django thumbnail applications alongside django-cumulus. The two main thumbnail libraries I focused on were sorl thumbnail and django-imagekit.

First I looked at sorl. Let me preface what I’m about to say by this: I’ve used sorl in a TON of projects. It’s great and does a fantastic job for local thumbnail creation. When it comes to using custom file storage sorl is a mess. First of all there are several repos: subversion (official), git, and mercurial. Then there are the forks. It’s like a never-ending spiral of discussions, tickets and code to try and find if there is a solution or someone is working on one. It’s also hard to tell if anything is progressing with sorl because commits to the various versions are as old as 11 months.

After hours of searching, reading, and fiddling with various forks of sorl, I decided to postpone support of sorl because it means I’ll have to dig into modifying sorl itself.

django-imagekit was another story. It’s actively maintained by the author and had already implemented the use of custom storage! So I fired up a test project using django-imagekit and it turned out I had to make some tweaks to django-cumulus to get it to play nice with the temporary files that django-imagekit uses. Those tweaks were incorporated in django-cumulus 0.2.2.

What’s next

Building django-cumulus has been a great experience thus far. I learned how important active maintenance, documentation and packaging are to the success of your application. I’m looking forward to maintaining a high level of quality for my releases moving forward by including documentation, tests, and proper packaging.

I’d love to hear of any features you think would be useful in django-cumulus. Feel free to create a ticket or leave a comment below.