Request for Comments: Fixing Ruby, Python and Node.js packaging on Debian, Redhat and variants

unfixable.md

Request for Comments

Fixing Ruby, Python and Node.js packaging on Debian, Redhat and variants.

Canonical URL: http://bit.ly/wtfpackages

It's unfixable.

You shouldn't be using distribution provided packages for your primary language runtime.

For you, it's your application platform, and you're the one who knows exactly which version best suits your needs. For others, it's the interpreter for another distribution-provided tool that they quite like using, such as chef or puppet. New (and stable) functionality that you wish to use happens to be saddled with changed APIs, fixed bugs and deprecated methods, all which might break those distribution-provided tools.

There is no happy middle-ground between your use case and theirs; every compromise is a poor one.

So, let's talk about an alternative:

A centralised resource

Community packages are scattered around the internet and are of varying levels of quality. One location, n versions, n languages, n distributions. Every package provided will follow the same rules (outlined below) and thus be somewhat predictable.

Use distribution tools

Community packages rarely have distribution-level support for pulling in updates. Integrate with apt & yum. Packages should be versioned/managed such that automatic system updates pull in security / bug fixes, but not entirely new versions.

Don't conflict with system packages

Community packages often replace the distribution-provided interpreter. This causes problems with dependencies and system toolsets. Use RVM (or rbenv) and virtualenv to ensure that system Ruby & Python don't conflict with the community-provided packages.

Don't usurp the language's packager

Pre-packaging every Ruby, Python and node.js library (as distributions try to) isn't useful when you're already attaching those libraries to your application with language-specific methods.

Or, I'd rather use rubygems and bundler than dpkg to manage my application's dependencies.

Test driven

Community packages age, or only work with specific distribution versions, or only work when a certain library (not listed in the dependencies) is installed. Each package's post-install script will confirm the basic behaviour of the package. Virtual machines will be used to automatically and continually install & test packages.

Community-driven

Use the successful homebrew + github model to manage community package contributions.

I think there needs to be a lot less of the US vs THEM attitude. The distribution developers are trying to make their packages usable for the majority. This often means they need to prioritise a new release of ruby with dropping several pieces of software. What they really need is more help to work with the Ruby community. Anyone reading this gist clearly cares about how ruby works, so why not go help a distro?

@smarthall It's not about distributions versus developers - it's about acknowledging that the requirements of the many (the distribution users) are never going to meet the needs of the few (language users).

Previous attempts at contributing to the distro - trying to bring the needs of the primary language users, that is - have resulted mainly in those US vs THEM attitudes you mention as people realise that nobody is going to get what they want.

I'm not entirely sure why you include python in the list "to be fixed" .. Packaging a "normal" python library is about 60 seconds -> 5 mins work. No joke - I do it all the time.

Ruby libraries on the other hand are next to impossible to package without all sorts of hacks and patches. Anything that includes the lines:

require 'rubygems'

or

require 'bundler/setup'

is instantly impossible to package without patches.

(I've ignored node.js because I've never looked at it.. So anything I said there would be pure speculation)

Re this:

Or, I'd rather use rubygems and bundler than dpkg to manage my application's dependencies.

Wearing my developer hat, I agree. Development is easier with those tools. I even use them during development.

Wearing my sysadmin hat, I won't even consider installing something (other than a web site) on a production server that uses these tools. Never. Not happening. I'll find another tool.

I do get both sides of this argument. I've used both Python and Ruby, and I believe Python has actually gotten it right. While Ruby (More specifically, RubyGems) has gotten it completely wrong.

Finally - This is an US vs THEM argument. But it's not developers vs distributors. It's web developers vs "non web developers".

Web developers don't care, and don't need to care, about things like distribution. RubyGems and Bundler serve this market brilliantly.

Non-Web developers on the other hand do typically care about these things, their applications are going to be installed on more than just 1 server (or a bunch of identical servers). OS packaging is vitally important for this kind of application.

@kiall Yes, this is web developer centric .. or at least centric to organisations that control the environment they're deploying their applications to.

I can't comment properly on Python, either, it's been about five years since I've worked with Python full time.

I'd invite you to reconsider your stance re bundler & rubygems .. but I think this will only happen if you develop a large web application that requires modern libraries. It is just so much easier and flexible than having to rely on distribution libraries or having to package those libraries yourself.