working_with_gems.md

Gem(spec|file(.lock)?) and You

A Howto on development of Gems and their interaction with Bundler, Continious Integration and Deployment in complex applications.

Introduction

Gems depend on other Gems by specifying a name and version range. This is defined in the gem's .gemspec. They intentionally don't care about where exactly the dependencies come from. When the gem is ultimately installed via the Gem command, the sources configuration determines where it is looking for the gems.

The .gemspec is a standard format for describing all of the information that gets packed with gems then deployed to rubygems.org. It is focused on the needs of the Gem author and the Rubygems ecosystem.

Ruby applications on the other side have a complex set of requirements that are generally tested against a precise set of dependencies. The developer usually doesn't care about the exact version used in development. Though she cares that the exact same version is also used for testing and production. This is where Bundler comes into the game.

The scope of Bundler and Gemfile and Gemfile.lock focuses on making it easy to guarantee that all of the same code (including third-party code) is used across machines. It does, however, contain a list of dependencies, including information about where to find them.

See Clarifying the Roles of the .gemspec and Gemfile for a more in-depth discussion about this topic.

Pain Points during Development

1. Unreleased dependencies that are not yet found in rubygems
2. Hyprid environment with gems coming from the file-system, git and traditional sources
3. Local Development without Git(hub) round trips

Gems will eventually become regular Gems and need to declare metadata in their .gemspec. Additionally, the hybrid environment is being created using Bundler and a Gemfile. Geez, complicated. Let's disect this...

Avoid Duplication

In order to make it easy to use the bundler tool for gem development without duplicating the list of dependencies in both the .gemspec and the Gemfile, bundler has a single directive that you can use in your Gemfile:

source 'http://moo-repo.wdf.sap.corp:8080/rubygemsorg/'
source 'http://moo-repo.wdf.sap.corp:8080/geminabox/'
 
gemspec

Ideally, this is all that needs to be in the Gemfile. But nothing is ideal and there is always edge cases.

Unreleased gems

If you find yourself in the situation that you need to develop against an unreleased gem, you can add the actual location to the Gemfile:

source 'http://moo-repo.wdf.sap.corp:8080/rubygemsorg/'
source 'http://moo-repo.wdf.sap.corp:8080/geminabox/'
 
gemspec

gem "hypothetical", git: "git://github.wdf.sap.corp/d038720/hypothetical.git"

The information from where this dependency is coming from shouldn't be included in the .gemspec since the Gem will not be usable when the transient path is not available anymore. In fact, it's not possible to do so. Though it should contain the reference:

...
gem.add_dependency('hypothetical')
...

This becomes even more apparent when using references to the local file system.

Avoiding Round Trips

It is cumbersome to always bounce the changes through Git and bundle install. A more convenient way is to specify a local path to the dependency:

source 'http://moo-repo.wdf.sap.corp:8080/rubygemsorg/'
source 'http://moo-repo.wdf.sap.corp:8080/geminabox/'
 
gemspec

gem "hypothetical", path: "~/Code/d038720/hypothetical"

No more bundle install, no need to check every change into Git.

Working in a Team

This house of cards only stands until the code needs to work on another machine, be it another developer or a CI agent. Let's fix it:

source 'http://moo-repo.wdf.sap.corp:8080/rubygemsorg/'
source 'http://moo-repo.wdf.sap.corp:8080/geminabox/'
 
gemspec

%w[hypothetical].each do |name|
  library_path = File.expand_path("../../#{name}", __FILE__)

  if File.exist?(library_path)
    gem name, :path => library_path
  else
    gem name, :git => "git://github.wdf.sap.corp/d038720/#{name}.git"
  end
end

This will check if the dependency is available on the local file-system and fallback to the git location otherwise. It assumes though the libraries are located in the same directory:

$ ls ~/Code/d038720
hypothetical
  Gemfile
  hypothetical.gemspec
hypothetical-theoretic
  Gemfile
  hypothetical-theoretic.gemspec

With this simple convention, we get the best of both worlds.

Gemfile.lock

It isn't apparent what should happen with the Gemfile.lock that is being emitted by Bundler.

• For gems it should not be checked in.
• For applications it should be checked in.

The reason is, that a gem installed with gem install will not honor the Gemfile.lock. It will just pull whatever gems it finds from rubygems.

So, during gem development we want to know immediately if some change in the overal ecosystem breaks the setup. We simulate this by bundle install the gems on the CI agents, which without a Gemfile.lock will pull the latest available versions. Just like a gem install would.

Further considerations for Continious Integration

Speeding up the Test

To speed up continuous integration it is often desirable to only install dependencies that are required for testing and omit everything else. In the .gemspec it's only possible to define development and runtime dependencies though.

To setup the test environment we want to:

bundle install --without development

This will not install the libraries for testing either though. We can work around it by specifying these dependencies twice. Not ideal, but usually also no problem.

source 'http://moo-repo.wdf.sap.corp:8080/rubygemsorg/'
source 'http://moo-repo.wdf.sap.corp:8080/geminabox/'
 
gemspec

%w[hypothetical].each do |name|
  library_path = File.expand_path("../../#{name}", __FILE__)

  if File.exist?(library_path)
    gem name, :path => library_path
  else
    gem name, :git => "git://github.wdf.sap.corp/d038720/#{name}.git"
  end
end

group :test do
  gem "rspec"
end

and the hypothetical.gemspec

...
gem.add_development_dependency('rspec')
gem.add_development_dependency('guard')
gem.add_development_dependency('guard-rspec')

gem.add_dependency('celluloid')
gem.add_dependency('httparty')
...

Take note, that the Guard gems will not be installed on the CI agents now.

Platform Specific Dependencies

We are developing mostly on Mac OS and running CI and production systems on Linux. Sometimes there is platform specific dependencies that fail when installed.

Workaound for .gemspec:

s.add_dependency 'rb-fsevent', '~> 0.9.1' if RbConfig::CONFIG['target_os'] =~ /darwin/i
s.add_dependency 'rb-inotify', '~> 0.9.0' if RbConfig::CONFIG['target_os'] =~ /linux/i
s.add_dependency 'wdm',        '~> 0.0.4' if RbConfig::CONFIG['target_os'] =~ /mswin|mingw/i

Workaournd for Gemfile

case RbConfig::CONFIG['target_os']
when /darwin/i
  gem rb-fsevent',  '~> 0.9.1', require: false
when /linux/i
  gem 'rb-inotify', '~> 0.9.0', require: false
when /mswin|mingw/i
  gem 'wdm',  '>= 0.0.4', require: false
end

The libraries mentioned here, do actually install silently on all platforms. So, this is just an example for demonstration purposes.

This is a semi-ugly workaround. More and more gems are handling this issue themselves. Some outdated gems (cough, Autotest) still need special treatment. Often there is working alternatives (cough, Guard).