S3 File Resource for Chef

recipe.rb

# Source accepts the protocol s3:// with the host as the bucket
# access_key_id and secret_access_key are just that
s3_file "/var/bulk/the_file.tar.gz" do
  source "s3://your.bucket/the_file.tar.gz"
  access_key_id your_key
  secret_access_key your_secret
  owner "root"
  group "root"
  mode 0644
end

s3_file.rb

#
# Author:: Christopher Peplin (<[email protected]>)
# Copyright:: Copyright (c) 2010 Bueda, Inc.
# License:: Apache License, Version 2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

class Chef
  class Provider
    class S3File < Chef::Provider::RemoteFile
      def action_create
        Chef::Log.debug("Checking #{@new_resource} for changes")

        if current_resource_matches_target_checksum?
          Chef::Log.debug("File #{@new_resource} checksum matches target checksum (#{@new_resource.checksum}), not updating")
        else
          Chef::Log.debug("File #{@current_resource} checksum didn't match target checksum (#{@new_resource.checksum}), updating")
          fetch_from_s3(@new_resource.source) do |raw_file|
            if matches_current_checksum?(raw_file)
              Chef::Log.debug "#{@new_resource}: Target and Source checksums are the same, taking no action"
            else
              backup_new_resource
              Chef::Log.debug "copying remote file from origin #{raw_file.path} to destination #{@new_resource.path}"
              FileUtils.cp raw_file.path, @new_resource.path
              @new_resource.updated = true
            end
          end
        end
        enforce_ownership_and_permissions

        @new_resource.updated
      end

      def fetch_from_s3(source)
        begin
          protocol, bucket, name = URI.split(source).compact
          name = name[1..-1]
          AWS::S3::Base.establish_connection!(
              :access_key_id     => @new_resource.access_key_id,
              :secret_access_key => @new_resource.secret_access_key
          )
          obj = AWS::S3::S3Object.find name, bucket
          Chef::Log.debug("Downloading #{name} from S3 bucket #{bucket}")
          file = Tempfile.new("chef-s3-file")
          file.write obj.value
          Chef::Log.debug("File #{name} is #{file.size} bytes on disk")
          begin
            yield file
          ensure
            file.close
          end
        rescue URI::InvalidURIError
          Chef::Log.warn("Expected an S3 URL but found #{source}")
          nil
        end
      end
    end
  end
end

class Chef
  class Resource
    class S3File < Chef::Resource::RemoteFile
      def initialize(name, run_context=nil)
        super
        @resource_name = :s3_file
      end

      def provider
        Chef::Provider::S3File
      end

      def access_key_id(args=nil)
        set_or_return(
          :access_key_id,
          args,
          :kind_of => String
        )
      end
        
      def secret_access_key(args=nil)
        set_or_return(
          :secret_access_key,
          args,
          :kind_of => String
        )
      end
    end 
  end
end

I've ran into an issue with the library, mostly with URI.parse - if the bucket name does not contain a dot in the FQDN portion of the URI, then URI.parse throws a URI::InvalidURIError exception, even if having an FQDN without a dot in S3 is completely valid.

I've used the following bits of code for uri, name, and bucket:

uri = URI.split(source).compact
name = uri[2][1..-1]
bucket = uri[1]

Thanks for the report - I'm happy to see someone else finds this useful. I'm not able to reproduce the bug you describe - is this an example of what returns a URI::InvalidURIError for you?

>> URI.parse("s3://bueda/test.zip").path[1..-1]
=> "test.zip"

Hey there,

Try the following:

>> URI.parse("s3://bueda_foobar/test.zip")
URI::InvalidURIError: the scheme s3 does not accept registry part: bueda_foobar (or bad hostname?)
    from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/generic.rb:195:in `initialize'
    from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/common.rb:492:in `new'
    from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/common.rb:492:in `parse'
    from (irb):4

Seems it's underscores that are causing issues...

Also, wanted to say that it was indeed very helpful! Thanks so much for coding this up, Christopher!

Thanks - I guess using URI is a bit of a stretch since bucket names don't have to be valid FQDNs. I've updated this to use the URI.split() method as you do.

How do you actually use this? Sorry, I'm new to Chef and trying to understand this as I build up my stack. Would these files go into a cookbook of their own? Could you give a quick example of how you're using them?

Sure, put s3_file.rb in the libraries/ folder of any cookbook (create it if it doesn't exist) and it should be automatically imported.

Alternatively, make a standalone s3 cookbook with the file in s3/libraries/ and in other cookbooks, just call include_recipe "s3" before using it.

hi I added support for all regions and removed the need for the aws-s3 gem

https://gist.github.com/1264036

Hi,

I've added to the excellent work of peplin and casualjim (thanks), to add checking for XML error responses from Amazon and to add dual support for S3 and other protocols provided by the out-of-the-box RemoteFile (e.g. http(s)).

https://gist.github.com/1351043

Thanks for the work, folks!

I'm getting an NameError: uninitialized constant Chef::Provider::S3File::AWS.
What I'm doing wrong?

@TiagoBrito @poobury @casualjim @peplin I've created an update to @peplin's original gist using AWS-SDK at https://gist.github.com/DavidAllison/5288249

It includes minor changes that make it work with the newer AWS-SDK gem (the current version as of this writing). Peplin's older gem won't work with the newer AWS-SDK gem since the API is no longer the same.

@poobury and @casualjim, I really like where you guys went with this, but unfortunately I wasn't able to easily make either or your recipes work with AWS's temporary S3 credential granting based on IAM policies (whereas the AWS-SDK gem from Amazon handles this quite nicely). So I ended up modernizing @peplin's original work.

These all have the same problem - to be generally useful, the temporary file needs to be written in binary mode! e.g. for the above:

file = Tempfile.new("chef-s3-file")
file.binmode
file.write obj.value

Getting a "no resource or method named s3_file for Chef::recipe "default" when we try to use this. ANy ideas?

Have you tried to add include_recipe "s3_file" into your cookbook recipe?

i get "no resource or method named s3_file for Chef::recipe "default" no matter what i try

Hi,

I want how to write (or) the S3 file.

Thanks
Karthi

Can this resource copy a directory of files to a local directory as well as just a single file?

Hello. I am getting the following... any help appreciated:

19: class Chef
20: class Provider
21>> class S3File < Chef::Provider::RemoteFile
22: def action_create
23: Chef::Log.debug("Checking #{@new_resource} for changes")
24:
25: if current_resource_matches_target_checksum?
26: Chef::Log.debug("File #{@new_resource} checksum matches target checksum (#{@new_resource.checksum}), not updating")
27: else
28: Chef::Log.debug("File #{@current_resource} checksum didn't match target checksum (#{@new_resource.checksum}), updating")
29: fetch_from_s3(@new_resource.source) do |raw_file|
30: if matches_current_checksum?(raw_file)

[2016-08-25T09:50:43-04:00] ERROR: Running exception handlers
[2016-08-25T09:50:43-04:00] ERROR: Exception handlers complete
[2016-08-25T09:50:43-04:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated
[2016-08-25T09:50:43-04:00] ERROR: uninitialized constant #Class:0x00000003144990::Chef::Provider::RemoteFile
[2016-08-25T09:50:43-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Hi,
this doesn't work for me when I don't pass access keys, actually the instance already has an IAM role assigned and has enough permission on bucket.