Skip to content

Instantly share code, notes, and snippets.

@theanalyst
Last active September 11, 2017 20:18
Show Gist options
  • Select an option

  • Save theanalyst/024a87738908865a48fe to your computer and use it in GitHub Desktop.

Select an option

Save theanalyst/024a87738908865a48fe to your computer and use it in GitHub Desktop.
RadosGW storage mechanism

Object storage

S3/Swift like systems have the concept of objects & buckets (or Containers). These objects are often striped into multiple RadosGW objects.

How objects are cut:

An object is split into a 'head' part and a logical 'tail' part. The head will never be more than a predefined chunk size (default is 512k). Objects can either have data in the head or not, and smaller objects might not have tail (if they fit into the head, and the head is actually used to hold the data). The head is mapped to a single rados object. It has a deterministic name, and it is mutable. The tail is immutable and may map into more than a single rados object. The tail is made of one or more logical parts, and each part is striped, using a specific stripe size. Each stripe is mapped into one rados object.

Manifests are object descriptors. They specify how an object is built. Prior to firefly, the manifest was just mapping offsets into rados objects ('explicit' object mapping). Currently, the manifest provides rules that specify how the different object parts are built and named, so that it scales much better than the previous method

Given this information

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment