S3 inconsistencies

s3_inconsistencies.md

Amazon S3 inconsistencies

Problem

• Let us start by listing out the buckets using the commandline awscli tool.

  

• Under lspl-prism-autorep, let us check out the available "folders"

  

• Under MIS/, what files do we have?

  

  Ok, we seem to have one file under this folder.

  Note: This file was uploaded directly to the target MIS/20151230153357HO20151229.zip under the lspl-prism-autorep bucket without explicitly creating the MIS folder.

• Let us now explicitly create a folder via the UI within the other lspl-testing bucket

  

• Now lets list the folders under lspl-testing

  

  As expected, OK.

• Lets now upload a file to that folder using awscli

  

• Following this, when we list the files under the MIS/ folder, the expectation is to see just the links.txt as we did in the previous case, buuuut:

  

  Creating a folder via the UI, creates a MIS/ file with a size of 0 bytes.

Understanding

Amazon S3 has no concept of folders. Everything is kept as keys and a key containing by a / is treated as being nested under a folder. So a key like foo/bar/baz.txt means that there exists a file baz.txt under a folder bar which is under a folder foo. Although in concept, there is just one file with the key foo/bar/baz.txt.

The API doesn't expose any way to create a folder. It assumes any file that ends in / is a folder. Which is why we see that when a folder is created via the UI, it creates a 0 byte file for us.

This actually brings us to another problem. When a file is thrown into S3 without an explicit folder having been created for it, and the said file is deleted, then the folder also gets deleted (obviously). But if an explicit folder has been created and the file under it is deleted, it doesn't delete that folder (obviously).

Another problem that was noticed was when using a 3rd party tool called S3 Browser. Creating a folder in S3 Browser actually creates a 1 byte file for some reason. Weird.

Expectation

Documentation. I've not been able to find this documented anywhere at all. It needs to be written down somewhere that folder creation via UI will create a 0 byte file that will be returned in the API response when listing files matching a prefix. This will help developers think about this in advance and work their way around it.