Camlistore on a VPS with S3 blob storage

camlistore-server-vps-s3.md

Let's set up Camlistore on a Linux server, with blobs stored in s3. This seems to be the currently best-supported option for "cloud" deployment.

This is meant as a supplement to the official server config doc. Read through both docs before you start.

http://camlistore.org/docs/server-config

This blog post is also recommended reading.

I've posted my config files for reference, but they will be created the first time you run camlistored (for the server) and camput init (for the client) so don't copy them.

Outline:

• Build on server
• Configure S3 bucket
• Configure client on laptop
• Ensure camlistored starts on server start

Get Camlistored running on your server

• Go 1.1+. For Ubuntu, you may need to upgrade, use a PPA, or use godeb to build your own packages.
• Run the server with bin/camlistored (this will create your initial server config)
• Edit ~/.config/camlistore/server-config.json (there is also a /setup URL for editing the config, but it's only accessible from localhost, so you'd need to use an SSH port forward: ssh -L 3179:localhost:3179 [email protected])

SSL

Follow the Ubuntu Docs to create a self signed certificate, and then put them in your server config.

Set up s3

• Create a bucket
• Create a IAM user
• Copy Creds for config
• Use the grant policies from this article. Might be overly permissive.

Startup script

Make sure camlistored starts when your server starts. On Ubuntu, you need an Upstart job config like camlistored.conf.

Client Setup

Just camput init and edit vim ~/.config/camlistore/client-config.json to add a new default server with auth string copied from server config.

You will need to add the SSL cert from the server to a "trustedCerts" key. Try to camput something and you will see a helpful error message about this.

Next Ideas

• Import your stuff! Use cammount ~/Store and then mkdir ~/Store/roots/Notes to start.
• Run multiple indexes/blobstores (laptop, beaglebone in closet)
• Serve a website right out of camlistore using a publish permanode and go templates

Background

• How to build a remote storage server
• https://groups.google.com/forum/#!searchin/camlistore/camlistore$20delete/camlistore/YsJvVRYIAcc/4ZUsYHdjAnAJ

camlistored.conf

## /etc/init/camlistored.conf
description "camlistored"

start on started networking
stop on runlevel [!2345]

setuid eric
setgid eric
env USER=eric
env HOME=/home/eric

script
  $HOME/camlistore/bin/camlistored
end script

client-config.json

{
  "servers": {
    "localhost": {
      "server": "http://localhost:3179",
      "auth": "localhost",
      "default": false
    },
    "ankh": {
      "server": "https://my.vps:3179",
      "auth": "userpass:fooname:foopassword",
      "default": true,
      "trustedCerts": ["1e6c5f976eda865ef13a"] // this value comes from an error message the first time you try to camput something
    }
  },

  "identity": "FFFFFFFF",
  "ignoredFiles": [
    ".DS_Store",
    ".*-Spotlight"
  ]
}

server-config.json

{
  "auth": "userpass:fooname:foopassword:vivify=barpassword",
  "identity": "FFFFFFFF",
  "identitySecretRing": "/home/eric/.config/camlistore/identity-secring.gpg",
  "listen": ":3179",
  "https":  true,
  "httpCert": "/home/eric/.ssl/server.crt", // generate these by hand, see above
  "httpKey": "/home/eric/.ssl/server.key",
  "publish": {},
  "s3": "KEY:SECRET:bucket-name",
  "sqlite": "/home/eric/var/camlistore/camli-index.db"
}

This is great! The most confusing thing about camlistore was figuring out where everything runs.

So the client gets/puts data into the server. Can it do anything else like keep partial copies of a larger blob store? Can the different camliervers talk to each other (and how!?). These are some questions I want to look into at the meeting. Trying to figure out the overlap between git-annex and camlistore basically.

@bcomnes there are three main parts I think: the blob store (which is where you write stuff), the indexer (which is what you query) and the client, which can evidently do some transparent caching. This is gleaned from the overview.

I think one possible answer to "how can I make sure I have access to stuff on the go" for devices with smaller storage is to have the client cache recently accessed stuff (and maybe predictively too?).

Also I'm curious if running an indexer on your laptop with the blob store remote might be another answer.

I'll have a better idea once I get a local copy syncing with the VPS/S3 instance.

Also check out http://camlistore.org/pkg/blobserver/cond/

also camtool sync -help and "how to autosync with another camlistore server?"

After mkdiring in the root folder in the mount, nothing seems to happen. The its logged that blobs were received, but nothing shows up in the web UI or in the mount. Any ideas. Running into a wall actually storing and getting data out of the server beyond camput and camget.

In case anyone sees the same problem as @bcomnes, I solved this by making sure the client and server were using the same identity, gpg keys. For example, I copied the identity_secring.gpg key from the server config to the client config, and I made sure on the client that client-config.json's field "identity" used the correct identity.