Skip to content

Instantly share code, notes, and snippets.

@bahamas10

bahamas10/README.md

Last active January 17, 2019 19:24
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bahamas10/12fc5509a047a3c294c15cc666080e2e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bahamas10/12fc5509a047a3c294c15cc666080e2e to your computer and use it in GitHub Desktop.
Download ZIP
Buckets Demo Setup on Vanilla SmartOS
Raw
README.md

Buckets Demo

Installation process on a vanilla SmartOS hypervisor. Based off of this gist.

Create the Zones

I created all of the zones JSON files in /zones/zonecfg:

mkdir /zones/zonecfg
cd /zones/zonecfg

On this particular global zone there is only 1 NIC, so all VMs were created on the "admin" network. rapture.com is my home domain but these values can be changed to anything to fit the environment.

muskie.json

{
  "brand": "joyent",
  "image_uuid": "04a48d7d-6bb5-4e83-8c3b-e60a99e0f48f",
  "autoboot": true,
  "alias": "buckets-muskie",
  "hostname": "buckets-muskie.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "10.0.1.2",
    "10.0.1.3"
  ],
  "ram": 512,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.120",
      "gateway": "10.0.1.1",
      "netmask": "255.255.255.0",
      "primary": true
    }
  ]
}

electric-moray.json

{
  "brand": "joyent",
  "image_uuid": "c6a275e4-c730-11e8-8c5f-9b24fe560a8f",
  "autoboot": true,
  "alias": "buckets-electric-moray",
  "hostname": "buckets-electric-moray.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "10.0.1.2",
    "10.0.1.3"
  ],
  "ram": 512,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.121",
      "gateway": "10.0.1.1",
      "netmask": "255.255.255.0",
      "primary": true
    }
  ]
}

demo0.json

{
  "brand": "lx",
  "image_uuid": "04179d8e-188a-11e7-af4a-1349e98cbd17",
  "autoboot": true,
  "kernel_version": "3.13.0",
  "alias": "buckets-demo0",
  "hostname": "buckets-demo0.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "10.0.1.2",
    "10.0.1.3"
  ],
  "ram": 1024,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.124",
      "netmask": "255.255.255.0",
      "gateway": "10.0.1.1",
      "primary": true
    }
  ]
}

demo1.json

{
  "brand": "lx",
  "image_uuid": "04179d8e-188a-11e7-af4a-1349e98cbd17",
  "autoboot": true,
  "kernel_version": "3.13.0",
  "alias": "buckets-demo1",
  "hostname": "buckets-demo1.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "10.0.1.2",
    "10.0.1.3"
  ],
  "ram": 1024,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.125",
      "netmask": "255.255.255.0",
      "gateway": "10.0.1.1",
      "primary": true
    }
  ]
}

postegres0.json

{
  "brand": "joyent",
  "image_uuid": "c6a275e4-c730-11e8-8c5f-9b24fe560a8f",
  "autoboot": true,
  "alias": "buckets-postgres0",
  "hostname": "buckets-postgres0.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "10.0.1.2",
    "10.0.1.3"
  ],
  "ram": 1024,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.122",
      "gateway": "10.0.1.1",
      "netmask": "255.255.255.0",
      "primary": true
    }
  ]
}

postgres1.json

{
  "brand": "joyent",
  "image_uuid": "c6a275e4-c730-11e8-8c5f-9b24fe560a8f",
  "autoboot": true,
  "alias": "buckets-postgres1",
  "hostname": "buckets-postgres1.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "10.0.1.2",
    "10.0.1.3"
  ],
  "ram": 1024,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.123",
      "gateway": "10.0.1.1",
      "netmask": "255.255.255.0",
      "primary": true
    }
  ]
}

Install all images required by the JSON manifests:

cat *.json | json -ag image_uuid | sort | uniq | xargs -n1 imgadm import

Install all VMs:

for f in *.json; do vmadm create -f "$f"; done

IP addresses for the zones:

# vmadm lookup -j | json -ag alias nics.0.ip
buckets-electric-moray 10.0.1.121
buckets-postgres0 10.0.1.122
buckets-postgres1 10.0.1.123
buckets-demo0 10.0.1.124
buckets-demo1 10.0.1.125
buckets-muskie 10.0.1.120

Configure Zones

Most of these zones I used zlogin to manage. Some of them, however, pull private bits from GitHub which requires access to an ssh private key/signer. In that case, I added my authorized_keys to the root user and SSH'd in with my agent forwarded.

Muskie

# deps
pkgin up
pkgin fug
pkgin in git build-essential
pkgin in python27

# clone muskie
cd ~
git clone git@github.com:joyent/manta-muskie.git
cd manta-muskie
git checkout buckets-demo
make

# fix libmanta deps
cd node_modules/libmanta/
vim package.json
# change dependencies.moray => "git+https://github.com/joyent/node-moray.git#buckets-demo"
~/manta-muskie/build/node/bin/npm install

# fix fast protocol
find . -type f -name fast_protocol.js
vim node_modules/libmanta/node_modules/moray/node_modules/fast/lib/fast_protocol.js
vim node_modules/marlin/node_modules/moray/node_modules/fast/lib/fast_protocol.js
vim node_modules/moray/node_modules/fast/lib/fast_protocol.js

# pull muskie config from gist
curl -O https://gist.github.com/kellymclaughlin/e1134e6f7814628e12061966cd9caa0e/raw/ac65127b0ce1b2c80cf887f3eecf440e31449c3f/muskie-config.json
vim muskie-config.json
# set "moray" to 10.0.1.121 (electric-moray zone)

# start muskie (tmux or something here)
./build/node/bin/node main.js -f muskie-config.json -v 2>&1 | ./build/node/bin/node ./node_modules/.bin/bunyan -o short

Electric Moray

# deps
pkgin in git build-essential

# clone electric moray
cd ~
git clone git://github.com/joyent/electric-moray.git
cd electric-moray
git checkout buckets-demo
make

# fix fast protocol
find . -name fast_protocol.js
vim node_modules/fast/lib/fast_protocol.js
vim node_modules/moray/node_modules/fast/lib/fast_protocol.js

# pull electric moray config from gist
curl -O https://gist.github.com/kellymclaughlin/e1134e6f7814628e12061966cd9caa0e/raw/ac65127b0ce1b2c80cf887f3eecf440e31449c3f/electric-moray-config.json

# start electric moray
./build/node/bin/node main.js -f electric-moray-config.json 2>&1 | ./build/node/bin/node ./node_modules/.bin/bunyan -o short

Postgres

Do this in both zones

# deps
pkgin up
pkgin in -y git build-essential

# source
cd ~
git clone https://github.com/kellymclaughlin/postgres.git
cd postgres
git checkout joyent/10.5

# compile and install
pkgin in -y flex
LDFLAGS='-L/opt/local/lib -R/opt/local/lib' ./configure --prefix=/opt/local --exec-prefix=/opt/local/pgsql --with-uuid=ossp
make
make install
(cd contrib/pgcrypto/ && make install)
(cd contrib/hstore && make install)

# setup environment
groupadd postgres
useradd -g postgres postgres
mkdir /postgres
chown postgres:postgres /postgres

# setup postgres
sudo -Hu postgres /opt/local/pgsql/bin/initdb -D /postgres/data
vim /postgres/data/postgresql.conf
vim /postgres/data/pg_hba.conf
sudo -Hu postgres /opt/local/pgsql/bin/pg_ctl -D /postgres/data start
sudo -Hu postgres /opt/local/pgsql/bin/createdb -U postgres test
sudo -Hu postgres /opt/local/pgsql/bin/psql -U postgres -c 'CREATE EXTENSION pgcrypto' test
sudo -Hu postgres /opt/local/pgsql/bin/psql -U postgres -c 'CREATE EXTENSION hstore' test

cd ~
# pull the correct file for whichever zone you are on
wget https://gist.githubusercontent.com/kellymclaughlin/e1134e6f7814628e12061966cd9caa0e/raw/ac65127b0ce1b2c80cf887f3eecf440e31449c3f/manta_bucket_demo_schema_1579.sql
sudo -Hu postgres /opt/local/pgsql/bin/psql -U postgres -d test -f manta_bucket_demo_schema_1579.sql

Buckets Demo

# deps
apt-get update
apt install build-essential git

# install rust
curl https://sh.rustup.rs -sSf | sh
. ~/.cargo/env

# clone buckets demo code
cd ~
git clone git@github.com:joyent/buckets-demo.git
cd buckets-demo

cargo build --release
# specify the correct postgres IP address.  demo0 -> postgres0, demo1 -> postgres1, etc.
cargo run -- --pg-url postgresql://postgres@10.0.1.123:5432/test --address 0.0.0.0

Test

From any machine that can hit the muskie zone:

$ curl -X PUT -i -H 'Content-Type: application/json; type=bucket' http://10.0.1.120:8080/14aafd84-a57f-11e8-8706-4fc23c74c5e7/buckets/test-bucket-1
HTTP/1.1 204 No Content
Connection: close
Date: Thu, 17 Jan 2019 19:18:27 GMT
Server: Manta
x-request-id: aae3cbf0-1a8c-11e9-8182-3b9783b9cc0f
x-response-time: 104
x-server-name: buckets-muskie.rapture.com

$ curl -sSi http://10.0.1.120:8080/14aafd84-a57f-11e8-8706-4fc23c74c5e7/buckets/test-bucket-1 | json
HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 155
Content-MD5: fOCOH0n+ALYSEqzRMonrvw==
Date: Thu, 17 Jan 2019 19:18:33 GMT
Server: Manta
x-request-id: aea17030-1a8c-11e9-8182-3b9783b9cc0f
x-response-time: 78
x-server-name: buckets-muskie.rapture.com

{
  "created": "2019-01-17T19:18:27.648156Z",
  "id": "4b4dfefe-e711-4e36-9f36-11de338fda6b",
  "name": "test-bucket-1",
  "owner": "14aafd84-a57f-11e8-8706-4fc23c74c5e7"
}

The user UUID 14aafd84-a57f-11e8-8706-4fc23c74c5e7 is hardcoded and can be used with no authentication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment