Skip to content

Instantly share code, notes, and snippets.

@yuanying

yuanying/create-dev-env-magnum.md

Last active March 11, 2019 09:16
Show Gist options
  • Save yuanying/f27cef22738acf8b1db5 to your computer and use it in GitHub Desktop.
Save yuanying/f27cef22738acf8b1db5 to your computer and use it in GitHub Desktop.
Download ZIP
Develop Magnum with Devstack
Raw
create-dev-env-magnum.md

Develop Magnum with Devstack

Vagrant

I'm using vagrant (parallels on Mac) to boot a devstack. Vagrantfile is below. Devstack IP Address is 192.168.11.197.

Vagrant.configure('2') do |config|
  config.vm.box = "trusty64"

  config.vm.define :devstack do |devstack|
    devstack.vm.hostname = "devstack"
    devstack.vm.network :private_network, ip: "192.168.123.10"
    devstack.vm.network :public_network, dev: 'br0', mode: 'bridge', ip: "192.168.11.197"

    devstack.vm.synced_folder ".", "/vagrant", type: "nfs"
    #devstack.vm.synced_folder "/home/yuanying/Projects", "/home/yuanying/Projects", type: "nfs"

    devstack.vm.provider :libvirt do |libvirt, override|
      libvirt.memory = 8192
      libvirt.nested = true
    end

    devstack.vm.provision "shell", path: "./install.sh"
  end

end

and vagrant up.

Inside Devstack VM

Install requirement packages

$ sudo apt-get update
$ sudo apt-get install libffi-dev libssl-dev git vim \
                       libxml2-dev libsqlite3-dev libxslt1-dev -y

Network settings

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Install DevStack

$ cd ~
$ sudo mkdir - p /etc/neutron
$ sudo chown -R $USER /etc/neutron
$ echo "dhcp-option-force=26,1400" >> /etc/neutron/dnsmasq.conf
$ git clone https://git.openstack.org/openstack-dev/devstack
$ cd devstack
$ ./stack.sh

local.conf is below.

[[local|localrc]]
HOST_IP=192.168.11.197
#SERVICE_HOST=192.168.202.4
#HEAT_API_HOST=${SERVICE_HOST}
#HEAT_API_CFN_HOST=${SERVICE_HOST}
#HEAT_ENGINE_HOST=${SERVICE_HOST}
#HEAT_API_CW_HOST=${SERVICE_HOST}

FLOATING_RANGE=172.16.12.0/24
Q_FLOATING_ALLOCATION_POOL="start=172.16.12.10,end=172.16.12.200"
PUBLIC_NETWORK_GATEWAY=172.16.12.1

Q_USE_SECGROUP=True
ENABLE_TENANT_VLANS=True
TENANT_VLAN_RANGE=1000:1999
PHYSICAL_NETWORK=default
OVS_PHYSICAL_BRIDGE=br-ex

NETWORK_GATEWAY=10.11.12.1
FIXED_RANGE=10.11.12.0/24
FIXED_NETWORK_SIZE=256

ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=stackdb
RABBIT_PASSWORD=stackqueue
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=tokentoken

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service q-lbaas
enable_service neutron

enable_service h-eng
enable_service h-api
enable_service h-api-cfn
enable_service h-api-cw

enable_plugin barbican https://git.openstack.org/openstack/barbican

#LOGFILE=$DEST/logs/devstack.log
DEST=/opt/stack
#SCREEN_LOGDIR=$DEST/logs/screen

[[post-config|/etc/neutron/dhcp_agent.ini]]
[DEFAULT]
dnsmasq_config_file = /etc/neutron/dnsmasq.conf

Magnum

Install

Magnum is outside of devstack. In this case, magnum will install to 192.168.11.132 host.

Magnum Server

$ cd ~
$ git clone https://github.com/openstack/magnum.git
$ cd magnum
$ tox -evenv -- echo 'done'

Magnum Client

$ cd ~
$ git clone https://github.com/openstack/python-magnumclient.git
$ cd python-magnumclient
$ tox -evenv -- echo 'done'

Configuration

Setup trust

TRUSTEE_DOMAIN_ID=$(
    openstack domain create magnum \
        --description "Owns users and projects created by magnum" \
        -f value -c id
)
TRUSTEE_DOMAIN_ADMIN_ID=$(
    openstack user create trustee_domain_admin \
        --password "password" \
        --domain=${TRUSTEE_DOMAIN_ID} \
        --or-show \
        -f value -c id
)
openstack --os-identity-api-version 3 role add \
          --user $TRUSTEE_DOMAIN_ADMIN_ID --domain $TRUSTEE_DOMAIN_ID \
          admin

Create config

$ sudo mkdir -p /etc/magnum
$ cd /etc/magnum
$ sudo vim magnum.conf

magnum.conf has below content. Change 192.168.11.197 to your devstack IP address.

[DEFAULT]
debug = True
verbose = True

rabbit_userid=stackrabbit
rabbit_password = stackqueue
rabbit_hosts = 192.168.11.197
rpc_backend = rabbit

[database]
connection = mysql://root:[email protected]/magnum

[keystone_authtoken]
admin_password = openstack
admin_user = nova
admin_tenant_name = service
identity_uri = http://192.168.11.197:35357
#user_domain_id = default
#project_domain_id = default

auth_uri=http://192.168.11.197:5000/v3

[api]

host = 0.0.0.0

[trust]
#trustee_domain_id = magnum
#trustee_domain_admin_id = trustee_domain_admin
trustee_domain_admin_password = password

Update trust config

# set trustee domain id
sudo sed -i "s/#trustee_domain_id\s*=.*/trustee_domain_id=${TRUSTEE_DOMAIN_ID}/" \
         /etc/magnum/magnum.conf

# set trustee domain admin id
sudo sed -i "s/#trustee_domain_admin_id\s*=.*/trustee_domain_admin_id=${TRUSTEE_DOMAIN_ADMIN_ID}/" \
         /etc/magnum/magnum.conf

# set trustee domain admin password
sudo sed -i "s/#trustee_domain_admin_password\s*=.*/trustee_domain_admin_password=password/" \
         /etc/magnum/magnum.conf

# set correct region name to clients
sudo sed -i "s/#region_name\s*=.*/region_name=RegionOne/" \
         /etc/magnum/magnum.conf

register magnum service to keystone

$ source ~/devstack/openrc admin admin
$ openstack service create --name=magnum \
                           --description="Magnum Container Service" \
                           container
$ openstack endpoint create --region=RegionOne \
                            magnum public http://192.168.11.132:9511/v1
$ openstack endpoint create --region=RegionOne \
                            magnum internal http://192.168.11.132:9511/v1
$ openstack endpoint create --region=RegionOne \
                            magnum admin http://192.168.11.132:9511/v1

Register Image to glance

$ curl -O https://fedorapeople.org/groups/magnum/fedora-21-atomic-5.qcow2
$ source ~/devstack/openrc admin admin
$ glance image-create --name fedora-21-atomic-5 \
                    --visibility public \
                    --disk-format qcow2 \
                    --os-distro fedora-atomic \
                    --container-format bare < fedora-21-atomic-5.qcow2

Add default keypair to demo user

$ ssh-keygen
$ source ~/devstack/openrc demo demo
$ nova keypair-add --pub-key ~/.ssh/id_rsa.pub default

Database

$ mysql -h 192.168.11.197 -u root -pstackdb mysql <<EOF
CREATE DATABASE IF NOT EXISTS magnum DEFAULT CHARACTER SET utf8;
GRANT ALL PRIVILEGES ON magnum.* TO
    'root'@'%' IDENTIFIED BY 'stackdb'
EOF

and create tables.

$ cd ~/magnum
$ source .tox/venv/bin/activate
$ pip install mysql-python
$ magnum-db-manage upgrade

Start Magnum

magnum-api

$ cd ~/magnum
$ source .tox/venv/bin/activate
$ magnum-api

magnum-conductor

$ cd ~/magnum
$ source .tox/venv/bin/activate
$ magnum-conductor

python-magnumclient

$ cd ~/python-magnumclient
$ source .tox/venv/bin/activate
$ magnum bay-list

Test magnum

Try to create bay

$ magnum baymodel-create --name kubernetes --keypair-id default \
                         --external-network-id public \
                         --image-id fedora-21-atomic-5 \
                         --flavor-id m1.small \
                         --docker-volume-size 1 \
                         --network-driver flannel \
                         --coe kubernetes

$ magnum bay-create --name k8s_bay --baymodel kubernetes

$ magnum baymodel-create --name swarm \
                         --image-id fedora-21-atomic-5 \
                         --keypair-id default \
                         --external-network-id public \
                         --flavor-id m1.small \
                         --docker-volume-size 1 \
                         --coe swarm

Try to create pod

$ magnum pod-create --bay-id 99cab72f-16a7-4564-8d73-d4497f51f557 \
    --pod-file redis-master.json

After reload

$ sudo ip addr add 10.0.0.1/24 dev br-ex
$ sudo ip addr add 172.16.12.1/24 dev br-ex
$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
$ sudo losetup /dev/loop0 /opt/stack/data/stack-volumes-default-backing-file ;
$ sudo losetup /dev/loop1 /opt/stack/data/stack-volumes-lvmdriver-1-backing-file ;
Raw
demo.md

Demo command list

nova list
neutron net-list

magnum baymodel-list

nova keypair-list

ssh-keygen

nova keypair-add --pub-key ~/.ssh/id_rsa.pub default

magnum baymodel-create \
  --name kubernetes \
  --keypair-id default \
  --external-network-id public \
  --image-id fedora-21-atomic-5 \
  --flavor-id m1.small \
  --docker-volume-size 1 \
  --network-driver flannel \
  --coe kubernetes

magnum bay-create --name k8sbay --baymodel kubernetes

heat stack-list

nova list

heat resource-list k8sbay-ap76ggow3rpc

https://github.com/openstack/magnum/blob/master/doc/source/dev/dev-tls.rst

openssl genrsa -out client.key 4096

cat > client.conf << END
[req]
distinguished_name = req_distinguished_name
req_extensions     = req_ext
prompt = no
[req_distinguished_name]
CN = [email protected]
[req_ext]
extendedKeyUsage = clientAuth
END

openssl req -new -days 365 \
    -config client.conf \
    -key client.key \
    -out client.csr

openssl req -text -in client.csr

magnum ca-sign --bay k8sbay --csr client.csr > client.crt

openssl x509 -text -in client.crt

magnum ca-show --bay k8sbay > ca.crt


KUBERNETES_URL=$(magnum bay-show k8sbay |
                 awk '/ api_address /{print $4}')

kubectl version --certificate-authority=ca.crt \
               --client-key=client.key \
               --client-certificate=client.crt -s $KUBERNETES_URL

kubectl config set-cluster secure-k8sbay --server=${KUBERNETES_URL} \
   --certificate-authority=${PWD}/ca.crt
kubectl config set-credentials client --certificate-authority=${PWD}/ca.crt \
   --client-key=${PWD}/client.key --client-certificate=${PWD}/client.crt
kubectl config set-context secure-k8sbay --cluster=secure-k8sbay --user=client
kubectl config use-context secure-k8sbay


cat > nginx.yml << END
apiVersion: v1
kind: Pod
metadata:
  name: nginx

spec:
  containers:
  - name: nginx
    image: nginx
    ports:
      - containerPort: 80
END

cat > nginx-service.yml << END
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ports:
    - port: 80
  selector:
    app: nginx
END

kubectl create -f nginx.yml
@HackToday
Copy link

yuanying, I have some questions, could you help that ?

  1. For now, magnum support container creation following ways:
    (1) bare docker
    (2) baymodel related ways

For (2), it used heat

Does bay equal with VM/Bare metal node ?
And Pods is a collection of containers.
And containers can be in any bay.

While Pods have N-M mapping to Bay ?

@yuanying
Copy link
Author

yuanying commented Jan 8, 2015

Does bay equal with VM/Bare metal node ?

Now, bay is cluster of VM/Bare metal nodes,
or manager of kubernetes cluster which know about kubernetes master and minions.

While Pods have N-M mapping to Bay ?

In this time, bay is just only kubernetes instance.
So bay has pods with 1-N mapping.

@HackToday
Copy link

Thanks @yuanying comments above.

Based on setup env update today,
It means it worked for this:

magnum pod-create finally talked with k8s, and k8s has cli calling, which means magnum and k8s cli installed in same machine.

and k8s cli finally create pod. Is it right ?

  1. Also
    Why "And create br-ex before devstack is created."
    Devstack created br-ex, I think

@yuanying
Copy link
Author

yuanying commented Jan 8, 2015

Why "And create br-ex before devstack is created." Devstack created br-ex, I think

I want to use interface p0 which added to br-ex bridge,
and associated ip address is used for connecting openstack api server.

@tobegit3hub
Copy link

Hi @yuanying, I have the problem to vagrant up and here's the output.

Bringing machine 'devstack' up with 'virtualbox' provider...
There are errors in the configuration of this machine. Please fix
the following errors and try again:

shell provisioner:
* `path` for shell provisioner does not exist on the host system: /home/tobe/temp/magnum_devstack/install.sh

So what's the content of install.sh and should I create it by myself?

@shinobu-x
Copy link

@yuanying,

Thank you for your great guide.
One question from my side.
Why we need to force mtu to 1400?

$ echo "dhcp-option-force=26,1400" >> /etc/neutron/dnsmasq.conf

I'm just curious about that.

Rgds,
Shinobu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment