Skip to content

Instantly share code, notes, and snippets.

@jhrcz

jhrcz/etn-opennebula-state-2013.md

Last active December 23, 2015 21:19
Show Gist options
  • Save jhrcz/6695904 to your computer and use it in GitHub Desktop.
Save jhrcz/6695904 to your computer and use it in GitHub Desktop.
Download ZIP
etn-opennebula-state-2013
Raw
etn-opennebula-state-2013.md

one1c (ETN One Cloud #1) overview 2013

  • opennebula 3.2.1, preparing for upgrade to 4.x (requires migrating gfs2clvm close to upstream lvm driver, renaming all lvm names )

architecture

  • 8 viritualisation nodes (one1-host-X )

  • virtualisation nodes with only SSD drives for system, storage for opennebula on central sas storage

  • opennebula implemented after the architecture was ready, ada pted to it

  • 152 volumes (12 nonperistent, 140 peristent), 105 vms

  • one single-instance management node (one1-mgmt-1)

  • virtualisation nodes interconnected in redhat cluster for clvm and gfs2

  • custom gfs2clvm driver implementing opennebula storage drivers for clvm and gfs2

  • gfs2 used for file-data shared across virtualisation nodes

  • clvm used for vm images, persistent and non-persistent

  • management node not part of the cluster, installed on independent virtualisation host

  • management node not fully HA, but on shared storage so it could be started elsewhere when hw fails

  • ha features not used - vm resubmit after host failer, etc ... requires manual administrator steps to delete/move vms to other node or fix node problems, bring it back up and resubmit vms on the same node

  • hook for renaming vms based on template name

  • hook for changing vm owner based on the template owner

common usecases

delivering persistent production server

creating os image as a copy of os image template, which is a post-kickstart centos instllation with minimal changes to be ready for future use as tempalte (optionaly ebs-like data image is created as datablock image of defined size), template and instantiating a persistent vm. this vm is used as a standard persistent virtual server and we need to meet defined SLA. OpenNebula is in role of virtualisation management, central management a server list.

some special volume sources could be used thanks to gfs2clvm vol://{volid} lvm://{vmid}/{vmdiskid}

creating vms reuires some additional manual steps - booking ip/mac in our ip-list and defining dns records (before template is created). mac (and nodename from dns name) is then used in vm template (internal doc contains copy-n-paste ready template for defining it in advanced tab)

non-persistent testing installation automation

OS image used as template for persistent servers is used as nonpesistent

non-persistent on demand servers

the same as testing installatoin automation

problems

  • very slov initial load of web management (sunstone) - changing to mysql backend didn't help. initial list of vms, templates etc in refresh state for 1-2 minutes

  • io-peaks because of almost identical vm os installation. similar cron/logrotate task. could be solved with using ionicea and nice to minimize cross-vm impact

  • when using munin libvirt plugins, its difficult to pair vm ids with vm names

  • creating persistent os image from template takes too long (in the meaning of users - admins) ex. 8-10 mins. depends on the 'dd' speed.

  • sometime volume size is detected incorrectly and inccorect info is shown in sunstone. lvm size is correct

  • adding new datablock volume requires vm shutdown, modify template and reinitiate it again - new vmid is generated - one 4 could help

  • overcommit could not be used - casuses problems because many vms run java/jvm

  • dd taks when os images are cloned executed from management interfaces and run on virtualisation nodes cause high load value, slow donw io

  • deleting a vm before it is finished, removes lvm but the dd command is still running

  • hook which is changing vm names do not take effect sometime

  • susntone does not report resouorce usage in useful way - this should be fixed with one 4

  • resource edit and renames not supported - sould be fixed by one 4

  • virtualisation nodes change their state to ERROR because of centos tmp cleaning, reuires onehost sync or changing mtime of remote scipts

  • io and cpu shaping does not work out of the box - some custom cgroups integration needed

accounting / reporting

we do not use internal openenbula accounting. i have custom scripts who reports vms, resources, volumes. it has cvs output, so comparing betwen months is possible. this could help detecting vmid changes of the "same" vm.

we are not ordinary cloud provider, we use opennebula for managing vms internaly for admins (evevn if vms are for clients, with SLA and with defined resources).

we use munin to have overview of virtualisation nodes, disk io, memory and network usage

custom scripts to report unused images and templates - they could be leftover with a purpose or just not deleted

future

  • upgrade to opennebula 4.x

  • build another independent environment as a cloud for developers - this will bring different users than admins

  • automate accounting

  • test other storage backend like ceph for distributing some storage

  • dns integration for generating records for non-persistent vms

  • use opennebula's iaas as a backend for some paas service (openshift?)

  • check if opennebula could be used as management of couple independent virtualisation host with their own storage

EOF.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment