FiFo on Single IP dedicated (hetzner) server, with NAT-ted Zones

0-INTRO

These are my interfaces:
========================

- Main interface (with internet connection, the only physical cable..)

igb0: flags=1104943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,DHCP,ROUTER,IPv4> mtu 1500 index 2
        inet ***.76.170.*** netmask fffffff8 broadcast ***.76.170.***
        ether 33:11:22:33:22:44


- A tunnel interface created by http://tinc-vpn.org/
  this will allow adding additional servers
  
tun0: flags=11008d1<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST,ROUTER,IPv4> mtu 1500 index 4
        inet 10.20.0.253 --> 10.20.0.253 netmask ff000000
        ether 40:2f:45:ad:25:ff
        

- The NAT gateway for Zones

nat0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 7
        inet 10.20.0.254 netmask ffffff00 broadcast 10.20.0.255
        ether 2:8:20:58:35:89


- And ofcourse localhost

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128


Some things I've tried:
=======================


1) Running chunter on nat0:  (10.20.0.254)
---------------------------

nat0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 7
      inet 10.20.0.254 netmask ffffff00

[gui]	no errors!

VM start state updates do not appear immediately,
but after another restart of chunter and some patience, it did.

HOWEVER, I cannot open and connect to KVM VNC.  But, zone console works.



2) Running chunter on igb0:1 with IP 10.20.0.252
----------------------------

igb0:1: flags=1100943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
        inet 10.20.0.252 netmask ffffff00 broadcast 10.20.0.255


[root@00-25-90-d8-3c-44 ~]# ifconfig |grep UP
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
igb0: flags=1104943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,DHCP,ROUTER,IPv4> mtu 1500 index 2
igb0:1: flags=1100943<UP,BROADCAST,RUNNING,PROMISC,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
tun0: flags=11008d1<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST,ROUTER,IPv4> mtu 1500 index 4
nat0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 7
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1


[gui]	chunter 	1cff5d03-e4a0-4483-bdd7-ecbb619aca8c 	Chunter server down.

Now VNC works! But live status updates do not. Or any status updates...
Chunter cannot reach the FiFo zone (or just howl?) through multicast, more or less.


3) Running chunter on a dedicated vnic (just like the zones themselves have vnics)
--------------------------------------

chunter0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 8
        inet 10.20.0.252 netmask ffffff00 broadcast 10.20.0.255
        ether 2:8:20:b8:c5:f1

[gui]	no errors!

VM start state updates etc appear immediately.

HOWEVER, I cannot open and connect to KVM VNC.  But, zone console works.

[root@dc9916e1-2de2-426e-9363-eb35e87823e4 ~]# telnet ***.76.170.*** 64840
Trying 144.76.170.210...



4) Adding another IP to igb0 again, this time only to connect igb0 directly to the nat0 (VM) network
----------------------------------

[gui]	no errors (yet!)

Now I can connect to VNC!  The reason is this:

[root@dc9916e1-2de2-426e-9363-eb35e87823e4 ~]# telnet ***.76.170.*** 64840
Trying 144.76.170.210...
Connected to gladis.ga.je.
Escape character is '^]'.
RFB 003.008

Without the second IP, fifo zone can't connect to the igb0 interface where VNC is bound.



============
|          |
| THE PAIN |
|          |
============


Once I had 4) figured out and running I was happy. But after a while, even a few minutes or so Chunter started failing, logging lots of crash messages and problems with mdns.

Restarting chunter doesn't help in this case.





===============
|             |
| THE DETAILS |
|             |
===============


Are attached as files in this gist.

1-chunter-conf-without-comments.sh

# [root@00-25-90-d8-3c-44 ~]# cat /opt/chunter/etc/chunter.conf |grep -v '#'
reserved_memory = 0MB
s3_upload_chunk_size = 5MB
parallel_uploads = 5
max_parallel_uploads = 10
preload = 2
s3_download_chunk_size = 5MB
parallel_downloads = 5
max_parallel_downloads = 10
kstat.metrics = enabled
kstat.arc = enabled
update_services_interval = 10s
snapshot_update_interval = 15m
zonemon_interval = 1s
zpool_interval = 15s
arc_interval = 30s
ip = 10.20.0.252:4200
cpu_type = default
libsnarl.realm = default
mdns.client.domain = .local
mdns.client.interface = 10.20.0.252
mdns.client.multicast_group = 224.0.0.251:5353
mdns.client.timeout = 1s500ms
mdns.client.retries = 40
mdns.client.retrie_delay = 150ms
mdns.client.max_downvotes = 5
mdns.client.pool.initial = 5
mdns.client.pool.max = 5
nodename = [email protected]
distributed_cookie = erlang
erlang.async_threads = 64
erlang.max_ports = 65536
log.console = file
log.console.level = info
log.console.file = /var/log/chunter/console.log
log.error.file = /var/log/chunter/error.log
log.debug.file = /var/log/chunter/debug.log
log.syslog = off
log.crash.file = /var/log/chunter/crash.log
log.crash.msg_size = 64KB
log.crash.size = 10MB
log.crash.date = $D0
log.crash.count = 5
log.error.redirect = on
log.error.messages_per_second = 100

2-fifo.js

// [root@00-25-90-d8-3c-44 ~]# vmadm get dc9916e1-2de2-426e-9363-eb35e87823e4
{
  "zonename": "dc9916e1-2de2-426e-9363-eb35e87823e4",
  "zonepath": "/zones/dc9916e1-2de2-426e-9363-eb35e87823e4",
  "autoboot": true,
  "brand": "joyent",
  "limit_priv": "default",
  "v": 1,
  "create_timestamp": "2013-11-14T11:11:32.695Z",
  "image_uuid": "fdea06b0-3f24-11e2-ac50-0b645575ce9d",
  "cpu_shares": 100,
  "zfs_io_priority": 100,
  "max_lwps": 2000,
  "max_physical_memory": 1024,
  "max_locked_memory": 1024,
  "max_swap": 1024,
  "cpu_cap": 100,
  "billing_id": "00000000-0000-0000-0000-000000000000",
  "owner_uuid": "00000000-0000-0000-0000-000000000000",
  "tmpfs": 1024,
  "dns_domain": "local",
  "resolvers": [
    "8.8.8.8",
    "8.8.4.4"
  ],
  "alias": "fifo",
  "nics": [
    {
      "interface": "net0",
      "mac": "52:cd:f2:5e:92:8c",
      "nic_tag": "admin",
      "gateway": "10.20.0.254",
      "ip": "10.20.0.10",
      "netmask": "255.255.255.0"
    }
  ],
  "uuid": "dc9916e1-2de2-426e-9363-eb35e87823e4",
  "zone_state": "running",
  "zoneid": 151,
  "last_modified": "2014-08-21T21:43:41.000Z",
  "firewall_enabled": false,
  "server_uuid": "00000000-0000-0000-0000-002590d83c44",
  "platform_buildstamp": "20140221T042147Z",
  "state": "running",
  "customer_metadata": {},
  "internal_metadata": {},
  "tags": {},
  "routes": {},
  "quota": 120,
  "zfs_root_compression": "lz4",
  "zfs_root_recsize": 131072,
  "zfs_filesystem": "zones/dc9916e1-2de2-426e-9363-eb35e87823e4",
  "zpool": "zones",
  "snapshots": [
    {
      "name": "pre-fifo-dev",
      "created_at": "2013-12-18T09:39:20.000Z"
    }
  ]
}

3-ipnat-config.sh

# [root@00-25-90-d8-3c-44 ~]# ipnat -l
List of active MAP/Redirect filters:
map igb0 10.20.0.0/24 -> 0.0.0.0/32
map igb0 10.20.0.0/24 -> 0.0.0.0/32 portmap tcp/udp auto
rdr igb0 0.0.0.0/0 port 22080 -> 10.20.0.80 port 22 tcp
rdr igb0 0.0.0.0/0 port 22090 -> 10.20.0.90 port 22 tcp