Profilling and Debugging Phoenix Erlang VM inside/outside nanobox

Profilling and Debugging Phoenix Erlang VM.md

Table Of Contents:

• On Local Machine
  • Outside nanobox
  • Inside nanobox
• On Remote Machine
  • Outside nanobox
  • Inside nanobox

Running erlang observer varies depending on whether it is local, remote, inside/outside nanobox

On Local Machine

Outside nanobox

• run phoenix inside iex iex -s mix phx.server
• run observer :observer.start

Inside nanobox

TODO

On Remote Machine

This section is based on a blog post observing remote elixir docker nodes continued

Outside nanobox

1. Enable GatewayPorts

   By default remote port forwarding can only bind to the loopback adapter on the remote host which will not be accessible from a container. To change this we must edit /etc/ssh/sshd_config on the server and add this line:

   GatewayPorts clientspecified

   Ensure there are no other GatewayPorts lines in the file. Next, reload the configuration by executing sudo service ssh restart. (These instructions are for Ubuntu, it might be slightly different on other distributions.)

2. Run a remote IEX session in a container

   We then start an Elixir node on the server in a Docker container.

   [email protected]:~$ docker run --rm -it elixir:1.4.2-slim bash
   root@866b1839d4fe:/# iex --cookie mycookie --name [email protected]
   Erlang/OTP 19 [erts-8.3.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
   
   Interactive Elixir (1.4.2) - press Ctrl+C to exit (type h() ENTER for help)
   iex([email protected])1>
   

3. Run a local IEX session

   We also start a local IEX session. We need to set the cookie and also the node name. The IP of the name is the Docker bridge interface (docker0) on the server (by default 172.17.0.1) as our remote node will use this to connect.

   We also use the inet_dist_listen_min/max parameters to ensure that node listens on port 19000 so we can forward this port.

   my-laptop$ iex --name [email protected] --cookie mycookie --erl "-kernel inet_dist_listen_min 19000 inet_dist_listen_max 19000"
   Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
   
   Interactive Elixir (1.4.2) - press Ctrl+C to exit (type h() ENTER for help)
   iex([email protected])1>
   

4. Forward local ports to the server

   Next we create the SSH tunnel and map the local EPMD port and port 19000 onto the Docker bridge IP on the remote server.

   ssh [email protected] -R 172.17.0.1:4369:127.0.0.1:4369 -R 172.17.0.1:19000:127.0.0.1:19000 -N
   

5. Connect the nodes together

   Now we can get the remote node to connect to our local node:

   iex([email protected])1> Node.connect :'[email protected]'
   true
   

   The connection is bi-directional so on our local node we can see that our remote node is now connected and we can also start the Observer. Once started, we can select our remote node from the Observer ‘Nodes’ menu.

   iex([email protected])2> Node.list
   [:"[email protected]"]
   iex([email protected])3> :observer.start
   :ok
   

Inside nanobox

This method involve direct ssh access to nanobox instance therefore you can manipulate live servers and containers that might cause damage/dataloss please proceed really CAREFULLY!!!

1. Enable TCP Port forwarding on your web component and start using node-start on boxfile.yml

    web.umbrella:
      start:
        nginx: nginx -c /app/config/nginx.conf
        start: node-start mix phx.server
   
      writable_dirs:
        - "_build"
   
      ports:
        - tcp:19000:19000
   
      network_dirs:
        data.storage:
          - app/crashDump
   

2. Get a copy of your nanobox instance ssh key

   follow instructions from nanobox-app-ssh-keys

3. Enable GatewayPorts

   By default remote port forwarding can only bind to the loopback adapter on the remote host which will not be accessible from a container. To change this we must edit /etc/ssh/sshd_config on the server and add this line:

   GatewayPorts clientspecified

   Ensure there are no other GatewayPorts lines in the file. Next, reload the configuration by executing sudo service ssh restart. (These instructions are for Ubuntu, it might be slightly different on other distributions.)

4. Run a remote IEX session in a container

   We then start an Elixir node on the server in a Docker container.

   my-laptop$ nanobox console <app-name> <component-name>
   /app $ node-attach
   Erlang/OTP 20 [erts-9.1.5] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:10] [kernel-poll:false]
   
   Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
   iex([email protected])1>
   
   

5. Run a local IEX session

   We also start a local IEX session. We need to set the cookie and also the node name. The IP of the name is the Docker bridge interface (docker0) on the server (by default 172.17.0.1) as our remote node will use this to connect.

   We also use the inet_dist_listen_min/max parameters to ensure that node listens on port 19000 so we can forward this port.

   my-laptop$ iex --name [email protected] --cookie nanobox --erl "-kernel inet_dist_listen_min 19000 inet_dist_listen_max 19000"
   Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
   
   Interactive Elixir (1.4.2) - press Ctrl+C to exit (type h() ENTER for help)
   iex([email protected])1>
   

6. Forward local ports to the server

   Next we create the SSH tunnel and map the local EPMD port and port 19000 onto the Docker bridge IP on the remote server.

   ssh ubuntu@<app-ip-address> -i /path/to/private_key -R 172.17.0.1:4369:127.0.0.1:4369 -R 172.17.0.1:19000:127.0.0.1:19000 -N
   

7. Connect the nodes together

   Now we can get the remote node to connect to our local node:

   iex([email protected])1> Node.connect :'[email protected]'
   true
   

   The connection is bi-directional so on our local node we can see that our remote node is now connected and we can also start the Observer. Once started, we can select our remote node from the Observer ‘Nodes’ menu.

   iex([email protected])2> Node.list
   [:"[email protected]", :"[email protected]"]
   iex([email protected])3> :observer.start
   :ok