IOTstack tutorial: Executing commands outside the Node-Red container

A reasonably common requirement in a Node-Red flow is the ability to execute a command on the host system. The standard tool for this is an "exec" node.

An "exec" node works as expected when Node-Red is running as a native service but not when Node-Red is running in a container. That's because the command is running inside the container.

To help you understand the difference, consider this command:

$ grep "^PRETTY_NAME=" /etc/os-release

When you run that command on a Raspberry Pi outside container-space, the answer is:

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"

If you run the command inside a Node-Red container, the output will be:

PRETTY_NAME="Alpine Linux v3.11"

The same thing will happen if a Node-Red "exec" node executes that grep command when Node-Red is running in a container. It will see the "Alpine Linux" answer. Docker doesn't provide any mechanism for a container to execute an arbitrary command outside of its container.

A workaround is to utilise SSH. This gist explains how to set up the SSH scaffolding so that "exec" nodes running in a Node-Red container can invoke arbitrary commands outside container-space.

Task Goal

Be able to use a Node-Red exec node to perform the equivalent of:

$ ssh «HOSTNAME» «COMMAND»

where:

«HOSTNAME» is any host (not just the Raspberry Pi running IOTstack); and
«COMMAND» is any command known to the target host.

Assumptions

SensorsIot/IOTstack is installed on your Raspberry Pi.
You have run the menu at least once, have chosen Node-Red and selected at least one node to be installed when Node-Red is built.
The Node-Red container is running.

These instructions are specific to IOTstack but the underlying concepts should apply to any installation of Node-Red in a Docker container.

Executing commands "inside" a container

These instructions make frequent use of the ability to run commands "inside" the Node-Red container. For example, suppose you want to execute:

$ grep "^PRETTY_NAME=" /etc/os-release

You have several options:

You can do it from the normal Raspberry Pi command line using a Docker command. The basic syntax is:
```
$ docker exec {-it} «containerName» «command and parameters»
```
The actual command you would need would be:
```
$ docker exec nodered grep "^PRETTY_NAME=" /etc/os-release
```
Note:
- The -it flag is optional. It means "interactive terminal". Its presence tells Docker that the command may need user interaction, such as entering a password or typing "yes" to a question.
You can open a shell into the container, run as many commands as you like inside the container, and then exit. For example:
```
$ docker exec -it nodered bash
# grep "^PRETTY_NAME=" /etc/os-release
# whoami
# exit
$
```
In words:
- Run the bash shell inside the Node-Red container. You need to be able to interact with the shell to type commands so the -it flag is needed.
- The "#" prompt is coming from bash running inside the container. It also signals that you are running as the root user inside the container.
- You run the grep, whoami and any other commands.
- You finish with the exit command (or Control+D).
- The "$" prompt means you have left the container and are back at the normal Raspberry Pi command line.
Run the command from Portainer by selecting the container, then clicking the ">_ console" link. This is identical to opening a shell.

Variable definitions

You will need to have a few concepts clear in your mind before you can set up SSH successfully. I use double-angle quote marks (guillemets) to mean "substitute the appropriate value here".

«HOSTNAME» (required)

The name of your Raspberry Pi. When you first booted your RPi, it had the name "raspberrypi" but you probably changed it using raspi-config. Example:

iot-dev

«HOSTADDR» (required)

Either or both of the following:

«HOSTFQDN» (optional)

If you have a local Domain Name System server, you may have defined a fully-qualified domain name (FQDN) for your Raspberry Pi. Example:

iot-dev.mydomain.com

Note that Docker's internal networks do not support multicast traffic. You can't use a multicast DNS name (eg "raspberrypi.local") as a substitute for a fully-qualified domain name.
«HOSTIP» (required)

Even if you don't have a fully-qualified domain name, you will still have an IP address for your Raspberry Pi. Example:

192.168.132.9

Keep in mind that a Raspberry Pi running IOTstack is operating as a server. A dynamic DHCP address is not appropriate for a server. The server's IP address needs to be fixed. The two standard approaches are:
- a static DHCP assignment configured on your DHCP server (eg your router) which always returns the same IP address for a given MAC address; or
- a static IP address configured on your Raspberry Pi.

«USERID» (required)

The user ID of the account on «HOSTNAME» where you want Node-Red flows to be able to run commands. Example:

pi

Step 1: Define a volumes mapping for SSH

Move into the IOTstack folder:

$ cd ~/IOTstack

Open docker-compose.yml in your favourite text editor. Find the "nodered" fragment and add this line to its "volumes" list:

      - ./volumes/nodered/ssh:/root/.ssh

The result should be something like this:

    volumes:
      - ./volumes/nodered/data:/data
      - ./volumes/nodered/ssh:/root/.ssh
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket

Note:

if you routinely use the menu to manage your IOTstack, you will have to think about how to make this change permanent (eg compose-override.yml).

Save the file then execute the following command:

$ docker-compose up -d nodered

This causes the Node-Red container to be rebuilt and the new folder mapping to be established. At this point, the new folder will be empty.

Step 2: Generate SSH key-pair for Node-Red (one time)

Create a key-pair for Node-Red. This is done by executing a command inside the container:

$ docker exec -it nodered ssh-keygen -t ed25519

Respond to every prompt by pressing return.

Note:

I'm using the "ed25519" elliptic curve algorithm to generate the key-pair. You can use the default RSA algorithm if you prefer it.

Step 3: Exchange keys with target hosts (once per target host)

Node-Red's public key needs to be copied to the user account on each target machine where you want a Node-Red "exec" node to be able to execute commands. At the same time, the Node-Red container needs to learn the public host key of the target machine. The ssh-copy-id command does both steps. The required syntax is:

$ docker exec -it nodered ssh-copy-id «USERID»@«HOSTADDR»

Examples:

 $ docker exec -it nodered ssh-copy-id [email protected]
 $ docker exec -it nodered ssh-copy-id [email protected]

The output will be something similar to the following:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_ed25519.pub"
The authenticity of host 'iot-dev.mydomain.com (192.168.132.9)' can't be established.
ED25519 key fingerprint is SHA256:HVoeowZ1WTSG0qggNsnGwDA6acCd/JfVLZsNUv4hjNg.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Respond to the prompt by typing "yes" and press return.

The output continues:

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
expr: warning: '^ERROR: ': using '^' as the first character
of a basic regular expression is not portable; it is ignored
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:

The response may look like it contains errors but those can be ignored.

Enter the password you use to login as «USERID» on «HOSTADDR» and press return.

Normal completion looks similar to this:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

If you do not see an indication that a key has been added, you may need to retrace your steps.

Step 4: Perform the recommended test

The output above recommends a test. The test needs to be run inside the Node-Red container so the syntax is:

$ docker exec -it nodered ssh «USERID»@«HOSTADDR» ls -1 /home/pi/IOTstack

Examples:

 $ docker exec -it nodered ssh [email protected] ls -1 /home/pi/IOTstack
 $ docker exec -it nodered ssh [email protected] ls -1 /home/pi/IOTstack

You should not be prompted for a password. If you are, you may need to retrace your steps.

If everything works as expected, you should see a list of the files in your IOTstack folder.

Assuming success, think about what just happened? You told SSH inside the Node-Red container to run the ls command outside the container on your Raspberry Pi.

Understanding what's where and what each file does

What files are where

Six files are relevant to Node-Red's ability to execute commands outside of container-space:

in /etc/ssh:
- ssh_host_ed25519_key is the Raspberry Pi's private host key
- ssh_host_ed25519_key.pub is the Raspberry Pi's public host key
  
  Those keys were created when your Raspberry Pi was initialised. They are unique to the host.
  
  Unless you take precautions, those keys will change whenever your Raspberry Pi is rebuilt from scratch and that will stop SSH from working.
in ~/IOTstack/volumes/nodered/ssh:
- id_ed25519 is the Node-Red container's private key
- id_ed25519.pub is the Node-Red container's public key
  
  Those keys were created when you generated the SSH key-pair for Node-Red.
  
  They are unique to Node-Red but will follow the container in backups and will work on the same machine, or other machines, if you restore the backup.
  
  It does not matter if the Node-Red container is rebuilt or if a new version of Node-Red comes down from DockerHub. These keys will remain valid until lost or overwritten.
  
  If you lose these keys or overwrite them by re-running ssh-keygen as above, SSH will stop working.
- known_hosts
  
  The known_hosts file contains a copy of the Raspberry Pi's public host key. It was put there by ssh-copy-id.
  
  If you lose this file or it gets overwritten, SSH will still work but will re-prompt for authorisation to connect. This works when you are running commands from docker exec -it but not when running commands from an exec node.
  
  Note that authorising the connection at the command line ("Are you sure you want to continue connecting?") will auto-repair the known_hosts file.
in ~/.ssh/:
- authorized_keys
  
  That file contains a copy of the Node-Red container's public key. It was put there by ssh-copy-id.
  
  Pay attention to the path. It implies that there is one authorized_keys file per user, per target host.
  
  If you lose this file or it gets overwritten, SSH will still work but will ask for the password for «USERID». This works when you are running commands from docker exec -it but not when running commands from an exec node.
  
  Note that providing the correct password at the command line will auto-repair the authorized_keys file.

What each file does

SSH running inside the Node-Red container uses the Node-Red container's private key to provide assurance to SSH running outside the container that it (the Node-Red container) is who it claims to be.

SSH running outside container-space verifies that assurance by using its copy of the Node-Red container's public key in authorized_keys.

SSH running outside container-space uses the Raspberry Pi's private host key to provide assurance to SSH running inside the Node-Red container that it (the RPi) is who it claims to be.

SSH running inside the Node-Red container verifies that assurance by using its copy of the Raspberry Pi's public host key stored in known_hosts.

Config file (optional)

You don't have to do this step but it will simplify your exec node commands and reduce your maintenance problems if you do.

At this point, SSH commands can be executed from inside the container using this syntax:

# ssh «USERID»@«HOSTADDR» «COMMAND»

However, the task goal is the simpler syntax, and that needs a config file:

# ssh «HOSTNAME» «COMMAND»

A config file does not just simplify connection commands. It provides isolation between the «HOSTNAME» and «HOSTADDR» such that you only have a single file to change if your «HOSTADDR» changes (eg new IP address or fully qualified domain name). It also exposes less about your network infrastructure when you share your flows.

The goal is to set up this file:

-rw-r--r-- 1 root root ~/IOTstack/volumes/nodered/ssh/config

The file needs the ownership and permissions shown. There are several ways of going about this and you are free to choose the one that works for you. The method described here creates the file first, then sets correct ownership and permissions, and then moves the file into place.

Start in a directory where you can create a file without needing sudo. The IOTstack folder is just as good as anywhere else:

$ cd ~/IOTstack
$ touch config

Select the following text, copy it to the clipboard.

host «HOSTNAME»
  hostname «HOSTADDR»
  user «USERID»
  IdentitiesOnly yes
  IdentityFile /root/.ssh/id_ed25519

Open ~/IOTstack/config in your favourite text editor and paste the contents of the clipboard.

Replace the «delimited» keys. Completed examples:

If you are using the «HOSTFQDN» form:

 host iot-dev
   hostname iot-dev.mydomain.com
   user pi
   IdentitiesOnly yes
   IdentityFile /root/.ssh/id_ed25519

If you are using the «HOSTIP» form:

 host iot-dev
   hostname 192.168.132.9
   user pi
   IdentitiesOnly yes
   IdentityFile /root/.ssh/id_ed25519

Save the file.

Change the config file's ownership and permissions, and move it into the correct directory:

$ chmod 644 config
$ sudo chown root:root config
$ sudo mv config ./volumes/nodered/ssh

Re-test with config file in place

The previous test used this syntax:

$ docker exec nodered ssh «USERID»@«HOSTADDR» ls -1 /home/pi/IOTstack

Now that the config file is in place, the syntax changes to:

$ docker exec nodered ssh «HOSTNAME» ls -1 /home/pi/IOTstack

Example:

 $ docker exec nodered ssh iot-dev ls -1 /home/pi/IOTstack

The result should be the same as the earlier test.

A test flow

In the Node-Red GUI:

Click the "+" to create a new, empty flow.
Drag the following nodes onto the canvas:
- One "inject" node
- Two "exec" nodes
- Two "debug" nodes
Wire the outlet of the "inject" node to the inlet of both "exec" nodes.
Wire the uppermost "stdout" outlet of the first "exec" node to the inlet of the first "debug" node.
Repeat step 4 with the other "exec" and "debug" node.
Open the first "exec" node and:
- set the "command" field to:
```
 grep "^PRETTY_NAME=" /etc/os-release
```
- turn off the "append msg.payload" checkbox
- set the timeout to a reasonable value (eg 10 seconds)
- click "Done".
Repeat step 6 with the other "exec" node, with one difference:
- set the "command" field to:
```
 ssh iot-dev grep "^PRETTY_NAME=" /etc/os-release
```
Click the Deploy button.
Set the right hand panel to display debug messages.
Click the touch panel of the "inject" node to trigger the flow.
Inspect the result in the debug panel. You should see payload differences similar to the following:
```
PRETTY_NAME="Alpine Linux v3.11"
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
```
The first line is the result of running the command inside the Node-Red container. The second line is the result of running the same command outside the Node-Red container on the Raspberry Pi.

Suppose you want to add another «HOSTNAME»

Exchange keys with the new target host using:

$ docker exec -it nodered ssh-copy-id «USERID»@«HOSTADDR»

Edit the config file at the path:
```
~/IOTstack/volumes/nodered/ssh/config
```
to define the new host. Remember to use sudo to edit the file. There is no need to restart Node-Red or recreate the container.

877dev/gist:fa67ee47341a0a04d2253598f32a9474 Secret