Lab: Connect an ESP32 to AWS IoT with MongooseOS

created 2019-12-20 by Noah Coad

Lab Overview

This lab walks you through connecting one of the most popular IoT chips, the Espressif ESP32, to AWS IoT Core. It includes loading firmware, MongooseOS RPC, AWS IoT Core connection, MQTT topics, Device Shadow, and an AWS IoT Rule.

What you need to know:

The Espressif ESP32 is one of the (if not the #1) leading IoT microcontroller.
A microcontroller is like a very very tiny computer, usually cheap (these are $3) that runs one program (no linux OS), and is designed to connect to other sensors, LEDs, etc.
The ESP32 chip has Bluetooth v4.2, WiFi, dual-core 240MHz, 18-ch 12-bit ADC, 2x 8-bit DAC, 10 touch sensors, 3x UARTs, 2x I2C, 4xSPI, CAN bus, 16x LED PWM, and much more
The dev board (pictured below) includes the ESP32, USB connector, two buttons (reset and user usable), two LEDs (power and user usable)
These dev boards are as little as $5/ea on ebay.com, the raw ESP32 chip without dev board is $3
There are versions of the ESP32, the ESP32-WROOM-32 is v2 chip
There are many OSes that can run on the ESP32, including Arduino (yes, program with Arudino software), Amazon FreeRTOS, MongooseOS, and many others
Ultra-low power consumption on deep sleep, only 10uA, which means it can be used to wake periodically and send like a temperature reading every hour for years on a small battery.
You can think of it as a really low cost much much more powerful Arduino.

Hardware to Get

I recommend this ESP32 dev board. It has the newer v2 hardware, two LEDs, and two buttons. One red LED is connected to power, and the other blue LED to Pin 2 so you can use it. And two buttons, one connected to EN (aka Enable), which resets the device, and a second button connected to Pin 0 that you can use. It costs $8 on amazon.com or $5 on ebay.com.

Connecting to AWS IoT

MongooseOS is fantastic operating system the runs on the ESP32, it's predicesor the ESP8266, and many other chips. It runs NodeJS programs on it, which is pretty slick. Does a great job exposing the power of the ESP32. Has an RPC (remote-procedure-call) that lets you remotely manage the OS via serial connection or even remotely via MQTT. Send commands to the chip OS like reboot, toggle pins, load files, configure wifi, etc. And it has built-in support for IoT major cloud providers (AWS, Azure, Google). There are many other great OS' like Amazon FreeRTOS and Arudino for ESP32. But for the purposes of this lab, we'll be using Mongoose due to it's simplicity.

Instructions are from Mac OSX. Most ESP32 boards have a "USB to Serial" chip on them. If the ls /dev/cu.* command below does not turn up a new port when you plug in your board, you may need to install an additional driver first. Try to find out what chip you ESP32 board is using. Here are some common ones: CH340G, SiLabs, FTDI

1. Prerequisits

Install these prerequisits first aws cli, jq, git, brew with these instructions.
Then also...

# also install MongooseOS and Mosquitto
brew install cesanta/mos/mos mosquitto

2. Load MongooseOS on ESP32

# before plugging the USB into the computer,
# run this command to see what ports are already on your computer
# then run again after plugging in to see the new port
ls /dev/cu.*

# set this to the port of the ESP32
export MOS_PORT=/dev/cu.SLAB_USBtoUART
export MOS_PLATFORM=esp32

# get a sample app
mos clone https://github.com/mongoose-os-apps/demo-js app1
cd app1

# build firmware and load onto chip
# the build command can take awhile
mos build
mos flash

3. Connect to AWS IoT

# configure wifi connection
mos wifi <your_ssid> <your_password>

# configure connection to AWS IoT
# this creates an AWS IoT Thing, device certificate, private/public keys,
# loads onto chip, and configures the chip to connect to AWS
mos aws-iot-setup

You're now connected to AWS IoT!

4. Test Connection via AWS Console

Go to the AWS IoT Console's "Test" page, "Subscription topic" enter "#" (without quotes) and click "Subscribe to topic" see the traffic coming into your account. Make sure you're viewing the same AWS region and account as your aws cli tool is configure to. Press the button on the ESP32 to see what happens. One of them will reset the ESP32, the other will send a button update message through AWS IoT Core.

5. Test Connection via MQTT

Now let's make an MQTT connection using the Mosquitto pub/sub clients. Mosquitto is an open source MQTT broker, and contains clients that can publish MQTT messages from the command line, or subscribe to an MQTT topic. This can be easier to use at times than going to the AWS console. This will connect to AWS IoT from the computer using the same certificate and key. We'll subscribe to an AWS IoT MQTT topic to see the messages come through.

# see what this AWS IoT device ID and AWS IoT Endpoint we're using
export DEVICE_ID=$(mos config-get device.id)
export MQTT_SERVER=$(mos config-get mqtt.server | cut -d':' -f1)
echo Thing Name: $DEVICE_ID\\nAWS IoT Endpoint: $MQTT_SERVER

# get the root CA that authenticates
# the AWS IoT cloud off of the ESP32
mos get ca.pem | tee ca.pem

# subsctibe to the # topic
mosquitto_sub -v --cert aws-${DEVICE_ID}.crt.pem --key aws-${DEVICE_ID}.key.pem --cafile ca.pem -h $MQTT_SERVER -p 8883 -t '#'

# you may see a lot of shadow update traffic, 
# so ctrl+c to close that, and run this one 
# to exclude shadow updated
# then press the ESP32 button to see those messages
mosquitto_sub -v --cert aws-${DEVICE_ID}.crt.pem --key aws-${DEVICE_ID}.key.pem --cafile ca.pem -h $MQTT_SERVER -p 8883 -t '#' -T "\$aws/things/${DEVICE_ID}/shadow/#"

6. Explore the MongooseOS RPC

MongooseOS supports sending commands directly to the OS on the chip. Call Remote Procedure Calls (RPC), see the Mongoose RPC doc.

# see all commands
mos call rpc.list

# try one, list the files on the ESP32
mos call fs.list

# get info
mos call sys.getinfo

# get all config
mos call config.get

# get a specific property
mos call config.get '{"key":"wifi.sta"}'

# a shortcut to the last command
mos call config-get wifi.sta

# on most ESP32 dev boards, there is a 2nd LED on pin 2
mos config-set board.led1.pin=2

# save config and reboot
mos call config.save '{"reboot": true}'

# confirm change
mos call config-get board.led1.pin

# now we can turn on/off the led on the board
mos call gpio.write '{"pin": 2, "value": 1}'
mos call gpio.write '{"pin": 2, "value": 0}'
mos call gpio.toggle '{"pin": 2}'

# see if the button is being held down
# "value": 0 = button is down
mos call gpio.read '{"pin": 0}'

7. Use RPC over MQTT

Previously all those RPC commands were being sent over the USB serial connection to the ESP32. But they can also be run remotely through AWS MQTT. Let's try that. For full effect, disconnect the ESP32 from the computer, and power it from a mobile phone backup battery or wall charger.

# create an alias to make follow commands shorter
# calling it 'rmos' for 'remote mos'
alias rmos='mos --cert-file aws-${DEVICE_ID}.crt.pem --key-file aws-${DEVICE_ID}.key.pem --port mqtts://${MQTT_SERVER}/${DEVICE_ID}'

# try the commmands again
rmos config-get mqtt
rmos call gpio.toggle '{"pin": 2}'
rmos get init.js

As an extra, run the last mosquitto_sub command above and watch the RPC traffic over MQTT. Remember you'll need to be in the app1 folder also set the DEVICE_ID and MQTT_SERVER environment variables again if you open a new terminal window.

8. Make the ESP32 button toggle the ESP32 LED via AWS IoT

Now lets connect the things we've done so far. Make it such that when you push a button, the LED toggles. This requires an ESP32 dev board with two LEDs, like red power and blue user, such as the board linked to from the top of this lab. And requires the previous lab steps that load the code on to the ESP32 and configure the LED pin to pin 2.

Note that your device ID Thing Name is going to be different than the example here...

On each button push, the sample code is publishing on the MQTT topic devices/esp32_61F6C4/events a message like {"ram_free":171552,"uptime":883.956661,"btnCount":9,"on":false}.
So we're going to create an AWS IoT Rule that watches for this message, and then sends an RPC command back down to the device to toggle.
The RPC MQTT command is on the topic esp32_61F6C4/rpc and looks like this {"src":"mos-1576885511","id":1998557816377,"method":"gpio.toggle","params":{"pin":2}}. See terminal image above. However the src and id fields are not required.
Notice that the AWS Thing Name as represented by ${DEVICE_ID} here, needs to be in the MQTT topic.

Let's test by publishing an toggle pin MQTT messages to the topic for the device

mosquitto_pub --cert aws-${DEVICE_ID}.crt.pem --key aws-${DEVICE_ID}.key.pem --cafile ca.pem -h $MQTT_SERVER -p 8883 -t "${DEVICE_ID}/rpc" -m '{"method":"gpio.toggle","params":{"pin":2}}'

If that works, we're now ready to create our AWS IoT Rule and a Lambda

Create a lambda called esp32_toggle_led with python 3.6+ and this code:

import boto3

client = boto3.client('iot-data')

def lambda_handler(event, context):
	client.publish(topic=event['devid'] + '/rpc', qos=1, 
	    payload='{"method":"GPIO.Toggle","params":{"pin":2}}')

Edit the lambda's role policy > Permissions section > "Manage these permissions" > click "Attach policies" > and choose "AWSIoTFullAccess"

Save the lambda. Now let's create the AWS IoT Rule that will call this lambda.

Got to AWS IoT Console, choose "Act" from the left menu > Rules > Create
Give it a name of esp32_toggle_led
Change the SQL like SELECT to SELECT topic(2) as devid, * FROM 'devices/+/events'
Choose "Add action" and select the new lambda
Click "Create rule" at the bottom

Now try pressing the button on the ESP32 and see if the LED toggles!

To debug, try watching the MQTT traffic and looking at the lambda cloudwatch logs.

9. Use Device Shadow Instead

Instead of sending a hardware pin toggle command to the underlying MongooseOS, the app running on MongooseOS (see the file in app/fs/init.js) is watching the device shadow, and will change the LED based on the device shadow state. This is a more robust and sustainable option.

The Device Shadow is a JSON document that's kept in both AWS IoT, and on the device. When a change is made, an MQTT message is sent, so that they two JSON docs can be kept in sync. This would let you set the cloud value seperate from the device, and when the device connects, it asks the cloud to see if there've been any updates to the shadow in the cloud, and if so, pulls down the latest update. This is useful for keeping state about the device in sync between the device and the cloud.

Change the lambda code to this:

import boto3, json

client = boto3.client('iot-data')

def lambda_handler(event, context):
	# read the current shadow state
	shadow = json.loads(client.get_thing_shadow(
		thingName=event['devid'])['payload'].read().decode())
	
	# new state should toggle LED state
	new_state = {'state':{'desired':{'on': not shadow['state']['reported']['on'] }}}
	
	# send the update
	client.update_thing_shadow(thingName=event['devid'], payload=json.dumps(new_state))

10. Explore the Device Code

If you look in the app1/fs/init.js file you'll see the code for the app. Or you can pull the code right off the devicePlug back into the computer for these next steps, unless you want to use the rmos command from above.

# download the init.js file right off the ESP32
mos get init.js | tee init.js

# edit in your favorite editor
vim init.js

# put back on the ESP32
mos put init.js

# reboot
mos call sys.reboot

11. Completed!

You've completed this lab! Congrats! Please provide feedback =)

CloudFormation

To create the resources above, use this CF stack

Feedback

Please provide feedback on this lab! Thanks!

noahcoad/lab_esp32_aws_iot_mongoose.md