[EdgeX Foundry] Export to Google IoT Core using app-service-configurable (insecure mode)

EdgeX Foundry Geneva / Hanoi / Ireland release
- Using edgexfoundry/docker-app-service-configurable:1.3.1 for Geneva / Hanoi
- Using edgexfoundry/app-service-configurable:2.0.1 for Ireland
Insecure mode (without Secret Store)

Disclaimer
Prepare key and certs to authenticate your EdgeX Foundry
Prepare Google IoT Core
Prepare JWT
Prepare EdgeX Foundry
- Geneva / Hanoi (1.x)
- Ireland (2.x)
Verify your data is available on IoT Core

Disclaimer

JWT is required to authenticate with Google IoT Core, and since there is no mechanism in the current app-service-configurable to automatically renew JWT, it is impossible to use this for a long time if you use app-service-configurable as is.

Prepare key and certs to authenticate your EdgeX Foundry

Create work directory.

mkdir certs
cd certs

Create an RSA key with a self-signed X.509 certificate.

openssl req -x509 -nodes -newkey rsa:2048 -keyout rsa_private.pem -out rsa_cert.pem -subj "/CN=unused" -days 3650

Download minilal root CA set and convert it to PEM format.

curl -O https://pki.goog/gtsltsr/gtsltsr.crt
curl -O https://pki.goog/gsr4/GSR4.crt
openssl x509 -in gtsltsr.crt -inform DER -out gtsltsr.pem -outform PEM
openssl x509 -in GSR4.crt -inform DER -out GSR4.pem -outform PEM
cat gtsltsr.pem GSR4.pem > rootca.pem

Now you have rsa_cert.pem, rsa_private.pem, and rootca.crt at least.

$ ls -l
-rw-rw-r-- 1 kuro kuro 1389 Oct 16 07:36 rootca.pem
-rw-rw-r-- 1 kuro kuro 1107 Oct 16 07:32 rsa_cert.pem
-rw------- 1 kuro kuro 1704 Oct 16 07:32 rsa_private.pem

Prepare Google IoT Core

Open [BIG DATA] > [IoT Core] and [ENABLE] API if disabled
Create new registry
- Click [CREATE REGISTRY]
- Enter any [Registry ID] and [Region]
- Click [Select a Cloud Pub/Sub topic] > [CREATE A TOPIC], then enter any [Topic ID] and click [CREATE TOPIC]
- Click [CREATE] to create registry
Register your EdgeX Foundry as "Device"
- Open [Devices] and click [CREATE A DEVICE]
- Open [COMMUNICATION, CLOUD LOGGING, AUTHENTICATION] and move to [Authentication (optional)] section
- Select [RS256_X509], then paste your rsa_cert.pem
- Click [CREATE] to create device

Prepare JWT

Note that JWT expire after a short time, and since there is no mechanism in the current app-service-configurable to automatically renew JWT, it is impossible to use this for a long time if you use app-service-configurable as is.

The Python script is based on the official sample.

# Prepare Python
python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install -U pip
python3 -m pip install pyjwt==2.2.0 cryptography

# Create minimal helper script
cat <<EOF > create_jwt.py
import datetime, os, jwt

def create_jwt(project_id, private_key_file, algorithm):
    token = {
        "iat": datetime.datetime.utcnow(),
        "exp": datetime.datetime.utcnow() + datetime.timedelta(hours=23),
        "aud": project_id,
    }
    with open(private_key_file, "r") as f:
        private_key = f.read()
    return jwt.encode(token, private_key, algorithm=algorithm)

if __name__ == "__main__":
    project_id = os.environ["GOOGLE_IOT_PROJECT"]
    private_key_file = os.environ["GOOGLE_IOT_PRIVATE_KEY_FILE"]
    algorithm = os.environ["GOOGLE_IOT_ALGORITHM"]
    print(create_jwt(project_id, private_key_file, algorithm))
EOF

# Create JWT
export GOOGLE_IOT_PROJECT=<YOUR_PROJECT_ID>
export GOOGLE_IOT_PRIVATE_KEY_FILE=rsa_private.pem 
export GOOGLE_IOT_ALGORITHM=RS256
python3 create_jwt.py

Keep your JWT to use later.

Prepare EdgeX Foundry

Geneva / Hanoi (1.x)

This is the snippet for app-service-configurable for your docker-compose.yml. Note that the name of environment variable is case sensitive.

See the comments for details.

  app-service-mqtt:
    # It is recommended that you use the images from the Hanoi release for Geneva too
    # as the issue have been fixed.
    # https://github.com/edgexfoundry/go-mod-bootstrap/issues/111
    image: edgexfoundry/docker-app-service-configurable:1.3.1
    ports:
      - "127.0.0.1:48097:48097"
    container_name: edgex-app-service-configurable-mqtt
    hostname: edgex-app-service-configurable-mqtt
    networks:
      - edgex-network
    environment:
      <<: *common-variables
      edgex_profile: mqtt-export
      Service_Host: edgex-app-service-configurable-mqtt
      Service_Port: 48097
      MessageBus_SubscribeHost_Host: edgex-core-data
      Binding_PublishTopic: events

      # Requied to use MQTTSecretSend
      Writable_Pipeline_ExecutionOrder: "TransformToJSON, MQTTSecretSend, MarkAsPushed"

      # Use LTS domain
      # https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#using_a_long-term_mqtt_domain
      Writable_Pipeline_Functions_MQTTSecretSend_Parameters_brokeraddress: tcps://mqtt.2030.ltsapis.goog:8883

      # Topic has to be formatted as "/devices/DEVICE_ID/events"
      # https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#publishing_telemetry_events
      Writable_Pipeline_Functions_MQTTSecretSend_Parameters_topic: /devices/edgex-foundry/events

      # Client ID has to be formatted as "projects/PROJECT_ID/locations/REGION/registries/REGISTRY_ID/devices/DEVICE_ID"
      # https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#configuring_mqtt_clients
      Writable_Pipeline_Functions_MQTTSecretSend_Parameters_clientid: projects/iot-project-000000/locations/asia-east1/registries/iot-registry/devices/edgex-foundry

      # Auth with username and password (JWT)
      Writable_Pipeline_Functions_MQTTSecretSend_Parameters_authmode: usernamepassword

      # Skip verify to accept unknown authority
      Writable_Pipeline_Functions_MQTTSecretSend_Parameters_skipverify: "true"

      # Seems any value is ok, but respect the document
      # https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#configuring_mqtt_clients
      Writable_InsecureSecrets_mqtt_Secrets_username: unused

      # Paste your JWT. JWT can be generated using create_jwt.py
      Writable_InsecureSecrets_mqtt_Secrets_password: eyJ0eXAiOiJKV1Qi...G7zjg6YRVzoIYZqA

      # Contents from rootca.pem
      Writable_InsecureSecrets_mqtt_Secrets_cacert: |
        -----BEGIN CERTIFICATE-----
        MIIBxTCCAWugAwIBAgINAfD3nVndblD3QnNxUDAKBggqhkjOPQQDAjBEMQswCQYD
        VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzERMA8G
        A1UEAxMIR1RTIExUU1IwHhcNMTgxMTAxMDAwMDQyWhcNNDIxMTAxMDAwMDQyWjBE
        MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
        QzERMA8GA1UEAxMIR1RTIExUU1IwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATN
        8YyO2u+yCQoZdwAkUNv5c3dokfULfrA6QJgFV2XMuENtQZIG5HUOS6jFn8f0ySlV
        eORCxqFyjDJyRn86d+Iko0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw
        AwEB/zAdBgNVHQ4EFgQUPv7/zFLrvzQ+PfNA0OQlsV+4u1IwCgYIKoZIzj0EAwID
        SAAwRQIhAPKuf/VtBHqGw3TUwUIq7TfaExp3bH7bjCBmVXJupT9FAiBr0SmCtsuk
        miGgpajjf/gFigGM34F9021bCWs1MbL0SA==
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
        MIIB4TCCAYegAwIBAgIRKjikHJYKBN5CsiilC+g0mAIwCgYIKoZIzj0EAwIwUDEk
        MCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBDQSAtIFI0MRMwEQYDVQQKEwpH
        bG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTEyMTExMzAwMDAwMFoX
        DTM4MDExOTAzMTQwN1owUDEkMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBD
        QSAtIFI0MRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWdu
        MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEuMZ5049sJQ6fLjkZHAOkrprlOQcJ
        FspjsbmG+IpXwVfOQvpzofdlQv8ewQCybnMO/8ch5RikqtlxP6jUuc6MHaNCMEAw
        DgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFFSwe61F
        uOJAf/sKbvu+M8k8o4TVMAoGCCqGSM49BAMCA0gAMEUCIQDckqGgE6bPA7DmxCGX
        kPoUVy0D7O48027KqGx2vKLeuwIgJ6iFJzWbVsaj8kfSt24bAgAXqmemFZHe+pTs
        ewv4n4Q=
        -----END CERTIFICATE-----

    depends_on:
      - consul
      - data

Then start your EdgeX Foundry, and send some data from your device. For testing purpose, device-virtual can be used to send dummy data.

# Start EdgeX Foundry
$ docker-compose up -d
...

# Check logs after sending some data from your device
$ docker logs edgex-app-service-configurable-mqtt
...
level=INFO ts=2021-10-16T08:04:20.540954927Z app=AppService-mqtt-export source=mqttsecret.go:125 msg="Connecting to mqtt server for export"
level=INFO ts=2021-10-16T08:04:20.777252528Z app=AppService-mqtt-export source=mqttsecret.go:134 msg="Connected to mqtt server for export"

Ireland (2.x)

This is the snippet for app-service-configurable for your docker-compose.yml.