This is a custom PicoC program to parse the SML energy readings in a tibber+Loxone setup.
Our system runs a tibber Pulse, connected to a EMH ED300L energy meter. We run this code on a Loxone Miniserver, which can run custom PicoC code.
We use the TCP stream method to read the current SML data packet from the tibber bridge
On macOS:
brew install picoc
Note that the picoc version used in homebrew has some weird quirks. For some tests, I've used this:
git clone [email protected]:zsaleeba/picoc.git
cd picoc
make
ln -s $(pwd)/picoc /usr/local/bin/picocThis has other quirks, but at least I've been able to fix the code.
On other systems: Install PicoC as you would install any other software.
Now, you can parse the example message:
./run.shThis runs the example message and checks that the dummy setoutput method was called correctly.
./test.shYou'll have to do this process whenever you encounter an energy meter with a different number/structure of reported values (f.ex. single-direction meter or dual for grid feed-in and consumption etc).
- Run curl to download an example SML message from your energy meter:
Be sure to replace the username, password, and IP address.curl "http://admin:[email protected]/data.json?node_id=1" -D - > sml_message.bin
- Run this to convert the file to a hex dump into example.h.
xxd -i sml_message.bin | awk '/,$/{printf $0;next} {print}' >> example.h
- Open example.h in a text editor, and make sure the format matches.
- Remove/comment out the old
example_messageconstants - Rename the variables to
example_messageandexample_message_len - Add the length in bytes (from
example_message_len) into the[]brackets afterexample_message.
- Remove/comment out the old
- Run
picoc -s reader.cto run the extraction once so you can read out the addresses.- Press Ctrl+C to abort the loop
- Search for these snippets in the output:
They might not be sorted.
-> output 2 is at offset 0x00b3, len 5 -> output 1 is at offset 0x00c8, len 5 -> output 0 is at offset 0x011c, len 4
- Open
quick.c, and replace the constantsoutput_num,output_offsets, andoutput_lengthswith the values you've just learned (after sorting them).
- Update configuration in
quick.c:HTTP_IPis the IP of your tibber bridge.- Set
output_num,output_offsets, andoutput_lengthsto the message offsets you desire (See Finding new addresses above.)
Note that you can't use newlines in arrays.
- Copy the script to a Loxone custom script block.
- Set inputs as directed in the top of quick.c
- Text input 1: base64-coded basicauth
username:password(without newline) for your tibber bridge:echo -n "admin:1234-ABCD" | base64 # output: YWRtaW46MTIzNC1BQkNE
- Numeric input 1: Interval (in seconds) to wait between updates
- Text input 1: base64-coded basicauth
- Capture outputs:
- Add a tracker for text output 1.
This will contain errors (as long as they persist) - Add virtual outputs for numeric outputs 1-n.
These will contain your desired readings.
- Add a tracker for text output 1.
To help development and understand the SML format, there's some helper files:
sml_message.binwhich contains a raw SML binary messagerun_perl.shwhich runssmldump.plon that messagerun_python.shwhich runssmllibon that message
To run this, you first need to runpip install smllib
First byte is Type & length. Length -1 in most cases.
Types (first nibble & 0x70):
- 0 bytearray
- 5 signed
- 6 unsigned
- 7 list (length not -1)
0x1b, 0x1b, 0x1b, 0x1b, SML Magic Number
0x01, 0x01, 0x01, 0x01, Version 1
0x76, List with 6 entries - SmlMessage
1 0x07 (transaction id): 0x00, 0x13, 0x00, 0x16, 0xc7, 0xa9,
2 0x62: 0x00 Unsigned 1 byte (k & 0xf - 1 -> 1 byte)
3 0x62: 0x00 Unsigned 1 byte
4 0x72, List with 2 entries
1 0x63: 0x01, 0x01 Unsigned 2 bytes
2 0x76: list 6 entries
1 0x01: 0 bytes
2 0x01: 0 bytes
3 0x07: 6 bytes: 0x00, 0x13, 0x04, 0xf0, 0x97, 0xe3
4 0x0b: 10 bytes bytes: <snip>
5 0x01: 0 bytes,
6 0x01: 0 bytes,
5 0x63: unsigned 2: 0x98, 0x8b
6 0x00 ?
0x76: list 6 - SmlMessage
1 0x07: 6 bytes 0x00, 0x13, 0x00, 0x16, 0xc7, 0xaa
2 0x62 unsigned 1b: 0x00
3 0x62: 0x00
4 0x72: list 2 - SmlGetListResponse
1 0x63: 0x07, 0x01 unsigned 2
2 0x77 list 7:
1 0x01
2 0x0b: 10 bytes: <snip>
3 0x07: 6 bytes: 0x01, 0x00, 0x62, 0x0a, 0xff, 0xff
4 0x72: list 2
1 0x62: 0x01
2 0x65: 0x04, 0xf0, 0x73, 0x85 current sensor time
0x77 list 7
1 0x77 list 7
1 0x07: 6b 0x81, 0x81, 0xc7, 0x82, 0x03, 0xff OBIS "Hersteller-Identifikation"
2 0x01
3 0x01
4 0x01
5 0x01
6 0x04: 0x45, 0x4d, 0x48 ("EMH")
7 0x01
0x77: list 7
1 0x07: 0x01, 0x00, 0x00, 0x00, 0x09, 0xff OBIS "Server-Id / Geraeteeinzelidentifikation"
2 0x01
3 0x01
4 0x01
5 0x01
6 0x0b: <snip>
7 0x01
0x77: list 7
1 0x07: 0x01, 0x00, 0x01, 0x08, 0x00, 0xff OBIS "Zaehlwerk neg. Wirkenergie (Einspeisung), tariflos"
2 0x64: 0x00, 0x01, 0x82
3 0x01
4 0x62: 0x1e -> 30 -> Wh
5 0x52: 0xff -> scale E-1
6 0x56, 0x00, 0x00, 0xe6, 0x3b, 0x5f -> 1508847.9 Wh
7 0x01
... The rest is up to the reader
(These are taken from smldump.pl)
0x00, 0x00, 0x60, 0x01, 0xFF, 0xFF: Seriennummer0x01, 0x00, 0x00, 0x00, 0x09, 0xFF: Server-Id / Geraeteeinzelidentifikation0x01, 0x00, 0x01, 0x08, 0x00, 0xFF: Zaehlwerk pos. Wirkenergie (Bezug), tariflos0x01, 0x00, 0x01, 0x08, 0x01, 0xFF: Zaehlwerk pos. Wirkenergie (Bezug), Tarif 10x01, 0x00, 0x01, 0x08, 0x02, 0xFF: Zaehlwerk pos. Wirkenergie (Bezug), Tarif 20x01, 0x00, 0x02, 0x08, 0x00, 0xFF: Zaehlwerk neg. Wirkenergie (Einspeisung), tariflos0x01, 0x00, 0x02, 0x08, 0x01, 0xFF: Zaehlwerk neg. Wirkenergie (Einspeisung), Tarif 1 (Bezug)0x01, 0x00, 0x02, 0x08, 0x02, 0xFF: Zaehlwerk neg. Wirkenergie (Einspeisung), Tarif 2 (Einspeisung)0x01, 0x00, 0x0F, 0x07, 0x00, 0xFF: Betrag der aktuellen Wirkleistung0x01, 0x00, 0x10, 0x07, 0x00, 0xFF: Aktuelle Wirkleistung gesamt0x01, 0x00, 0x24, 0x07, 0x00, 0xFF: Aktuelle Wirkleistung L10x01, 0x00, 0x38, 0x07, 0x00, 0xFF: Aktuelle Wirkleistung L20x01, 0x00, 0x4c, 0x07, 0x00, 0xFF: Aktuelle Wirkleistung L30x81, 0x81, 0xC7, 0x82, 0x03, 0xFF: Hersteller-Identifikation0x81, 0x81, 0xC7, 0x82, 0x05, 0xFF: Public Key
➜ ./run.sh
OPEN /dev/tcp/10.0.1.134/80
WRITE GET /data.json?node_id=1 HTTP/1.0
Authorization: Basic <snip>
(80)
READ(128, 5000) -> 128 (408 remaining)
READ(128, 5000) -> 128 (280 remaining)
READ(128, 5000) -> 128 (152 remaining)
READ(128, 5000) -> 24 (24 remaining)
READ(128, 5000) -> 0 (0 remaining)
total bytes: 408
Headers are correct.
76: list 6:
07: <snip> bytes (6)
62: 00 unsigned 1: 0
62: 00 unsigned 1: 0
72: list 2:
63: 01 01 unsigned 2: 257
76: list 6:
01: bytes (0)
01: bytes (0)
07: <snip> bytes (6)
0b: <snip> bytes (10)
01: bytes (0)
01: bytes (0)
63: 9d 18 unsigned 2: 40216
00: bytes (-1)
76: list 6:
07: <snip> bytes (6)
62: 00 unsigned 1: 0
62: 00 unsigned 1: 0
72: list 2:
63: 07 01 unsigned 2: 1793
77: list 7:
01: bytes (0)
0b: <snip> bytes (10)
07: <snip> bytes (6)
72: list 2:
62: 01 unsigned 1: 1
65: 04 f4 5c 4d unsigned 4: 83123277
77: list 7:
77: list 7:
07: 81 81 c7 82 03 ff bytes (6) <- OBIS
01: bytes (0)
01: bytes (0)
01: bytes (0) <- unit
01: bytes (0) <- scale (0)
04: 45 4d 48 bytes (3) scale 0.000000 <- value = 0.000000 ?
01: bytes (0)
77: list 7:
07: 01 00 00 00 09 ff bytes (6) <- OBIS
01: bytes (0)
01: bytes (0)
01: bytes (0) <- unit
01: bytes (0) <- scale (0)
0b: <snip> bytes (10) scale 0.000000 <- value = 0.000000 ?
01: bytes (0)
77: list 7:
07: 01 00 01 08 00 ff bytes (6) <- OBIS
64: 00 01 82 unsigned 3: 386
01: bytes (0)
62: 1e unsigned 1: 30 <- unit
52: ff signed 1: -1 <- scale (-1)
56: 00 00 e8 3a 6d signed 5: 15219309 scale -1.000000 <- value = 1521930.900000 Wh
01: bytes (0)
77: list 7:
07: 01 00 01 08 01 ff bytes (6) <- OBIS
01: bytes (0)
01: bytes (0)
62: 1e unsigned 1: 30 <- unit
52: ff signed 1: -1 <- scale (-1)
56: 00 00 e8 3a 6d signed 5: 15219309 scale -1.000000 <- value = 1521930.900000 Wh -> output 1
SETTING OUTPUT 1 to 1521930.900000
01: bytes (0)
77: list 7:
07: 01 00 01 08 02 ff bytes (6) <- OBIS
01: bytes (0)
01: bytes (0)
62: 1e unsigned 1: 30 <- unit
52: ff signed 1: -1 <- scale (-1)
56: 00 00 00 00 00 signed 5: 0 scale -1.000000 <- value = 0.000000 Wh -> output 2
SETTING OUTPUT 2 to 0.000000
01: bytes (0)
77: list 7:
07: 01 00 10 07 00 ff bytes (6) <- OBIS
01: bytes (0)
01: bytes (0)
62: 1b unsigned 1: 27 <- unit
52: ff signed 1: -1 <- scale (-1)
55: 00 00 03 ee signed 4: 1006 scale -1.000000 <- value = 100.600000 W -> output 0
SETTING OUTPUT 0 to 100.600000
01: bytes (0)
77: list 7:
07: 81 81 c7 82 05 ff bytes (6) <- OBIS
01: bytes (0)
01: bytes (0)
01: bytes (0) <- unit
01: bytes (0) <- scale (0)
83+02: ba df 00 d0 00 00 00 00 00 00 00 00 00 00 00 00 bytes (48) scale 0.000000 <- value = 0.000000 ?
01: bytes (0)
01: bytes (0)
01: bytes (0)
63: 88 49 unsigned 2: 34889
00: bytes (-1)
76: list 6:
07: <snip> bytes (6)
62: 00 unsigned 1: 0
62: 00 unsigned 1: 0
72: list 2:
63: 02 01 unsigned 2: 513
71: list 1:
01: bytes (0)
63: a9 b9 unsigned 2: 43449
00: bytes (-1)
00: bytes (-1)
00: bytes (-1)
end of message
crc actual: 0x78a0, swap endian: 0xffffa078, expected: 0xffffa078
bytes (-1)
Notice the stray output of bytes (-1) at the end -- only the PicoC version in the homebrew repos does that.
gitlab.com/zsaleeba/picoc and clang don't.
Here's a list of quirks and undocumented behavior I've found while developing this program:
On the Loxone Miniserver especially:
- It's slooooow
- Don't try to run CRC on it or other fancy stuff
- One round of
reader.ctakes about 15s on my test device, and only 2-3s of that is the HTTP request
- stream_read() takes a timeout value, and will take exactly that long to complete.
- So prefer using shorter timeouts, and loop a while until you have the data you want.
- PicoC is running in script mode, so your
main()will not automatically be executed unless you add a linemain();at the bottom (a.k.a.picoc -s) - You can use printf() to debug
- Output is written to
/log/def.logon the SD-Card - You can read the file in your browser using http://miniserver/dev/fsget/log/def.log or via (S)FTP
- Output lines are terminated automatically
- If you add
\nin your call, you'll cause an empty line - You can't printf lines piece-by-piece because of this
- If you add
- Output is written to
- The program is only run once
- If it calls
exit(), or the script ends, it will stop running until you restart the Miniserver (f.ex. by updating the config) - If you want to react to input, you'll have to run a loop and use
sleep()orsleeps()
- If it calls
- If your program causes an error (segfault, PicoC errors, etc), the Miniserver will disable the script and reboot
- All in-memory info will be lost - and you possibly won't get logs telling you what went wrong
- The PicoC version on the Miniserver:
- Crashes when you try to use function-scoped
staticvariables -- use global variables instead - Will throw syntax errors when trying to do ternary operations (f.ex.
a>b ? a : b) - Doesn't support
#include
- Crashes when you try to use function-scoped
exit()doesn't take an argument- but on regular PicoC, it needs a return code like
exit(1)
- but on regular PicoC, it needs a return code like
sleep(int time)on Loxone takes milliseconds- on latest PicoC & regular C it takes seconds
- There's
sleeps(int time)on Loxone if you need it
General PicoC quirks:
- You can't concatenate string constants by breaking them into multiple pieces like this:
"a " "string" - Note that PicoC exerts weird behavior when using
if (DEBUG) {in some places- Examples are suddenly throwing type errors in if-conditions involving struct member accesses
- we use
if (DEBUG) {instead to avoid these
#defineworks like a function call- It can only return an expression
- It can't be used for type definitions
typedef structdoesn't work- Arrays have to have their size hardcoded
- f.ex.:
char* bytes[4] = {1, 2, 3, 4}
- f.ex.:
- Arrays have to be defined in a single line, no linebreaks or
\allowed
- Wikipedia: Smart Message Language (German)
- Loxone PicoC documentation
- Perl SMLDump script
- Python SmlLib
- PicoC interpreter
- OpenSML
- EMH ED300L Manual (German)
- tibber Pulse (German)
- Loxone Miniserver (German)
- Information about tibber Bridge API (German)
- Some more info
- PicoC underlies the BSD-3-Clause license.
- smldump.pl and Python SmlLib are both provided under the GNU GPLv3
- The CRC checking code in
reader.cis taken from OpenSML, licensed under the MIT license. - This picoc program, documentation and auxiliary files are provided under the BSD 3-Clause license.