Terry Ellison TerryE

Review Notes on Changes to app/lua53

These are my notes following my own review of the lua53 commit diff file:

Reasons for functional changes

The type LUA_TTABLE now has subtypes LUA_TTBLRAM and LUA_TTBLROF, with handling of these subtypes following the model adopted for strings being split into short and long subtypes. In general the variant coding for table subtypes is managed as low as possible in the ltable.c routine. The new ROTable is a cut down version of Table which only includes the fields used in ROTables.
The getXXX(o) access macros replace (o)->XXX field accesses for lu_byte fields in records that could be in constant (flash-based) memory.

Background and Objectives

The NodeMCU firmware is currently based on the Lua 5.1.4 core with the eLua patch and other NodeMCU specific enhancements and optimisations ("Lua51"). This paper discusses the rebaselining of NodeMCU to the latest production Lua version 5.3.5 ("Lua53"). Our goals in this upgrade were:

NodeMCU should offer a current Lua version, 5.3.5 that is as functionally complete as practical.
Lua53 will adopt a minimum change strategy against the standard Lua source code base, that is changes to the VM and runtime system will only be made where there is a compelling reasons for any change, for example Lua53 preserves some valuable NodeMCU enhancements, for example the addition of Lua VM support for constant Lua program and data being executable directly from Flash ROM in order to free up RAM for application use to mitigate the RAM limitations of ESP-class IoT devices
NodeMCU will provide a clear and stable migration path for both existing hardware libraries and ESP Lua

If you need to do C level debugging, then you need to compile the relevant files / subdirectories with -ggdb and -O0. Don't compile everything with -O0 because the resulting firmware image will be too large.

In this case I am debuggin the lua core so I have modified the app/lua/Makefile to set the debug defines and select the correct compiler options. I have also enabled break at startup on GPIO pin low. With the pin pulled low, the RTS traps into the gdb client, and I can connect to it using the xtensa remote debugger on the host.

What this debugging feature does is to call a common routine which

Lua Flash Store (LFS)

Background

Lua was originally designed as a general purpose embedded extension language for use in applications run on a conventional computer such as a PC, where the processor is mounted on a motherboard together with multiple Gb of RAM and a lot of other chips providing CPU and I/O support to connect to other devices.

ESP8266 modules are on a very different scale: they cost a few dollars; they are postage stamp-sized and only mount two main components, an ESP SoC and a flash memory chip. The SoC includes limited on-chip RAM, but also provides hardware support to map part of the external flash memory into a separate memory address region so that firmware can be executed directly out of this flash memory — a type of modified Harvard architecture found on many IoT devices. Even so, Lua's design goals of speed,

	#!/usr/bin/env python
	# Minimal file downloader

	import os
	import serial
	import sys
	import argparse
	import time
	import logging
	import re

	#! /usr/bin/env lua
	-- Cut-down Lua uploader for testing uart.on('data') functionality
	local dev = '/dev/ttyUSB0'
	local px = require "posix"
	local tt = px.open(dev, px.O_RDWR + px.O_NONBLOCK) or
	error('Could not open serial port '..dev, 0)
	local fn = arg[1]:match('[\\/]([^\\/]*)$') or arg[1]
	local f = io.open(arg[1]) or error("Must specify a valid file",0)
	local fc = f:read("a") -- '' is ignored in Lua 5.3
	local null,soh,ack,nack = '\0', '\1','\6','\21'

	local post, yield = node.task.post, coroutine.yield
	local max_namelen = 31
	local request = '\z
	%s %s%s.img HTTP/1.1\r\n\z
	User-Agent: ESP8266 app (linux-gnu)\r\n\z
	Accept: application/octet-stream\r\n\z
	Accept-Encoding: identity\r\n\z
	Host: %s\r\n\z
	Connection: %s\r\n\r\n'
	local host, dir, image, onDone

	#!/usr/bin/env python

	# This requires Python 2.7. This will not work with Python 3
	# Thanks to PaulW for the idea. In this case it was a matter of using
	# the Ff Inspector Netwroks analyser to log the interactive query with
	# "Persistent Logs" enabled, dumping the HAR to to pick out the key
	# queries and with a bit of trial and error to refine them.
	#
	# I run this daily as a cron job.

	--[[ A telnet server T. Ellison, May 2018

	This version is more complex than the simple Lua example provided in our distro. The
	main reason is that a single Lua command can produce a LOT of output, and the server
	has to work within four constraints:

	- The SDK rules are that you can only issue one send per task invocation, so any
	overflow must be buffered, and the buffer emptied using an on:sent cb

	- Since the interpeter invokes a node.output cb per field, you have a double

	--[[ A simple ftp server

	This is my implementation of a FTP server using Github user Neronix's
	example as inspriration, but as a cleaner Lua implementation that has been
	optimised for use in LFS. The coding style adopted here is more similar to
	best practice for normal (PC) module implementations, as using LFS enables
	me to bias towards clarity of coding over brevity. It includes extra logic
	to handle some of the edge case issues more robustly. It also uses a
	standard forward reference coding pattern to allow the code to be laid out
	in main routine, subroutine order.