Moddable SDK 3.5.0

This release is version 3.5.0 of the Moddable SDK. It contains changes to the Moddable SDK made between November 1, 2022 and December 1, 2022

Highlights of this release include:

• All new JavasScript profiler to quickly identify performance bottlenecks in scripts
• Early support for ESP32 family boards with only USB ports including Adafruit QT Py ESP32-S3
• Asynchronous I²C using ECMA-419 IO for ESP32
• Improvements to tools for more reliable builds and updates
• New streaming audio modules with example
• Support for M5Atom U and M5StickC PLUS

This month we are particularly interested in hearing about your experience using our new JavaScript profiler in xsbug. If you have questions or suggestions about anything here start a new Discussion on our GitHub repository, drop by our Gitter to chat, or contact us on Twitter at @moddabletech.

About Version Numbers

The Moddable SDK has never had a version number before this release. Instead, each release has had a git tag based on the date (OS221101). We are switching to a Semantic Versioning style version to make it easier for developers to track Moddable SDK updates.

In switching from date-based tags to version numbers, we had to choose an initial version number. The choice is somewhat arbitrary. The Moddable SDK has been used in large-scale commercial deployments for many years, so the version number should be at least 1.0 to reflect that it is production-ready. The team decided 3.5.0 was a good place to start.

Our XS JavaScript engine has always had a version number (currently 13.2.0) which uses Semantic Versioning style. This version number is embedded in byte-code binaries, for example, to ensure the byte-code is only executed with a compatible version of the XS engine. XS will continue to have its own version number. Other parts of the Moddable SDK may have their own version numbers moving forward.

This release also adds a version number to the suite of Moddable SDK tools. This version number is now used by the Moddable SDK build system to detect when the binary tools build is out of sync with the Moddable SDK sources. if a version mismatch is detected when building a Moddable SDK project with mcconfig or mcrun, the binary tools are automatically rebuilt and any intermediate build results are removed.

Our goal in this work on version numbers is to improve the developer experience. We welcome your feedback on the changes and suggestions for further improvements.

Release Details

• XS
  • New sampling profiler for JavaScript
    • Identifies functions using the most time
    • Integrated into xsbug
    • Results displayed in realtime
    • Always available - no special build option needed
    • Supported in simulators and ESP8266, ESP32, and Pico device builds
    • Documentation explains how to interpret results
  • Fixes for issues found with fuzz testing
    • Several issues found by Fuzzilli in testing by @Agoric. For the curious, here's a small script that crashed prior to this release.

     const v5 = "".matchAll(); const v6 = v5.next(); v6[0] = true; new Int16Array(v5);
    

    • Out-of-bounds stack write in the RegExp parser found by Google's oss-fuzz
  • Names for more functions shown in xsbug stack traces and profiles
    • Anonymous functions are numbered so they can be distinguished
    • Simulators now extract native function names for more functions
  • New xsIsCallable and xsmcIsCallable APIs so C code can easily determine if an object may be called as a function
  • Removed FinializationRegister.prototype.cleanupSome. XS implemented it before it was removed from the final language standard
• Tools
  • xst
    • Profiling support
    • Option to parse input file as JSON for testing (useful for fuzzing)
    • Smaller initial VM allocation when fuzzing to reduce ASAN overhead
  • Screen test, the original simulator in the Moddable SDK, has been completely removed. All projects should now be using mcsim.
  • Tools now detect mismatch between binary and sources and automatically rebuild when needed
  • ESP32 builds in environments that have not sourced $IDF_PATH/exports.sh now automatically source it
  • ESP32 builds now report any mismatch between current and recommended ESP-IDF version
  • ESP8266 builds now report error if expected tools not found
  • Pico builds now report error if expected tools not found
• Devices
  • Experimental support for developing with ESP32 family boards that have only USB ports, including Adafruit QT Py ESP32-S3. See the discussion thread to learn how to try this out.
  • Added support for M5Atom U devices (contributed by @stc1988)
  • M5StickC PLUS - additional examples, mod support, and documentation (contributed by @stc1988)
  • ESP-IDF 4.4.3 now recommended for ESP32 builds
  • Accessing LED on Pico W target no longer crashes when Wi-Fi is uninitialized (reported by @rei-vilo)
  • Neopixel on ESP32
    • Warning of deprecated API fixed
    • Creates pixels as data buffer (following best practices)
• Modules
  • Timers on Windows
    • Fix for code that reschedules time in their own script callbacks #955 (reported by @brenocastrocardoso)
    • Now implemented using multimedia timers, allowing for millisecond precision to better match other platforms. This limits the number of simultaneously active timers in the Windows simulator to 15, which should be enough for most projects.
  • CRC module now correctly handles calculations split across multiple buffers #956 (reported by @ralphwetzel)
  • Fetch module now uses standard URL module
  • ECMA-419 sensor drivers all updated to use standard SMBus function names
  • Piu scroller option to center target when revealing hidden content
  • WavReader class from wav2maud tool now available as standalone module for projects that need to parse WAVE file headers.
  • AudioOut
    • Fixes for Windows
    • Supports a separate callback function for each stream (instead of a single callback function used for all streams)
  • SNTP client recognizes and handles version 3 NTP packets (there are still public servers in use returning version 3 packets)
  • Neopixel getPixel API
• ECMA-419, ECMAScript® Embedded Systems API Standard
  • HTTP Server implements routes consistent with latest TC53 consensus. Routes are a lightweight plug-in mechanism for the server
    • WebSocket server extension
    • Server Sent Events server extension
    • Web Page server extension
  • Asynchronous I²C and SMBus implementations for ESP32. Implements full draft spec, entirely non-blocking using a dedicated FreeRTOS task.
  • Implementation of FT6206 touch driver using asynchronous SMBus. Experiment to apply asynchronous I²C in Sensor Class Pattern.
  • Some SMBus function names were based on an early draft of ECMA-419 1st Edition and were changed in the final draft. These have been corrected. Compatibility shims are in place temporarily to ease the transition.
  • Separate manifest for DNS resolver to allow it to be easily included by other manifests
  • HTTP client
    • read function supports passing in existing buffer to read into (in addition to a number specifying the requested bytes to read)
    • Fix for exception if callback closed instance
  • WebSocket client
    • Sends optional HTTP request headers
    • Correctly handles receiving 0 length messages
• Examples
  • New audio streaming example
    • Streams over HTTP using ECMA-419 HTTP client
    • Low memory and CPU overhead
    • Recover from network stalls
    • WAVE file streaming using WavStreamer module
    • SBC compressed streaming using SBCStreamer module
    • Mix with other audio sources
    • See documentation for details
  • Piu Wi-Fi config example no longer throws exception #967 (reported by @rei-vilo)
• Documentation
  • Added ESP32-S3 and ESP32-C3 to ESP32 document.
  • Secure Socket (TLS) documentation updated with clientKey & clientCertificates, fix to applicationLayerProtocolNegotiation, new section on secure MQTT connections, and note on PEM/DER conversions. Inspired by discussion thread started by @nickthiru about using AWS with the Moddable SDK
  • Audio output documentation updated with recently added features (suggested by @meganetaaan)
• TypeScript Typings
  • Improvements to typings, mostly Commodetto. Eliminates all warnings about Moddable typings from Visual Studio. (contributed by @Templarian...

November 1, 2022

This release contains changes to the Moddable SDK made between October 1, 2022 and November 1, 2022

Highlights of this release include:

• Support for seven more Raspberry Pi Pico dev boards
• ECMA-419 I/O for Raspberry Pi Pico
• Enhanced support for ESP32-C3 and ESP32-S3
• Faster builds

If you have questions or suggestions about anything here, drop by our Gitter to chat, start a new Discussion on our GitHub repository, or contact us on Twitter at @moddabletech.

• Devices
  • Raspberry Pi Pico
    • Build targets for seven more Pico development boards
      • Adafruit ItsyBitsy RP2040
      • Adafruit QT Py
      • LILYGO T-Display RP240
      • Pimoroni PicoSystem
      • Pimoroni Tiny 2040
      • Seeed Studio XIAO RP2040
      • Sparkfun Pro Micro RP2040
    • Neopixel support
    • Raspberry Pi Pico W LED (resides on Wi-Fi controller) available as Digital pin 32
  • ESP32
    • M5StickC Plus device support added (contributed by @kitazaki)
    • AI Thinker ESP32 C3-32S-Kit build target – supports 2 MB flash and LEDs
    • SPI support for ESP32-C3
    • ESP32S3 runs at 240 MHz now by default
    • New resetReason module to retrieve the value of esp_reset_reason()
• Ecma-419
  • Ecma-419 I/O support for Raspberry Pi Pico
    • Analog
    • Digital
    • DigitalBank
    • I²C
    • PWM
    • Serial
    • SMBus
    • SPI
  • Serial read on ESP32 and ESP8266 now properly matches specification when passed a buffer
  • Analog and SPI now build for ESP32-S3
  • M5StickC supports hat I²C bus (contributed by @kitazaki)
• Modules
  • Audio Output
    • Fixes for buffer validation
    • Mixer option to output to existing buffer
    • Guard against invalid host data
  • Networking
    • TLS (SecureSocket) now queries the native socket to determine space available to write. Avoids write failures due to overflow and under-utilization of available space
    • HTTP client supports URL and HTTP headers of arbitrary length
    • Wi-Fi accepts WPA PSK on both ESP32 & ESP8266. #870 (Suggested by @MKGaru)
  • Outline rendering of arcs with CanvasPath no-longer overflow buffer
  • XPT2046 touch driver supports flipping X & Y axis
• XS JavaScript engine
  • Fixes for stability issues found by fuzz testing
  • Yes...
    • ...this was an unusually quiet month for XS
    • We're hard at work on a very cool new feature that will debut later this month
    • Stay tuned!
• Tools
  • Faster ESP32 builds on macOS by running tools in parallel
  • Faster macOS tools builds by runnings tools in parallel
  • Instrumentation in xsbug now shows "turns" for ESP32 and ESP8266, the number of times per second the event loop runs
• TypeScript
  • Type definitions added for deepEqual and structuredClone (contributed by @meganetaaan)
• Documentation
  • Added new Raspberry Pi Pico devices
• Tests
  • Piu Sound tests added
  • audioOut tests fixed to eliminate out-of-memory errors
  • HTTP large headers test added
  • BLE client tests for scanning and whitelist

October 1, 2022

This release contains changes to the Moddable SDK made between September 1, 2022 and October 1, 2022

Highlights of this release include:

• Raspberry Pi Pico W support!
• HTML standard URL module
• XS virtual machines on ESP32 can now be over twice as large
• Preloading of regular expressions
• New tests for BLE server and audio output

Note: The original simulator in the Moddable SDK, screentest, will be removed from the Moddable SDK in October. The default simulator has been mcsim for several months. If you haven't already transitioned to mcsim, now is the time.

If you have questions or suggestions about anything here, drop by our Gitter to chat about it, start a new Discussion on our GitHub repository, or contact us on Twitter at @moddabletech.

• XS
  • Regular Expressions can now be preloaded! This allows regular expressions to be used in more projects by reducing their RAM footprint and moving their initialization from runtime to build-time. Read more about the tiny, but full-featured, regular expression engine in XS, in our blog.
  • The XS memory allocator for microcontrollers has been enhanced to support discontiguous allocations. This allows a single virtual machine on ESP32 to allocate unlimited internal ESP32 memory, increasing the maximum VM size to nearly 300 KB from about 128 KB for devices with no PSRAM. Details are in the manifest documentation.
  • Fix mistake in operation order in WeakRef.prototype.deref
  • Fixes to snapshot generation to eliminate divergences
  • Fixes for several obscure crashes found with Fuzzilli fuzz testing
• Ecma-419
  • Initial support for ESP32-C3: Digital, DigitalBank, I²C, SMBus, Serial
  • Initial support for Raspberry Pi Pico and Raspberry Pi Pico W – Digital, DigitalBank
  • TCP Socket Listener for Linux
  • UDP socket for lwip doesn't require onReadable callback
  • UDP socket on macOS and Linux implement port property
  • HTTP server updated to allow detach to be called from onDone callback to support WebSocket server exploration
  • WebSocketClient and WebSocket constructors support attaching to existing socket
  • Sensor driver for MPU6886 IMU
• Modules
  • MCU-optimized implementation of HTML standard URL class together with tests. The implementation fully conforms to the standard with two exceptions: Punycode and
  • IDNA support are unimplemented. These are used primarily for the display and safe handling of user-entered URLs in browsers, which are not generally a concern on embedded systems. With some effort (and increase in code size), the implementation could support both.
  • modTimerRemove on Linux doesn't crash on NULL timer
  • Add manifest.json for AudioOut module for consistency and to simplify builds
  • AudioOut fixes
    • Sample accurate mixing
    • Accurate tone generator frequency
    • Allow multiple compressed formats on a single channel
  • Fix crash on close in NimBLE BLE client if connection never completed
  • Add manifest.json for CRC module for consistency and to simplify builds #938
  • structuredClone fix to reject host objects
• Devices
  • Raspberry Pi Pico W support added – Wi-Fi works
  • Analog pin support for ESP32-C3
  • I²C build fix for ESP32-C3 to account for fewer I²C ports compared to other ESP32 models
  • Keeping up with ESP-IDF audio changes
  • PDM audio output option for ESP32
• Tests
  • Add BLE server module tests
  • BLE test client (to support server tests)
  • Complete URL module tests
  • Add audio output tests for mixer and playback

September 1, 2022

This release contains changes to the Moddable SDK made between August 5, 2022 and September 1, 2022

Highlights of this release include:

• Deep cloning objects with the standard web structuredClone function optimized for microcontrollers. Complements deepEqual added last month.
• Big progress on implementation of the ECMA-419 standard: RTC, sensors, HTTP server, TypeScript typings for I/O, and more
• Networking enhancements including TLS certificates updates and improved reliability for dropped Wi-Fi connections
• Detailed documentation on writing unit tests to run on microcontrollers using testmc

Note: The original simulator in the Moddable SDK, screentest, will be removed from the Moddable SDK later this month. The default simulator has been mcsim for several months. If you haven't already transitioned to mcsim, now is the time.

If you have questions or suggestions about anything here, drop by our Gitter to chat about it, start a new Discussion on our GitHub repository, or contact us on Twitter at @moddabletech.

• XS JavaScript Engine

  • Updated Compartment implementation (for Secure ECMAScript / Hardened JavaScript) to match the latest proposal discussed in TC39. See the XS Compartments document for details and a link to the proposal.
  • Fix memory leak when writing XS snapshot

• Modules

  • littefs reports detailed error messages for exceptions in debug builds and error code number in instrumented and release builds
  • File systems on macOS and ESP32 support read/write of more kinds of buffers
  • structuredClone now available! See "Going Deep with XS: deepEqual and structuredClone" for details.
  • Updated built-in TLS certificates to replace expired certificates and add newer certificates. Reported by @stc1988. #828
  • Piu idle handling optimization eliminates frequent calls from C to JavaScript
  • Fixes to lwip Socket to improve reliability when Wi-Fi connection dropped unexpectedly
  • Original MQTT client properly parses topics of more than 255 bytes
  • SNTP client update to clean-up implementation and handle failures reliably
  • Start of MCU-optimized implementation of HTML standard URL class together with tests. Expected to completed in September 2022.

• ECMA-419

  • First draft of proposed ECMA-419 HTTP Server with example.
  • First draft of high-level async-generator based HTTP server inspired by fetch and Deno built on proposed ECMA-419 HTTP server with example.
  • Sensor driver implementations updated to match draft ECMA-419 2nd Edition sensor classes
    • CCS811 - environment
    • BMP180 - pressure and temperature
    • BMP280 - pressure and temperature
    • Capacitive moisture
    • Qwiic soil moisture
    • AK8963 - magnetometer
    • HMC5883 - magnetometer
  • RTC (Real-Time Clock) driver implementations updated to match draft ECMA-419 2nd Edition RTC peripheral
    • DS1307
    • DS3231
    • MCP7940
    • PCF85063
    • PCF8523
    • PCF8563
    • RV3028
  • Updated RTC examples and hosts using RTC to latest API
  • ECMA-419 Listener implementation for macOS

• Devices

  • Update to ESP-IDF v4.4.2 for ESP32 builds (previously using v4.3)
  • Re-enable stop bit control in I²C APIs, now that underlying issue is fixed in ESP-IDF v4.4.2 #929
  • ECMA-419 Serial on ESP32 has fewer dependencies on ESP-IDF serial driver after being broken by ESP-DF v4.2.2 update. Reported by @meganetaaan. #931
  • Pico LCD 1.3 device now supported. Contributed by @stc1988. #905

• TypeScript

  • Type declarations (aka typings) for most ECMA-419 I/O classes: Digital, DigitalBank, PWM, I²C, Serial, PulseCount, SMBus, SPI, TCP, Listener, and UDP. Contributed by @meganetaaan. #928

• Tools

  • xsbug options to color traces to console log and show/hide exceptions, colorize TypeScript and *.mjs source files, and show correct cursor over non-linkable log lines. Contributed by @cmidgley. #909
  • New mcprintski tool to regenerate TLS certificate index (macOS only)
  • xsbug resolves symbolic links so Moddable SDK tests no longer need to be copied to test262 directory to be used with test262 and testmc
  • XS Linker no longer strips async iterator when async generator is used
  • nodered2mcu tool to support Node-RED MCU Edition initiative. Optimizes flows by compiling portions to JavaScript.
  • M5Paper simulator updated to work with Compartment updates that blocked access to MODEL global. Reported by @gingerbeardman in #933.

• Documentation

  • New documentation on "Writing Tests for the Moddable SDK" to get developers started writing unit tests to run with testmc.
  • Correct documentation on unscheduling a Timer
  • Update Files documentation to describe use of iterator to enumerate directory contents

• Testing

  • test262-style tests added for Compartment
  • New BLE client tests for rssi and properties in advertisements.
  • testmc allows tests to extend assert (convenient for test fixtures)
  • testmc times out when attempting to connect to unavailable Wi-Fi access point

August 5, 2022

This release contains changes to the Moddable SDK made between June 28, 2022 and August 5, 2022.

Highlights of this release include:

• A new deepEqual module that provides deep equality comparison of objects emulating the behavior of Node.js
• A new littlefs file system that provides a full-featured file system for even the smallest microcontrollers
• Raspberry Pi support is back! Build, run in the simulator, debug with xsbug, and even try out network protocol implementations based on proposals for Ecma-419 2nd edition.

If you have questions or suggestions about anything here, drop by our Gitter to chat about it, start a new Discussion on our GitHub repository, or contact us on Twitter at @moddabletech.

• Modules
  • New deepEqual module implements deep object comparison emulating the behavior of assert.deepEqual() and assert.deepStrictEqual() from Node.js. Implementation is in C using XS internals, so it is both small and fast enough to be practical on microcontrollers. More details in documentation and the deepEqual tests (adapted from the excellent tests by @ljharb in his is-equal module).
  • New littlefs file system module. The littlefs module provides a true hierarchical file system with long file names. It has full resilience to power-loss and wear leveling support while using almost no RAM (under 1 KB in many cases). The implementation is API compatible with the Moddable SDK File module. Supported on ESP32, ESP8266, and simulators. Details in the documentation.
  • Added driver for PCF85063 Real-time Clock (RTC) based on PCF8563
  • Hex.toString() refreshes write pointer after garbage collector may have moved memory
  • Piu text field on Linux correctly handles gtk_entry_get_text returning NULL
  • Socket on iOS no longer reports error on write of 0 bytes
• Ecma-419
  • TCP socket now available on Linux
  • UDP socket now available on Linux
  • TCP socket for macOS no longer reports error on write of 0 bytes
  • TCP socket for lwip (ESP32, ESP8266) no longer invokes callbacks after being closed
  • DNS UDP resolver handles "localhost" and IPv4 addresses
  • WebSocket client
    • No longer stalls when receiving zero length payloads
    • ping and close replies are deferred to allow onWritable to be called with updated count to eliminate client buffer overflows
    • binary is set correctly on received messages when more is true
  • HTTP and WebSocket clients handle empty strings for the path
• TLS
  • TLS handshake supports ECDHE_RSA key exchange on certificate verify message for compatibility with AWS IoT (ECDHE-ECDSA-AES128-GCM-SHA256 cipher suite)
  • TLS client now verifies certificates signed with ECDSA-SHA*
  • Fix heap corruption caused by BigInt helper functions (mod_square, mod_mul, mont_square) when called with r and a arguments using the same pointer. This caused intermittent crashes during the TLS handshake with certain cipher suites.
  • Crypt digest functions defend against possible memory moving due to garbage collection
• Devices
  • Timer module updated to work correctly on Linux
  • Several fixes so Moddable SDK builds on Raspberry Pi works reliably again
  • Remove unnecessary configuration in esp32_thing manifest (contributed by @Templarian)
  • M5Stack Core2 virtual buttons supported (reported by @jorge-aqp-23)
• XS
  • Snapshots significantly reworked so that creating a snapshot of a virtual machine does not cause changes to future snapshots on the same instantiation of that virtual machine
  • Fixes for JavaScript parser issues found by ClusterFuzz (OSS-Fuzz) by Google
  • Fixes to Array.prototype.sort and WeakSet crashes discovered through fuzz testing with Fuzzilli performed by @Agoric
  • Small optimization to JavaScript parser to reduce memory use
  • JSON parser correctly handles empty strings stored in ROM
• TypeScript
  • Add typing for Net.get("dns")
  • Update typings to reflect that Piu TextureSkinDictionary accepts an array of colors. (contributed by @meganetaaan)
  • Rename PixelsOut typing to work on case sensitive file systems
• Documentation
  • Added documentation for deepEqual and littlefs
  • Broken links in documentation fixed
  • Formatting more consistent across all documentation
• Tests
  • New BLE client tests that use new BLE server test app
• Examples
  • ls013b4dn04 example needed to include CFE (Commodetto Font Engine) (reported by @Templarian)
  • wificonnection example initializes the Wi-Fi mode rather than depending on host to do so
• Tools
  • Experimental option to mcconfig to specify xsbug connection mcconfig -d -m -x localhost:5004 (requested by @ralphwetzel)
  • When the XS linker (xsl) encounters an error during preload, the full stack trace is displayed to make pinpointing failures easier (implemented using recently added Error stack feature)
  • Updated builds of tool binaries to correct path issues
  • Build binaries include instructions for use when delivered as a download

June 28, 2022

This release contains changes to the Moddable SDK from June 1 through June 28, 2022.

Highlights of this release include:

• Our XS JavaScript engine now supports the latest version of JavaScript, ECMAScript 2022
• We've added a large suite of tests to validate the Moddable SDK and XS
• The ESP32-C3 MCU is supported by a new port, our first to RISC-V(!)
• Binaries for all Moddable SDK tools provided for macOS (x86 and ARM), Windows (x86 and ARM), and Linux (x86)
• Binaries for xst (the XS test tool) are provided for use by jsvu and es-host.

If you have questions or suggestions about anything here, drop by our Gitter to chat about it or start a new Discussion on our GitHub repository.

• XS
  • XS now supports the ECMAScript 2022 standard which adds the following features:
    • Class fields
    • Private fields
    • Top-level await
    • Class static block
    • Ergonomic brand check for private fields
    • .at() on Array, TypedArray, & String
    • Object.hasOwn()
    • RegExp Match Indices
    • Error cause
  • Note that "Ergonomic brand checks for private fields" is often misunderstood. This post by Axel Rauschmayer explains the feature well.
  • xst and xsbug (test support) now enable these test262 tests
    • Object.hasOwn
    • class-fields-private-in
    • class-static-block
    • error-cause
  • xst and xsbug (test support) disable test262 tests for unimplemented stage 3 features
    • array-grouping
    • decorators
    • regexp-v-flag
  • Conformance issue report fixes
    • XS appears to incorrectly ignore comma at end-of-line. #899 (reported by @gibson042)
    • "modulo signed zero incorrect?" #895 (reported by @devsnek)
    • "incorrectly interprets zero when used as a multiplication factor" #335 (reported by @jugglinmike)
    • "(,) is incorrectly treated as valid" #726 (reported by @gibson042)
    • "Maps and sets incorrectly distinguish NaN values" #888 (reported by @gibson042)
  • Fuzz testing
    • Fixes for runtime issues found by Fuzzilli testing by @Agoric and @moddabletech
    • Fixes for parser issues found by ClusterFuzz (OSS-Fuzz) by Google.
  • Uninitialized data in XS snapshot by initializing unused dummy padding field
  • Fix test262 Promise failures on ARM Linux targets (reported by @warner)
  • Fix ancient memory leak in dtoa (rarely happened)
  • Implement memory optimization for Promise proposed by @mhofman
• TLS (SecureSocket)
  • Certificate transforms
    • PEM to DER
    • PrivateKey to PrivateKeyInfo
    • (transforms support on-device AWS provisioning)
  • Application Layer Protocol Negotiation (ALPN)
    • Fixed exception caused by use of const #916 (reported by @barianand7)
    • Support binary and UTF-8 protocol names
  • BER/DER support
    • Encoding now outputs all integers in canonical format
    • OID encoding fixed
    • Use resizable ArrayBuffer when encoding
    • Fix atring decoding
  • SSL Streams
    • Uses resizable ArrayBuffer for more efficient writes
    • Correct output of all UTF-8 data (previously only handled ASCII correctly)
  • Remove unused Base64 support from crypt/bin module (data/base64 is preferred)
• Device porting
  • ESP32-C3 is now supported. This is the first Moddable SDK port to RISC-V!! Thank you to @LokiMetaSmith for making this happen.
  • BLE now supported on the ESP32-S3 port
• ECMA-419
  • Provide manifests for network protocol implementations to make it easier to include them in projects
  • fetch implementation can now be preloaded to reduce RAM use
• Tests
  • Over 500 test files containing thousands of individual tests are now part of the release. These cover many of the modules in the Moddable SDK and areas of XS not covered by test262.
• Documentation
  • New document: Testing the Moddable SDK
  • Fix broken links
  • Make formatting more consistent across documents
  • Add Timer.unschedule
  • Add note about Preference key and value size limits varying by host
  • Updated XS conformance document with latest results
  • Added documentation for TLS certificate transforms
• Tools
  • Simulator now gracefully handles situation where a device cannot be found
  • mcconfig now correctly handles stand-alone string values in the platform section of a manifest platforms (previously only arrays of strings worked)

June 1, 2022

This release contains changes to the Moddable SDK from May 2 to June 1, 2022.

Highlights of this release include:

• Two new MQTT client classes, including one that is API compatible with the widely used MQTT.js package
• Updates to many ECMA-419 modules, mostly focused on validating features in-progress for 2nd Edition
• Several fixes for operations on preloaded objects
• Lots of bug fixes based on unit test, user reports, and fuzz testing

Thank you to everyone who contributed to this release with a PR or an issue report. Especially thank you to Agoric for continuing their amazing support of our fuzzing efforts. The results are making XS more secure, reliable, and conformant on all devices.

If you have questions or suggestions about anything here, drop by our Gitter to chat about it or start a new Discussion on our GitHub repository.

• Runtime
  • MQTT client
    • New ECMA-419 MQTT Client implementation
    • New MQTT.js compatible implementation built on ECMA-419 MQTT Client
  • ECMA-419
    • New ECMA-419 DNS resolver implementation using UDP. Matches current draft specification.
    • ECMA-419 TCP implementation on macOS no longer drives CPU use to 100% when there is unread data
    • ECMA-419 HTTP client write returns updated writable count, consistent with ECMA-419 MQTT and WebSocket clients
    • ECMA-419 TCP instances have remoteAddress and remotePort properties consistent with Node.js sockets.
    • ECMA-419 WebSocket client no longer gets confused when receiving a 0 tag byte
  • Preloaded objects
    • Proxy constructor and Proxy.revocable now work when passed host function primitives. Reported by @cmidgley.
    • Adding object in read-only memory (flash / ROM) to a WeakMap or WeakSet throws instead of crashing. Reported by @cmidgley.
    • JSON.stringify throws "read-only value" instead of misleading "cyclic value" when trying to serialize objects in read-only memory (flash / ROM). #340. Reported by @takeru.
  • Improved input parameter validation
    • ECMA-419 SPI for ESP8266 and ESP32
    • Hex module (converts between hexadecimal string and binary buffer)
    • Inflate and Deflate (zlib)
    • QRCode
  • Fixes
    • Calling Timer.clear() on a one-shot timer after it fired would crash. Reported by @cmidgley.
    • Inflate and Deflate fixes to work reliably when streaming
    • Piu Text objects no longer crash when trying to render if there is no style set. #887. Reported by @tony-abraham.
    • Top-level await now optionally supported for main module of a Moddable SDK project. Enable by defining MAIN_ASYNC in project manifest. Reported by @dtex to support J5e examples.
  • More general buffer support
    • DNS parser uses xsmcGetBufferReadable to support more kinds of buffers
    • Logical operations module uses xsmcGetBufferReadable to support more kinds of buffers
  • Devices
    • M5Stack Core2 – don't throw exception at start-up if MPU6886 not found
    • ESP8266 and ESP32 hosts no longer crash if the root virtual machine cannot be cloned because of insufficient memory. Instead, a debug build traces can't clone: no memory.
    • ESP32 CPU load instrumentation fixes (as a result of reports by @LokiMetaSmith)
      • Don't get idle task for second core on single core targets
      • Rework native timer callback to work on all ESP-IDF targets
• XS
  • Fixes for at least 15 crashes found by fuzz testing with Fuzzilli. These generally only occurred in unusual situations that normal JavaScript code would not encounter. A few examples:
    • Calling Symbol() recursively created duplicate symbols and corrupt the global symbol table, eventually causing a crash
    • Fix crashes in lockdown() when environment was in unanticipated states (indexed property added to Date constructor)
    • Fix crash in RegExp parser
    • String.prototype.toUpperCase() and String.prototype.toLowerCase() would crash if the converted string required more bytes to store than the original string and fxRenewChunk couldn't resize the output block
  • Fuzzing support
    • Fix fxRenewChunk when mxNoChunks diagnostic mode is enabled to re-use block when shrinking
    • Fix memory leak of script buffer when fuzzing. Reported by @arirubinstein.
    • Add entry point for Google's OSS-Fuzz. Contributed by @arirubinstein.
    • Option to initialize data buffers to pseudo-random data
  • Snapshots
    • Initialize padding bytes to 0 to generate deterministic output
    • Save and restore stack property of Error instances. Reported by @Agoric.
  • Fixed a subtle memory leak in WeakMap by changing visitation order of delete. Reported by @mhofman.
  • Migrated Secure ECMAScript / Hardened JavaScript built-ins from the xsnap application to xsLockdown.c for wider use
  • String.fromArrayBuffer now also accepts SharedArrayBuffer and host buffers as input. Requested by @cmidgley. Thanks to @HipsterBrown for reporting the Windows build failure.
• Tools
  • mcsim release build works again
  • xsbug fix for path mapping when the paths are different from the root (for example, mapping a Linux path to Windows path). #882. Reported by @cmidgley.
• Examples
  • ePaper Photos example is now available on M5Paper
  • Fix typo in font name of Commodetto Fonts example
  • Inflate example demonstrates how to stream using onData and onEnd callbacks
  • ECMA-419 TCP protocol examples updated to use device.network.[protocol] in config.js file. Eventually this will migrate to the network protocol implementations.
• Documentation
  • Remove Git Bash-based Windows instructions and assume Git Command Prompt integration instead. #510, #632, #897
  • Update documentation on Hex module to describe all arguments

May 2, 2022

This release contains changes to the Moddable SDK from April 2 to May 2, 2022. There are two major areas of focus in this release:

1. Hardening XS. There has been a great deal of work recently to improve the robustness of our XS JavaScript engine. This work has been done together with Agoric and independent researchers. The majority of issues have been uncovered using the Fuzzilli fuzzing engine with customizations to target specific details of XS. Most of vulnerabilities found are extremely difficult to trigger in normal operation and none have been demonstrated to be exploitable. All have been fixed. The majority of issues were found using a special build of xst, the XS test tool. Check out the Hardening XS page for more information.
2. Ecma-419 networking. Work is well underway on the 2nd Edition of Ecma-419, the ECMAScript Embedded Systems API Specification. One of the key goals of 2nd Edition is to specify standard APIs for common network protocols. There are now proposals in place for HTTP, WebSocket, and MQTT. The Moddable SDK contains implementations for each of those, together with high level implementations of Fetch and WebSockets from HTML, and the MQTT module use with Node. These bring APIs familiar to web developers to embedded. See below for details.

The following is a summary of the changes. Thank you to everyone who contributed!

• XS
  • Vulnerabilities
    • Fix vulnerability in ArrayBuffer.fromBigInt with negative minBytes argument
    • Fix "Out-of-bounds Read in fxUint8Getter" vulnerability. Reported by @Q1IQ and @P1umer. #896
    • Fix vulnerability caused by RegExp engine not initializing capture reference name in one code path
    • Fix garbage collector triggered vulnerabilities
      • String.prototype.normalize
      • String.prototype.indexOf
      • String.prototype.lastIndexOf
      • Compartment constructor
      • Promise constructor
      • Accessing arguments
    • Fix vulnerabilities reported by @arirubinstein of Agoric
      • "AddressSanitizer: memcpy-param-overlap (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3db88) in __asan_memcpy+0xbc" #889
      • "AddressSanitizer: global-buffer-overflow xsBigInt.c:936 in fxBigInt_ffs" #890
      • "AddressSanitizer: heap-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3dc70) in __asan_memcpy+0x1a4" #891
      • "AddressSanitizer: stack-overflow xsLexical.c in fxGetNextCode" #892
  • Fix lockdown when Date has index property
  • Fix obscure case where colors of objects in ROM were confused with instance alias
  • Fix "Built-in function %ThrowTypeError% is incorrectly recognized as a constructor" conformance issue. Reported by @gibson042. #903
  • Fix Function.prototype.bind to work with proxies
  • Snapshots
    • Properly reload tagged templates
    • Zero unused trailing chunk bytes to support deterministic snapshots
  • mxNoChunks option now works with over-allocated arrays
  • Migrate Hardened JavaScript (also known as Secure ECMAScript) functions from xsnap to xs/xsLockdown.c to be able to use more widely
• Runtime
  • Ecma-419
    • Ecma-419 implementation of Display class for ILI9341. Poco and Piu updated to support (conditionally compiled, not yet default).
    • Ecma-419 implementation of TCP for macOS
    • Ecma-419 implementation of UDP for macOS
    • Ecma-419 DigitalBank for ESP32 only calls gpio_isr_handler_remove for pins for which it previously called gpio_isr_handler_add
  • Graphics and UI
    • BMP writer now supports writing 32-bit BMP images. png2bmp can now output 32-bit BMP images from PNG with alpha.
    • Optimize redraw of centered content in Piu by correctly checking that it is constrained
  • Fix analog audio output on ESP-IDF 4.4 (broken by API changes in ESP-IDF)
  • Update to more recent commit on ESP-IDF 4.4 to get analog audio fixes
  • xsbug connection was incorrectly marked as verified on ESP32, ESP8266, and Pico
  • Build changes to allow xsbug to work with ESP32-S3 board. Reported by @liurenqiu520. #902
• Tools
  • Resolve xsbug crash when null character received from debug target
  • Fix memory leak in Piu PC which caused xsbug to exhaust memory after running for a long time
  • xsbug uses Maps instead of object properties to maintain list of paths for cached eval code. Faster and more scalable for long test runs.
  • When building xst on macOS, use a much bigger margin to check for pending stack overflows to account for unexpectedly huge stack frame of XS run loop (fxRunID) in debug builds (nearly 90 KB on 64-bit ARM!)
  • TypeScript type declaration files for BLE and HID peripherals. Contributed by @chigix. #901
• Documentation
  • BLE updates to use Class Client consistently. Contributed by @chigix. #901

March 30, 2022

This is the first release in a little while. These release notes cover February 28, 2022 to April 1, 2022. Moving forward, the goal is to provide release notes at least once a month.

There's a lot of great work in this release – new features, updates to existing features, and the inevitable bug fixes. The release notes contain extensive links to make it convenient for you to explore the changes.

Thank you to everyone who reported an issue or contributed a PR. We appreciate you sharing your time and expertise.

• Tools

  • Simulator
    • mcsim is now the default simulator in the Moddable SDK on macOS, Windows, and Linux. The original simulator, Screen Test, is still available but not recommended and will eventually be removed.
    • mcsim simulator updates instrumentation dynamically when stepping, consistent with screen test
    • Add M5StickC target to mcsim simulator. Contributed by @wilberforce.
    • Dark mode now available in xsbug and mcsim
    • mcsim now calls setup/* modules in the same way as device targets. Contributed by @FWeinb.
    • M5Paper simulator now shows power-pin status. Contributed by @FWeinb.
    • mcsim has simulators for Pico Display and Pico Display 2
  • The compileDataView tool has been integrated into the Moddable SDK. This tool simplifies working with binary data structures in JavaScript. It converts C language data structures to JavaScript implementations. The js/views example shows how to use compileDataView in a project manifest to share a C header between JavaScript and C source code.
  • Fix duplicate path problem building TypeScript modules. Reported by @MKGaru. #869.
  • xst (XS Test Tool) now contains support for fuzzing XS with Fuzzilli. Based on contribution by @jessysaurusrex. #817
  • Improvements to test262 tool to avoid buffer overflows when transferring test script source code to target

• Devices

  • Raspberry Pi Pico is now a supported platform! Check out the Pico documentation for supported features, set-up instructions, and a list of supported device targets.
  • Added support for Wemos OLED Lolin32 (based on Heltec WiFi Kit 32 port). Contributed by @costa-victor.

• Examples

  • New collection of BLE HID (Human Interface Device) peripheral examples for building BLE keyboards and media controllers. Read the blog post by @andycarle for demos and details.
  • New httpbridge example that demonstrates a new architecture bringing together Wi-Fi hotspot support, HTTP server, WebSocket server, and a very efficient use of ZIP files to deliver an embedded web site. Lots of cool techniques and useful code. Contributed by @wilberforce.
  • New Commodetto fireworks example. Created to show off the Pico Display, it works on any device with a display.
  • New Piu QRCode example in response to a request from @wilberforce.
  • Update xsuse example to eliminate device dependency on instrumentation indices and to calculate slot size at runtime for compatibility with 64-bit platforms

• XS

  • Fix JavaScript language conformance issues reported by @gibson042
    • "TypedArrays incorrectly write to 'NaN' properties". #886
    • "ToNumber incorrectly accepts 'INFINITY'". #885
    • "Change Date.parse to round milliseconds down". #879
    • "function name is set incorrectly on a method for a property key that is a registered symbol". #876
  • Fix crash in Compartment Function eval after lockdown
  • Unhandled Promise rejection now reports reason
  • Fix stack overflow in degenerate Array sort cases

• Runtime

  • Network
    • Add implementation of proposed Ecma-419 standard WebSocket client, with example.
    • Add implementation of HTML5 WebSocket class built on Ecma-419 standard WebSocket client, with example. Tested on macOS, ESP32, and ESP8266.
    • WebSocket server now supports creating a server from an existing socket. This allows one listener to be shared by HTTP and WebSocket servers. #755
    • Socket implementation on Windows supports reading outside of the callback, as on other platforms.#755
    • SecureSocket now propagates error state to client
    • Socket implementation on macOS, Linux, and Windows now detects receive error correctly to avoid infinite loop. Reported by @michaelfig.
    • Ecma-419 TCP implementation for lwip (ESP32 and ESP8266) now clears all native callbacks when socket is closed or disconnected
  • Base
    • Timer.clear may now be passed undefined without throwing an exception
    • Timer module times now correctly schedule when millisecond wraps 32-bit value. Reported by @Arorar3. #875
    • WebWorker instrumentation now guarded by semaphore to prevent debugger cross-talk with multiple threads
    • Merge updates to File module type declarations from @cmidgley
    • File module rename function handles slashes consistently across flat and hierarchical file systems. #878
    • Flash module updated to accept any kind of buffer for write
  • Fix Outline module compilation on Windows
  • Ecma-419 provider for Moddable One had SPI in and out pins reversed

• Documentation

  • CRC8 and CRC16 classes added to Data reference documentation.
  • Fix mix-up with PICO_SDK_DIR in Raspberry Pi Pico docs. By @HipsterBrown.

August 26, 2021

This release of the Moddable SDK moves to ESP-IDF 4.3 from 4.2. Developers building for microcontrollers in the ESP32 family must update their ESP-IDF to use this release of the Moddable SDK. Details on how to update your ESP-IDF build are available for macOS, Windows, and Linux.

The ESP-IDF 4.3 has many changes. Here are some highlights for developers using the Moddable SDK:

• Latest stable release from Espressif
• Supports ESP32-S3 Beta SoC. This is supported in the Moddable SDK with the esp32s3 build target. (Let us know if you try this out on ESP32-S3 hardware!)
• FreeRTOS v10.2.0
• Over-The-Air updates no longer block for several seconds when starting (more information below)

If you have questions or comments about this update, the Moddable SDK in general, or using JavaScript for embedded development in general, please join us on our Discussions page on GitHub. Follow @moddabletech on Twitter to keep up with our latest releases and news.

• XS
  • Metering support added for TypedArray
  • Support for marshalling alien arrays across workers
• Modules
  • TextDecoder rejects overlong UTF-8 sequences
  • OTA module for ESP32 sets OTA_WITH_SEQUENTIAL_WRITES to avoid erasing more blocks than required and blocking for several seconds when starting an update. (This uses an enhancement to the ESP-IDF contributed by Moddable)
  • File and Preference modules work in WebAssembly runtime (without persistence)
  • Wi-Fi module simulator implements access point mode (contributed by @wilberforce)
  • mDNS module adds debug logging of additionals
  • mDNS module performs case insensitive compares as required by the specification (fixes some failures)
  • wificonnection module supports alternate access points and ending connection attempts using new getAP message
  • lwip socket module on ESP32 joins multicast group on all active network interfaces
  • Remove unused ca-subject data from TLS manifest #691 (fix by @gavrilyak)
• Examples
  • New httpzip module and example shows how to serve static web site from a an embedded ZIP file, including compressed data and ETAG caching. Uses WebSocket for dynamic two-way communication. Based on an idea by @wilberforce, developed by @wilberforce with help from @phoddie.
  • New Over-the-Air (OTA) Update example using OTA module for ESP32
  • New continuous servo rotation example (contributed by @stc1988)
  • New IR send and receive examples for ESP32 using RMT module (contributed by @stc1988)
  • mDNS http server example publishes http service in addition to claiming name
  • Digital monitor example fix to detect falling edges (fix by @Frida854)
• Documentation
  • Update documentation on how sdkconfig files are processed (contributed by @jparker324)
  • Removed unnecessary slashes in paths in ESP8266 documentation (fix by @danhellem)
• TypeScript
  • Declarations added for RMT module (contributed by @stc1988)
  • Declarations added for CRC module (contributed by @stc1988)