[Must Read] V3 Project Plan & Features

[ayushsharma82]

Hello everyone!

It has been 3 years since I created this library and it has received unprecedented love from the open-source community which has been the driving force to keep enhancing this library.

Today marks as a stepping stone for V3 of ElegantOTA which will further reduce the size of webpage, add exciting new features, potential bug fixes with new optimized source code and most importantly support for new microcontrollers.

To make this possible, I'll need your help in figuring out what's best and what you guys need in ElegantOTA. So comment down any feature you feel is missing in ElegantOTA and I'll add it in my project plan.

Supporting New Platforms:
For past 3 years ElegantOTA has been stuck with ESP32 and ESP8266 but now I wish to increase the coverage of ElegantOTA so that more people can make use of it to update their microcontroller conveniently. If you know or work with any microcontroller ( other than ESP32 or ESP8266 ) that has OTA capability please write it down in the comments below. I'll look into it.

The microcontroller doesn't necessarily needs to have in-built WiFi, it can also be a NCP ( Network Co-Processor ) architecture like Nano 33 IoT.

[vortigont]

Hey, @ayushsharma82,
How about the compressed updates support? For esp8266 it's pretty easy out of the box since it's supported by eboot code, just need to add file format detection.
For esp32 you need some additional code to decompress an upload on the fly.
Not so long ago I've made a small lib implementing zlib-compressed OTA uploads for ESP32.
Should be pretty easy to adopt your project to use esp32-flashz. Let me know if you like the idea, I may come up with PoC PR or something.
Cheers!

[ayushsharma82]

@vortigont Good Idea! That will be much appreciated and it will also ensure quick uploads for those who are experiencing upload issues due to signal loss.

[silverfisk]

I'd like to be able to upload cryptographic keys at the first installation.
I don't have an exact workflow how this could be done, but on a conceptual level both encryption and signing would be nice.

Before uploading a new OTA file, the certificate has been deployed to the device as part of the initial installation.

ElegantOTA then require a file encrypted and/or signed with the private file. The device then cannot be updated without having access to the private certificate file.

[ayushsharma82]

@silverfisk That will be a nice addition as well.

We can use an approach with AES128-CBC or AES256-CBC encryption where we have a logic to buffer incoming chunks from the HTTP server and pass the buffer with correct length for decryption on-the-fly. As with CBC based encryption we already know the block size, we will be able to decrypt the file as we receive it from the user.

I'm only concerned about the performance drawback which will be incurred due to decryption, I hope it doesn't deter the HTTP upload process.

Very nice overall suggestions from the community! Love it.

[ayushsharma82]

If anybody knows a public key cipher which can work with parallel computing ( decrypting ), then please do let us know. I really prefer public key cryptography in IoT.

Till then, I only see AES to work as it's successfully implemented on the hardware level and in SDK.

[vortigont]

asymmetric crypto is veeery slow, so it is used only for key exchange for symmetric ciphers.
i.e. it makes no sense to encrypt/decrypt the firmware during upload and flashing. Only to sign the update and provide keys and certs.

[silverfisk]

@vortigont how slow are we talking?

At least signing would be nice to validate the binary. But if it's too slow to decrypt perhaps it's better to use HTTPS to encapsulate the payload in symmetric encryption?
The binary will still contain the WiFi password and be treated as a secret though.

[vortigont]

@silverfisk the difference is huge, 3-5 orders of magnitude or so
AES

openssl speed aes-128-cbc
...
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc    1724778.47k  2039762.90k  2088857.86k  2083973.46k  2101630.29k  2093569.37k

RSA

openssl speed rsa2048
CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x18c05fdef3bfa7eb 
                                sign    verify    sign/s verify/s
rsa 2048 bits 0.000239s 0.000015s   4183.0  68721.9

actually I do not know any real life use case for asymmetric crypto other than KEX or signatures.

HTTPS for fw transfer is definitely right approach to secure the transfer itself. Other than this Espressif has all the required features for signed fw in their SDK, no need to reinvent the wheel. I'm not that deep into this area, mostly use open source firmware's for my needs :)

[ayushsharma82]

@vortigont Nice info regarding ciphers. Let's see if we can get at least the firmware signing integrated. Espressif does provide features like SecureBoot V2 and encryption but for ESP32 only. I'm going to make this library supported on other platforms as well so it will be a very important feature to have.

[vortigont]

@ayushsharma82
it's going to be a real pain to fit different arch'es into one lib, especially with things like encryption, lib dependencies, etc... You might wanna think twice about the design before stepping that sloppy road :) Might worth splitting the project into separate front-end/back-end parts. Well, anyway it's just a plans, isn't it? :)))

As for compression, I've done some dirty sketching and your front-end works pretty well with my flashz lib. Although there are some gaps that needs to be considered. Maybe open a discussion thread for this?

[ayushsharma82]

@vortigont You are pretty fast :)) I'll open a discussion thread. I did some renovations in past few days which switched the webpage from ancient 'Vuejs 2' to SvelteJS framework ( 73%~ size reduction - 54KB ( old ) to 15KB ( new ) ) and changed C++ library structure.

Do you want to wait for the changes to be live in a v3 branch? Or you can integrate it in the current version and I'll port it to V3 for you. ( whichever way you prefer )

Here's a sneak peak:

[vortigont]

nice, will continue in discussion #88 thread

[vortigont]

BTW size reduction is a must. 50k is too much nowadays for a simple upload form :)

[ayushsharma82]

I decided to merge AsyncWebServer support with V3. The only problem I'm facing is that the files are getting included irrespective of preprocessor conditions. Even when ELEGANTOTA_USE_ASYNC_SERVER is set to 0, the compiler/linker is grabbing async server header file.

@vortigont Can you take a look at v3-alpha branch to see if it's getting resolved by any means? ( Maybe my includes or structure is wrong ).

[vortigont]

When lib is being build, preprocessor directives from the main sketch are not included.
It could be handled pretty easy with global build_flags in platformio, but for Arduino IDE it does not work. This problem lasts for years already with no progress. Only this is enough reason to anyone not to use Arduino IDE, ever...

[ayushsharma82]

I actually never tried it in Arduino IDE for now. Tested with PIO and it's the same in there as well which is pretty weird. I used the reference from @vshymanskyy / TinyGSM lib which also uses preprocessor directives to include appropriate library. If it works in that case then it should work in ElegantOTA as well.

[vortigont]

It works if you place all of your lib's code only in .h header files, i.e. no .cpp's. Headers are build on every inclusion and not processed if not included in any .cpp. Not an elegant solution, mostly used due to this ugly Arduino IDE heritage. With PIO you can use build_flags for the same purpose, no need to fit all code into .h, but in that case you will loose Arduino support.
I can provide an example if you like.

[ayushsharma82]

@vortigont My javascript oriented mind struggles with C++ haha.

I've had some users pointing me out earlier when ElegantOTA had everything in a single header file, that lead to people unable include ElegantOTA header more than once ( for example usage in their own wrapper class ). Also Arduino compatibility is a must as apart from my personal interest, there are many users who use & prefer Arduino IDE.

Priority 1 is to make the v3 library compatible with Arduino IDE, then comes PIO ( which will most likely work as it is bound to be compatible with Arduino IDE )

I'll try some other way if the existing v3-alpha approach doesn't work out.

[NickelNuts]

I don't know if this is needed, but reading the issues log on here, I found that the filesystem binary file for upload is supposed to be generated by the Arduino IDE (v1.8.19)compiler. When I compile for export on Debian Sid, I only get the NODEMCU binary. I can't find any spiffs binaries anywhere with Catfish. Maybe because SPIFFS is now deprecated? But LittleFS is overkill for this project and bogs me down. So if this is a real issue, and not just my ignorance, it would make sense to have a way to upload a tarball of the data folder that can be extracted to SPIFFS, or even a tool to upload files individually. EDIT! -- never mind, I found the SPIFFS.bin file location. Arduino IDE builds this in the /tmp folder on Debian. Not very nice of them IMHO.

[vortigont]

@ayushsharma82 with Arduino IDE you are doomed to misery, sorry. And if you are going to support other MCU's and their dependent libs it's getting even worse. So, better plan your way in advance.
Only 3 ways possible and I'm not sure which one is less ugly than the others :)

• all code in headers (and yes, you need to resolve issues with multiple inclusions)
• separate lib for every different arch/dependency
• teach users how to edit defines inside lib's .h files (or better to teach 'em using some other IDE/build system)

you can check this PR and it's descendants to find how deep this rabbit hole goes. Actually, I'm very skeptical about seeing this resolved any time soon.

I've gone through your code, can suggest some improvements besides flashz compression, but let's settle design questions before moving forward.

[vortigont]

@NickelNuts abandon Arduino IDE and you'll get most of your issues resolved :)
Arduino officially dropped support for SPIFFS and switched to LittleFS. If you use PlatformIO on Debian for your project than building LittleFS image is as easy as running a command pio run -t buildfs and grabbing your littlefs.bin image file from .pio/build/* dir.
For individual file management I can suggest using ftp-lib

[ayushsharma82]

@vortigont Good info. I just checked Blynk library. They also support multiple platforms and in their case, instead of using preprocessor directives to select the appropriate platform, they rely on users to include their platform's header file ( which is somewhat acceptable in my sense ) For example: BlynkSimplyESP8266.h.

What do you think about this approach?

[NickelNuts]

@vortigont Thanks, I have only been messing with these controllers for about a month or so, and have run into some stability issues with the Arduino IDE. I will look into PlatformIO. These projects I'm doing are not urgent, and mostly for my education, so learning another platform sounds like a good idea. That ftp server looks exactly what I need though

[vortigont]

@ayushsharma82 looks like you are js-biased and do not get the idea of c/cpp build system (no offense, just explaining here) :))) Builder compiles .cpp's (and .ino) files and includes headers during compilation of those files. If you write a lib than all it's .cpp's are build separately of the user code without ANY idea of what is inside users files (defines, headers, includes etc...). If inside your lib's cpp you can set guards like #ifdef ESP8266 than you are good to go, because such flags are set by build system depending on selected tool-chain (i.e. board for Arduino IDE).
But if inside ANY of your code files you have same (or different methods/functions) depending on different libraries for the same platform (i.e. async and embedded arduino http-server) than you have no way to find out which one user is going to use! So, you must provide both dependent libs to be build to make linker happy afterwards.
If you put your lib code inside .h files than builder just ignores them until some cpp files includes this header and it is going to be a user code file, where he can define some tunable for his need that could be applied to your lib's code. But this introduces another issues with multiple inclusions and slow building (building same code multiple times).
Other build systems (PIO, Eclipse, CMAKE, etc) provide instruments for user to define global build flags that that are supplied to both user and lib code during build, that way he can configure lib build-time options to his needs.

[ayushsharma82]

@vortigont Thanks for the detailed explanation, this will certainly help me in the long run 👍🏻.

If inside your lib's cpp you can set guards like #ifdef ESP8266 than you are good to go

I tried this and it worked perfectly :) we can use a combo of flags set by build system and custom definitions like ELEGANTOTA_USE_ASYNC_SERVER. I don't expect anybody to run both types of server simultaneously in the same code ( even if they do, that will probably crash the ESP as these both don't play well together ).

[ayushsharma82]

I've gone through your code, can suggest some improvements besides flashz compression

What else would you like to suggest for the v3-alpha branch?

[vortigont]

I meant some code optimizations here and there.
As a new features it would be nice to have http-client update integrated also. Like you provide URL to the firmware image via WebUI and MCU downloads and flash the image itself. Something like in Tasmota's update, etc... I have this feature in flashz lib already, but without fancy UI :)

[ayushsharma82]

You may have already spotted bits of it in the v3 branch :) , I'm working on ElegantOTA Cloud ( almost complete ) which is much secure than hosting the firmware on your own.

The ElegantOTA Cloud will feature:

1. TLS connection ( cert management will be done automatically )
2. Push updates
3. A dashboard to manage all the connected devices along with device status
4. Approximate device location on map based on IP address
5. 'Cluster' support in which you can have a single firmware source for multiple devices.

This is geared towards businesses but I'll have a forever free plan for individuals so that they can benefit from the platform for their own projects/hobby.

[playmiel]

it is possible to return a special error if it does not have enough memory to upload a file ?

[limaomerces]

Maybe my question is invalid, probably because I'm an amateur. However, how can we add other html pages to this project? Example:

Access an HTML "index" on esp32's ip, without the need for a my_ip/update

The second question is:
What is the difference between Elegant Ota and AsyncElegantOta?

[NickelNuts]

@limaomerces The difference is that one is built for the Async Web Server. The use of the /update folder by this project leaves the web server root folder available for your html code. The best way to visualize this is probably to load an "Example" for the web server you plan to use, then add the Elegant OTA to that example.

[chimeranzl]

I realise this is asynchronous, but the ota updates often lock up at various % on one of my particular ESP8266 chips that is doing some fairly heavy lifting, so i suspect the poor little processor is struggling to run multiple tasks. would there be the ability to have some sort of callback function that runs just before the upload occurs? so I could for example, halt other processing via code when an update is running?

[chimeranzl]

one runs synchronously one runs asynchronously. use async.

[Sejny]

Automatic redirect to index.html(or configurable target for redirect) after OTA Sucess and some timeout(configurable).

I have index.html in filesystem. ElegantOTA page in flash program space has big advantage. Work as rescue mode if filesystem is broken or not formatted.

[costyn]

Just inserted AsyncElegantOTA into my project today. I couldn't believe how easy it was. Really cool! Only issue was I was already using the path /update on my AsyncWebServer, which I had to rename in my project to another path. Not a big issue, but my feature request would be configurable path for the /update for a future version.

[DiverBW]

I'm a fairly new user of ElegantOTA, but I absolutely love it! Great work!

I did think of one thing I wish it did. Before I found this thread, I took a quick look at the source code and I am pretty sure this is not a current feature of ElegantOTA, so let me suggest here. My apologies if this has already been brought up before...

My ESP32 apps often use a LCD display. When the OTA update executes, the LCD just goes blank until the ESP32 reboots on the newly downloaded .bin file.

I would like to be able to flash a message on the LCD stating that an OTA update is in progress, please wait. Not sure the best way to implement this, but one would be to add the ability to register a callback function pointer. By default, this function pointer would be null so nothing would be any different from now. But if the app had registered the function pointer, the function would get called one or more times (you decide) to inform the application that an OTA update is starting, finished, rebooting, whatever. You will know far better than I what state ElegantOTA goes through internally and will know which states it may be useful for the app to know about. I also would not have the callback function pass any actual message text; I think it would be better to simply pass an integer value that has one or more values defined. Let the app itself create the message it wants to display based on this integer code, the type of output device it has access to, etc.

Another advantage of this would be for apps that have some sort of logging mechanism, or work in conjunction with other processors via MQTT. It would allow the app to log the OTA occurrence, broadcast a MQTT message, etc.

It would be SUPER nice (but definitely an extra) if information on the .bin file that is being installed was passed; perhaps file name, file update timestamp, etc. I assume (from much low-level prior experience with HTTP on other platforms) that there is probably already a way to get info on the remote client (the IP address) that executed the OTA.

[dmitrykez]

As many of you I integrated this library into my project and it worked flawlessly in a matter of minutes.
The only lack I feel in my project is that when file is being uploaded it doesn't pass any file name or firmware type inspection before flashing. The firmware type is defined by 'radio' button only.
I afraid that someone of my project users will select 'Firmware' type but upload 'Filesystem' bin file (or vise versa).
Am I missing anything?

I use version 2.2.6 with some custom modifications.
I wanted to modify JS code to perform some basic file name check, but the code seems too complicated for me.
Even if file name will be transferred to the host controller for local inspection, it will be great.

[doanerock]

@dmitrykez I like your idea and I thought I would expand on it.

It would be great if something like you would upload a zip file which would contain a config file and the firmware or file system or both. The config file would contain the names of the firmware/filesystem to be applied and a md5/sha hash for each file which the esp32 could compare to the hash listed in the config file. If the hashes do not match then the upgrade would not be completed and you would have to re upload. The config file could also possibly contain version numbers so it could only apply newer versions and not older version.

[dmitrykez]

@doanerock For debug purposes I need to downgrade sometimes.
But all the rest is exactly what I thought.

[mclane]

Hello,

I have tried your library on the ESP8266 and liked it very much. However, I would like to migrate my project to the raspberry pi pico W now. Can you provide a version for this micro? I would be ready to test...

[ayushsharma82]

@DiverBW , Someone recently created a pull request with exactly these "callbacks" added in the code. Soon I'll release it under the same v2 source.

Thanks for the comment!

[robhellstrom]

Hi is it possible to not require STA mode but say, use AP mode so that a client (eg. Smartphone could connect with the sole purpose of performing a remote upload? I have used elegant no problems in my own wifi LAN (and a commercially available alternative supporting remote OTA over the internet) but in some use cases there is no wifi LAN, or at least none with access permission or credentials known. A workaround is a small battery powered wifi router I have to carry with me; but surely this could be done in software?

[wimmatthijs]

@robhellstrom I was thinking the same and then integrate it with a WiFiManager as well so the AP mode has one entry point to set both the WiFi and do an OTA upgrade.
In my particular case this would be nice for hard to reach ESPs that dont always have a router available.
Also in regular operation i switch off the WiFi to save power and i dont even create instances for WiFi Manager etc. to have a bigger heap/stack.

[MikeF5J]

Hi, it will be great to be able to call the url with all variables already in it: IP + FW filename and then the FW update starts automatically without further screen input and button presses

e.g.
ip/update/ x??x "fw filename & path" + button submitted

By sending the URL to the browser the IP should be started and the fw file selected and added and start the OTA automatically.

Perhaps a 2nd page to call without any nice to haves or graphics even to speed up the process.
i.e. you have now "IP/update". 2nd page will be "IP/aupdate". then add on the URL the file name and the FW update starts as soon as URL is submitted

[ayushsharma82]

A bit update on the previous V3 plan. It is now divided in 2 phases.

Phase 1 is already done where I update the library to include improvements, UI optimization and gear it towards almost all wireless platforms instead of just focusing on ESP devices.

Phase 2 will commence in a few months where Cloud support will be added to the library.