Hosting webrepl from ESP8266 - feasible?

[cefn]

Increasingly impressed with all the work on micropython and looking forward to incorporating it into what we do at @ShrimpingIt ! Trying to bottom out a few issues about how we should pre-flash ESP01 (1Meg) and NodeMcu v2 boards for the greatest possible accessibility for our learners.

Being able to program them standalone from a laptop, tablet or phone is really powerful. However, currently I have been visiting http://micropython.org/webrepl/ then switching Wifi network to be able to connect to the ESP8266 module. This will not be possible for all our learners and ideally having a mobile phone would be enough to be able to program or issue commands to these devices.

This suggests we should attempt to serve the webrepl.html and supporting files direct from the ESP8266. However, I am surprised not to see this configuration documented anywhere. Is there a reason this hasn't been attempted which I can't yet fathom?

Is it worth putting together a module which serves the browser interface files ( https://github.com/micropython/webrepl ) on some port or is it somehow incompatible to host a webserver on http://192.168.4.1:80 at the same time as the repl socket on ws://192.168.4.1:8266/?

[dpgeorge]

Is it worth putting together a module which serves the browser interface files ( https://github.com/micropython/webrepl ) on some port or is it somehow incompatible to host a webserver on http://192.168.4.1:80 at the same time as the repl socket on ws://192.168.4.1:8266/?

In principle this should work, and you can probably even do it in pure Python already. I doubt it will ever be part of the main release because it uses a lot of resources: an extra port for HTML server, RAM for the server, and at least 60k for the WebREPL client files.

[cefn]

Thanks @dpgeorge thought there might have been some technical reason I couldn't find examples of this. Maybe it's only been feasible relatively recently.

I gather that a dos-fs image can be inserted at a specific point in memory to make a whole filesystem available (meaning it would be easy to sync everything from https://github.com/micropython/webrepl and other sources after flashing micropython).

Thinking I might therefore use a loopback filesystem to marshall all needed files into the final image. However I couldn't find much detail about the approach needed to push the dos-fs image itself. Is there a reference/example/discussion for this? Guess it's just a crafted execution of esptool pointing to the image file?

Which of the filesystem parts need to be in such a dos-fs image - e.g. can it just contain my main.py and my other modules, or does it have to replicate chunks of the original scripts like _boot.py or boot.py in order to come up successfully?

Alternatively is there a better way to 'synchronize' a filesystem? My fallback plan is to write a python routine which traverses a local folder and throws file upload requests via webrepl_cli.py assuming an empty filesystem.

Any concerns with these proposed approaches, e.g. wearing out the memory if I'm making small changes yet syncing the whole image when prototyping?

By all means point me to source or other places where I can find this out for myself, but so far my searches about wrangling the micropython filesystem portion of the ESP8266 memory (except through one-by-one upload/download) haven't been successful.

[hoihu]

This suggests we should attempt to serve the webrepl.html and supporting files direct from the ESP8266. However, I am surprised not to see this configuration documented anywhere. Is there a reason this hasn't been attempted which I can't yet fathom?

You might have a look at a pending pull request:
micropython/webrepl#1

the approach there is to use a feature in HTML5 which allows you to use webpages offline (using a cached version). This way it's possible that the phone/tablet can connect to the esp8266 by using the cached webrepl.html (It will use the cached version if it thinks it is offline)

It's not merged because of concerns that not all browsers are compatible with. I heard you need to explicitly disable all possible Internet connections (3G/4G) because otherwise the phone will try to establish a connection over this connection instead of the wifi and hence will not establish a connection to the esp8266.

If you try it out and give feedback, that might convince some of the maintainers to have another look at it.

[cefn]

@hoihu thanks, it's great to hear about all the alternatives, but the issue I'm aiming to handle is that a learner is unboxing a pre-flashed ESP8266 and either...
a) has no internet access
b) could do without the challenge (and complex mental model) of loading a page from a remote site, then switching Wifi networks, then connecting over websockets etc.

After working with the module for a while, we'd hope this could make sense to them, but ideally the beginning would be an easy on-ramp. It's also a good way of avoiding the inevitable 'it doesn't work' bug reports @ShrimpingIt which are to do with someone failing to perform all the steps.

Connect to Wifi, visit URL is a reasonable series of steps. All the alternatives seem nightmarish from a ship and support point of view. I imagine we are likely to need to support a second configuration; guiding through the ESP8266 negotiating an IP on the household's Wifi SSID (static or DHCP), but equally this should be common-sensical as a sequence and simple ways should be documented to ensure success.

See http://start.shrimping.it/kit/ and http://start.shrimping.it/#project for an idea of the way we'd like to package pre-configured ESP8266 modules and combine them with components and craft materials to make accessible projects for learners and educators - the same way we have for ATMEGA328 Arduino-compatible projects.

[hoihu]

Actually I was looking for a similar thing.

Deployment of ESP boards in an environment where no WiFi connection is available is quite reasonable (imagine a remote wheater station). It would be great if this station is still configurable over your tablet/phone, directly via the browser.

Of course you could always skip webrepl and write custom http pages to handle configuration but ideally you'd want to keep REPL functionality (especially!) in remote areas.

Caching of webrepl.html seemed like a valid alternative to self-hosting. But it's cumbersome, I agree, and not ideal for your application.

One of the problems I see is the terminal.js file which consumes app.50k - you'd want to serve that somehow from a frozen ROM region but I haven't seen examples on how to serve static contents like webpages from ROM.

[dpgeorge]

I haven't seen examples on how to serve static contents like webpages from ROM.

It would be nice to have a frozen filesystem that allowed arbitrary file types (not just scripts). But there is a way around this limitation at the moment: put your txt/html/css/js/etc file as a string in a Python script, asigned to a variable, and put that script in the esp8266/modules/ directory. It'll then be frozen when you build the firmware, and you can access the entire string via the variable name (from Python, after importing said script). The data for the file will stay in "ROM" and not go via RAM.

[dhylands]

It would be nice to have a frozen filesystem that allowed arbitrary file types (not just scripts).

That's what memzip is.

[hoihu]

looking at #1569 it is not clear to me if memzip is supported if both modules (frozen&memzip) are used? Is it "one or the other"?

assign it to a python variable is an option. but it would need some hacky glue code to serve the "files" I believe? (E.g "if browser requests terminal.js then return content of xxx variable")

a pure RO filesystem (aka memzip) would really make sense for web contents.

[dhylands]

I use memzip in teensy. Currently it's an either/or, but I think that's because of this one file:
https://github.com/micropython/micropython/blob/master/lib/memzip/lexermemzip.c

I think you'd just need to modify the esp8266 mp_lexer_new_from_file function to support memzip.

[deshipu]

Correct me if I'm wrong, but the filesystem on the ESP8266 (and on the pyboard too) supports arbitrary file types, not just scripts. You can open() them and read() and write() them as in normal python. All that is missing is someone actually sitting down and writing a simple http server that serves them (preferably in small chunks, as to not use too much memory).

[pfalcon]

That's what memzip is.

No, that's what standard Python frozen modules pseudo-filesystem is. There's a standard API for accessing data files, etc. It's not fully implemented for uPy because nobody needs it much enough:
#1346 . (And those who seem to need, instead spawn non-compliant solutions like memzip, instead of implementing Python.)

[cefn]

OK, I definitely didn't understand the nature of filesystem as provided via native open() and manipulated by upload/download through Webrepl and webrepl_cli.py, then.

I understood it was simply accessing a specially formatted area of the flash memory and operating as a real filesystem, but that doesn't seem consistent with @dpgeorge proposing declaring string variables to avoid (RAM?) impact when placing files in the filesystem. Or is this to do with the filesystem not really being a filesystem, and therefore any files having to be loaded monolithically into RAM to read them?

Looking at http://docs.micropython.org/en/latest/esp8266/library/uos.html?highlight=uos#module-uos it seems to suggest there is a /flash filesystem. I gather this isn't read-write, then, and may be limited to only .py files? Confused by the assumed context of this discussion and struggling to find documentation indicating the architecture.

Do changes to the /flash filesystem through open() write() persist through resets? That's where I'd expect to put a directory of files to serve a bit like SimpleHTTPServer. If it's not a filesystem I can manipulate from my laptop OS (e.g. a loopback pseudo-block device), then I could write a synchronisation routine via listdir() and webrepl_cli.py calls negotiated over serial.

@deshipu the conclusion I draw is that there is a RAM-oriented filesystem which therefore has only very limited capacity and is best avoided for anything but small scripts/data. In addition there is a ROM-oriented memory area which is 'frozen' in that it's not expected to be modified at runtime, and is managed by pre-compiling python bytecode from a collection of modules defined at build-and-flash time. Confused as to how this appears as /flash if it's only capable of servicing bytecode, though. Seem to be a some major things I'm missing.

Other research

Things I've puzzled about to see if there's a pure-python way to access a block of bytes in flash as a filesystem - looking at pyFilesystem, pyfat and vfatshuffler via FAT32 driver in python and Construct's ad-hoc FAT16 driver and ext2 driver though earlier versions seem to have support for FAT32 and Ext3 which have now been pulled.

[cefn]

This forum comment indicates the approach I was originally expecting to use, and which I thought would allow me to insert arbitrary files in a pseudo-block-device filesystem which could be manipulated on my laptop via a loopback mount (rsynced helpfully from various other folders, when experimenting with the build), uploaded as part of firmware flashing and later be read as files through listdir(), open().

[deshipu]

@cefn there is a normal read-write fat filesystem on the flash on the esp8266, where you can create, upload, store, modify and delete files, which is persisted between resets and even (as long as the memory map doesn't change) between reflashings of MicroPython versions.

[dpgeorge]

If you want to create an ESP8266 image that contains a filesystem with certain files (eg WebREPL client) then you can do it by first uploading the stock firmware image, then adding your files (eg via WebREPL), then download the entire flash image to a local file on your PC (excluding the last 5 sectors, which contain config data). Then you should be able to upload this (big) firmware image that includes the filesystem, and you'll have a cloned device.

[cefn]

Thanks @deshipu that's really clear.

@dpgeorge I think I misunderstood the relevance of your suggestion to get the files into the 'frozen' filesystem, then.

Why would I do this if there's a regular fat filesystem which provides equivalent functionality, but where files can be treated as files? Or is it to do with the convenience of following the established tutorials for 'bespoking' the firmware flashing process, which are all module-centric, and where the fat filesystem is treated as something to be messed with at runtime (e.g. the behaviour to write /boot.py in inisetup.py)?

Is there no way to manipulate the filesystem as a pseudo-block device on my laptop against a locally stored file initially dd-ed full of zeroes, then formatted with mkfs.vfat (is this the right format), then mounted as loopback? Of course to get the filesystem onto the ESP8266 module, I'd need to write the file containing the pseudo-block device with the proper offset and length using esptool.py after unmounting/syncing.

If this is possible I'd really benefit from guidance how to get the proper format, offset and length values as it would be by far the easiest way to work.

Side thought

Love the suggestion @dpgeorge of reading the flash image then writing back to 'industrialise' the preparation of these modules. I wonder if I could configure an ESP8266 module to do the flashing (writing bytes from its own memory over serial, with another module in flashing mode).

Reference Server code

For reference, This source indicates the approach I was expecting to adopt for a minimal static HTTP server (even simpler than SimpleHTTPServer ).

[mkarliner]

Is the webrepl protocol documented? I'd be interested in providing other clients that use it.

[cefn]

@mkarliner probably best handled at https://github.com/micropython/webrepl/issues as this thread relates to the narrow case of hosting the webrepl files from an ESP8266 running micropython. See especially the protocol commentary on the repo landing page.

[cefn]

Aha, looks like the information defining the offset and length of the VFAT filesystem is within the initialisation python code at https://github.com/micropython/micropython/blob/master/esp8266/modules/flashbdev.py so I should be able to calculate the dimensions of the pseudo-block-device and the offset and length needed to write with esptool.py from that, with any luck, then work via loopback.

[hoihu]

@deshipu I think the issue was that the OP wanted a cloned or flashed device without using webrepl (and initial wifi setup etc) for transferring files. Principally though I guess you are right it's just a matter of doing, all the components are available.

Using a memzipped or frozen region for read only content is more reliable than /flash. /flash has the danger that it can get corrupted when something goes wrong during flash Write. I guess another benefit is that you can place your frozen staff anywhere in ROM and not using precious read/write space from /flash.

So from the reliability point of view a read only FS makes sense. Maybe it can be used as an SOS recovery section: if /flash is corrupted, boot from read-only FS (/sos? 😉)

[cefn]

precious read/write space from /flash

OK, I figured out some details from https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-beautifully-easy-iot/posts/1501224 which help to suggest how there is a big deal about which different parts of the FlashROM data is written into. Some of this is starting to make sense now.

Couple of questions, then.

Which one of...

BootROM -- 64KB -- InstROM
Data RAM -- 96KB -- DataRAM
Instruction RAM -- 32KB -- InstRAM
Flash cache -- 32KB -- InstRAM
Flash code -- 1MB -- Mapped via InstRAM cache

...is the /flash filesystem in, and which one is the 'frozen' stuff in?

Can I indeed write a dos-fs filesystem image from a given offset to populate the /flash filesystem? or any other path.

How can the remainder of the Flash memory be accessed/used as part of a micropython setup on ESP8266 (there's 4Meg on the NodeMCU modules I'm playing with). If so, how?

Really sorry if there's meant to be a reference for this somewhere, but I simply can't find it.

[dpgeorge]

The filesystem and uPy firmware (incl frozen stuff) all live in the large, flash code segment (1mb in the above post).

The layout of the filesystem is configurable. The set-up is written in Python (see modules/_boot.py and modules/flashbdev.py) and you can change it to what ever you want. Eg on a 4mb device you can use just the last 1mb for your filesystem, and the middle 2mb for other data in whatever format you see fit (access it via esp.flash_read, esp.flash_erase and esp.flash_write). In fact, pending #2140, you could potentially mount 2 indpendent filesystems, one of them read only.

Can I indeed write a dos-fs filesystem image from a given offset to populate the /flash filesystem? or any other path.

Yes. As long as it has the correct parameters (4k sector size).

[cefn]

Some further thoughts.

@dpgeorge

at least 60k for the WebREPL client files

Just experimented with a combination of Uglify, Tar and GZip and assuming that the files webrepl.html, FileSaver.js and term.js are enough to deliver the WebREPL UI, then it's possible to get WebREPL down to 19k.

The thinking is that a NodeJs script could be written which uglifies and composes all the JS inline into a single webrepl_ugly.html file, then the whole stream of this file could be cached within the (runtime or frozen) file-system as gzipped bytes so that clients supporting Accept-Encoding: gzip - considered universal by caniuse.com - can be served the pre-compressed stream, (possibly including headers), given a matching inbound request.

Wondering the best way to get a 'reference' stream is to sniff the transaction between a real 'browser' and gzip-supporting server (after uglifying) then playback the bytes destined for the clients from the ESP8266 server. Might be possible to automate this whole packaging operation in Node, combining uglify, express, compression and http.request, starting with a checkout from the WebREPL repo.

[mcauser]

I used a similar node (grunt) flow to compile the html,css,js,svg and mp3s on my useless throwie.
Compiles sass, concats, uglifies, base64s, inlines, minifies and gzips.
Using inline css, javascript and base64 mp3s so that the entire page can be served as a single request.
17380 B source, minified to 15362 B, then gzipped to 8755 B
https://github.com/mcauser/esp8266-useless-throwie
https://github.com/mcauser/esp8266-useless-throwie/blob/master/src/micropython/hello_gzip.py

Building the grunt workflow is easy. Convincing the python guys to use it, now there's a challenge!

[jumasheff]

@cefn Hey, did you find a solution? If so, can you write a blog-post about it?

[cefn]

Will be focusing on this soon, and it's a big emphasis of my work to document technology build and config sequences publicly and share under CC ( I am Instigator at http://shrimping.it )

[cefn]

OK, I've now completed the first draft of this work and have a functional WebREPL served direct from http://192.168.4.1:8080

You can find the example inlined and gzipped files at https://github.com/ShrimpingIt/cockle/tree/master/replserver along with a deploy.py utility which runs on an Ubuntu laptop to put both the gzipped webrepl source and a minimal server main.py in the filesystem, causing the device to boot-to-webrepl. This depends on adafruit-ampy being installed via pip.

I tested on a NodeMCUv2 which was configured with Micropython 1.8.6 following the retrieve.py and deploy.py firmware-wrangling routines found here . This depends on esptool being installed via pip.

Probably changing the port to match your platform will make these work on Mac and Windows too. That's the long-term aim.

For reference, I created the webrepl inlined gzipped file using npm's inliner package, with the result put through gzip, from within the latest checkout from the webrepl repo, like...

inliner webrepl.html | gzip > webrepl-inlined.html.gz

This work is expected to be properly documented and written up as a more accessible blog for January, given the current workload. Good luck.