SLIP MQTT protocol definition for FORTH client implementation #430
Comments
|
Maybe I have a pointer alignment problem - among many others. This produced different hard to debug effects, among them the infamous grepping through my source trees, I found
and after possibly uneven I'll try to figure out whether it is a problem rooted in mecrisp or in the I hope this finding brings me a bit closer to a succesful FORTH-MQTT-link. |
|
just replaced my reset-wire by a diode as outlined here This is not the primary cause, of the issue at hand, just a aggravation of symptoms, but may help to track down the cause of unplanned soft resets - at least I can read the last console printings just before a crash. |
|
hacked a simple string lib to proceed |
|
Making progress - hitting visible walls already. I manage it to get a message parsed by ESP-Link as supposed-to-be slip. This is what I read from the ESP-Link log. How is the CRC to be generated? this is my implementation in forth: Compared to the template, I added two Any pointer? |
|
could it be a serial communication issue?
|
|
That crc compute could be changed by different endianness in esp vs arm?
One is big other is little
See
https://www.reddit.com/r/esp8266/comments/4ve8b3/what_is_the_endianess_of_the_esp8266/
and
https://electronics.stackexchange.com/questions/183021/stm32f103c8xx-big-or-small-endian
…On Tue, Jan 29, 2019 at 10:18 AM wolfgangr ***@***.***> wrote:
could it be a serial communication issue?
ran the examples at 460800
switching back to 115200 , ruined my test example, so I have not exactly
sam strings.
but.... what puzzles me is that the trail of the strings are responded
differntly in both cases
I list below
- last 6 bytes sent
- crc reported as bad by ESP-Link log
- last bytes as responded by ESP-Link log
But obviously, the CRC arrived correctly at the ESP-Link - this would
not point t a communicatin issue
01 00 00 43 F4 C0
rcv=f443
\01\00\00C\F4
01 00 00 A3 C4 C0
rcv=c4a3
\01\00\00\A3\C4
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#430 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ab_a_QvEdYbwIspdL8D3I8UN08tC1ATmks5vIAPKgaJpZM4aU4pg>
.
|
|
I'm just thinking into endianness regarding the last step - putting the CRC result to the message string. Both the C and my FORTH code are implemented in integer math. |
|
The forth side is easy to examine. How can I inject a test code into ESP-Link implementation? |
|
Are there examples of just one crc step, so that I could cross check my incremental bytewise updater? I think I'll try to untwine the SLIP wrapper layer from the MQTT inside. But for the moment, live comes into the way.... |
|
If you-re not familiar with ESP development tools you can compile an
arduino sketch for ESP8266 and print CRC intermediate results incrementally
…On Tue, Jan 29, 2019 at 10:55 AM wolfgangr ***@***.***> wrote:
Are there examples of just one crc step, so that I could cross check my
incremental bytewise updater?
Hm, looks like I have to dig for a C hello world an paste the code in?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#430 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/Ab_a_WpWXP9m9LI3imbC0j66QJN0dNZ8ks5vIAxkgaJpZM4aU4pg>
.
|
|
Your forth CRC code looks correct to me. You need to make sure that the |
|
I found my error. Now I got the message: I interpret it that way that I have got the SLIP thing right and can proceed with MQTT now. |
|
Making progress: first succesful MQTT :-) ESP-link debug log: and this is my console running |
That's a common issue in the FOSS world. So let me add my 3 cents here.
so this breaks up as follows?
Have not yet screwed my forth interpreter to process responses coming along the same line I use to talk to it. But should not be that difficult, I expect. When I process the answer, is it ok to consider anything enclose between two C0 as SLIP (well, if not excaped...)
Good to know. And I suppose I have to remove it from responses as well, OK? I'd like to keep things simple. Can I avoid nonprintble characters alltogehter? |
|
OK, next let's dissect my first succesful washer test string. This is how it looks in memory: Ouh, should I push my git to github to refer to the code that did this? anyway... My code template says 'Command to sync up the esp-link '
Next is a string literal, consisting of a 16-bit length The next one There is another one (not used in my first example) The last one I simply copied without any clue. I really would appreciate some simple manual on the standard.
|
voila: preliminary hack of a buffered string library the constants I referred to: the hot spot - work in progress while I write in this issue |
|
@wolfgangr nice work, but why FORTH? |
FORTH provides - to my knowledge - the largest spread between extremes on a couple of dimensions:
So on this question you could blow every thread, and there are enough of these discussions around. |
|
back to start again? - Well, hitting a different wall, now. But does not work. So obviously the command got processed by the SLIP layer and handed over to the MQTT_PUP layer. |
|
For reference, again the log in the case of success ... and of current fail: we have a line 140: So I think this stanza generates the successful debug message and just after calls the handover to the MQTT server. However, I can't see where the fail message |
Is there any hope that this other project has some docu we could build upon? I see this (C)
but only referring the outgoing calls (ESP-link -> MQTT-Server) --- edit --- OK ? - I found the correct standard aka haystack and just heve to look for the needle... |
|
https://github.com/jeelabs/esp-link/blob/v2.2.3/mqtt/mqtt_cmd.c 105: 113: 120: my ESP-link keeps sending mqtt state messages like I see no premature returns after this last check and the succesful debug message. |
Line 120 in fe4f565 Line 159 in fe4f565 I don't see such backdoor. |
|
https://github.com/jeelabs/el-client
404: Page not found It refers to doygen. So the docu lives in here? hm... a first glance tells me that is a layer beyond what I'm looking for. @tve |
|
OK - hope disappointed - back to track. who is printing E:M ?
who is calling MQTTCMD_Publish ? Line 74 in fe4f565 in Version 2.2.3 this line sits at 41: - 33 lines less Thorsten, @tve '============== this is a line 76: (in v2.2.3) : we can search for the integer index CMD_MQTT_PUBLISH on our command table as well .... and find.... Line 49 in fe4f565 |
|
https://github.com/jeelabs/esp-link/blob/v2.2.3/cmd/cmd.h |
|
https://github.com/jeelabs/esp-link/blob/v2.2.3/cmd/cmd.c#L83 Let' recall our log: OK, so we have found the point BEFORE the error encurred. hm .... Looping closed and overlooked the point? |
OK, down again the rabbit hole and collect the crumbs on the way. https://github.com/jeelabs/esp-link/blob/v2.2.3/cmd/cmd.c#L89 This resembles our last valid debug print, so we know for sure (+-) that So somewhere between |
|
E:M means out of memory for alloc |
|
need to find some readings on forth for uC. any link? |
|
I see forth is not so easy ;) |
|
OK, after som googlin & RTFS, I try this hack
|
|
added After normal response, this sequence is inserted: compare what the debug says: I don't see any immediate SLIP'ed response after submitting a MQTT subscribe. This are responses to subscribed messages For the moment let's focus on the format of the messages: This is really weir'd.... |
obviously the internals of ESP-link or the serial line itself get screwed by The response to setup now reads: which I'd dissassemble as
Now the subscription callbacks read manually regrouped as follows
which is not new to us any more I can remember that with 0x0a in the callback, the ESPL-Log displays "cmd=3" |
|
So we need a two layer parser:
special case: empty data field: we see that
|
|
here #430 (comment) I found
here #430 (comment) I knew
I still prepend this to any of my commands. let's recall the Master's words: @tve wrote here #430 (comment)
Have I learned to understand all that words now?
|
Well, yes, it's more complicated than a pipe symbol |
|
@tve
It's like |
|
Spent a hard night with stalling reads. I hoped I might get closer to the metal, ... and Well, I think this is bare metal... So what? O do we have some kind of flow control? |
|
neither sending 0x0d nor xon aka 0x11 helps. |
|
here a transcript of debug data: in plain and with explanation: Do I have a echo problem? lets have a look at the ESP-debug log:
Hm... I'd expected ESP-link to either complete SLIP packages or not start them at all. |
|
buffer overrun? So no blocklength, but a timing issue? |
of course not, if I do not have to. code snippets related to hardware adresses:
So let's hope that USART1 is hardware equivalent and has the same registers. and of course in the datasheet of the manufacturer Hints for the hook addresses for forth code as interrupt handlers we can check on the console: Obviously both interrupts are unassigned and wait for our actions. There is this And there is the BluePill's STM32F103C8T3, in contrast, is labelled as Cortex-M3 by STM. So let's RTFI..nternet and find on page 199/1134 (admittedly haven't read them all, but I know ESP programmers would be happy with even a tiny subset of that ...) we find on a 'Table 61': This matches the USART2 settings from embello, whom we trust, of course, so we hope we can trust this table, too. Cleaning up my browser windows, I'd like to save this link:
|
|
first draft: |
|
This is weird. just hacked and run it on the console: Can't believe it. Hacked. Works. Full stop :-)
@uzi18 Therefore :-) |
|
nice, but have no time to understand ;) http://hightechdoc.net/mecrisp-stellaris/_build/html/index.html |
|
I think that there is no input buffering in stock mecrisp, since the ASM-primitves read directly from USART registers. The forth-compiler aka REPL seems to be fast enough to read that without loss, but obviously not my approach to extract SLIP-encapsulated MQTT-frames from the input stream. The STM32 hardware would implement RTS/CTS flow control, but on the ESP-link side, this is still on @tve 's to do list (not on the very top, I'm afraid...) I have now (configurable) 128 byte of ring buffer, so I hope I can read data as fast as it arrives and have enough time to process then. |
|
so add buffering/dma auto fill |
Actually I had expected that this all had long be implemented by @jcw . And on the embello github I find credits for both mecris-stellaris-Wizard Matthias Koch and ESP-link chief maintainer @tve Thorsten van Eicken. I can only hope that @jcw is not going to make a living with stuff I'm going to publish right now.... |
I hope that this is what I have done. This is a bit of the the upside-down-wrong way around, made it hard to code/understand/debug. I could hook into the USART-IRQ handler, but I've learned that's bad programming practice - IRQ handlers are supposed to stay as short as possible. I could use the multitasking framework, but this I'd like to kee open for application development. But I think there are 1-ms-systick interrupts as well. |
? When I have REPL or SLIP stalls due to communication quirks, I'd like to keep ram state and just do a simple REPL 'quit'. Clear stack and resume user prompt. Any errorneous SLIP-message might be silently ignored. Well, maybe it's a good idea to create a syslog entry. |
|
Plan: Preliminary considerations regarding buffer size:
But there are still too many open quetions for a final decision |
|
Let' start at my example
Similiar tasks may typically be implemented as MQTT events, triggered by some MQTT subcription message, instead of a fixed periodic schedule. So let's dig deeper into the anatomy of the mecrisp-stellaris multitasking framework:
We find that
To pin down the concepts with as little distractive complexity. we refer to Matthias Koch's "blinky" examples
All methods have their merits, whose pro's and con's are beyond consideration at this point. However, all of them require some tiny code whith the message as parameter kicked off at successful parsing completion of a SLIP message. So my idea were:
The cooerative task mechanisms then require some kind of message passing that might be triggered at the end of a systick routine that completes a message parse run. The interrupt handler will then only leave the required message chunks and maybe some flag in some kind of message box. Any stalling / blocking / long runnig routines are forbidden as they may end up in system instability. Detail can be found in any decent multitasking tutorial. A simple one we find here - search for "Multitasking on the Quick" |
|
======================== draft 2019/02/05 08:10:27 ===================
|
no doubt, DMA would have the great advantage that once it runs, we didn' have to care whith IRQ and timing and task management interference issues any more at a later point in time. I see that I get distracted from by first goal quite a lot, but may be this is skd of nasty work that simply has to be done... So I just tried to read into the basic nuts'n bolts of STM32 DMA What puzzles me is the first sentence on pg 278: That's a sad thing, because that's what would have made a ring buffer implementation quite straightforward. My dry swimmers considerations for workarounds: For RX, For TX, write-pointer - DMA_CNDTRx = read-pointer (sorry, no RPL...) We have to increment this value every time we add to the ======================== After scanning the STM manual sections for DMA and USART, I think
Essence:
|
|
Live comes into the way. Refocus: Top goal: required for basic funtionality:
required for quality functionaltiy:
nice add non feature:
|
looks good: both RX and TX are handled over IRQ-driven ring buffer: next step is 'SLIP-decoder in the input stream' Can I include the slip-decoder in the IRQ-handler? |
|
draftet some kind of state machine:
hav been there a week ago :-((( |
|
Have an eye on the reference implementation: https://github.com/jeelabs/el-client/blob/master/ELClient/ELClient.h#L100 and here I i see that this is their state machine: So they delay the untwining of SLIP/non-SLIP to the loop(), prepending a ring buffer ( I suppose that's what a 'stream' attached to a serial line boils down to) Do they have to provide for a situation where SLIP command processing stalls while ohter chars have to be processed? https://github.com/jeelabs/el-client/blob/master/ELClient/ELClient.h#L105 OK, forget it. Well .... does ESP-Link end us commands at all ???? |
Inspired by this hack
https://hackaday.com/2017/02/13/hacking-on-the-weirdest-esp-module/#comment-5820116
I tried something similiar:
MQTT from an STM32 Bluepill running mecrisp FORTH, connected to ESP-Link.
However, when I try to issue test strings, the communication between ESP-LINK and the STM32 breaks down. I have to reset both the µC at http://192.168.1.88/console.html and ESP-Link at http://192.168.1.88/log.html using the HTML-Buttons at those pages.
I suspect that some wrong command string confuses ESP-LINK and renders it in a non-responsive state.
I assume that the communication ESP-LINK <-> µC is neither pure MQTT nor SLIP but a dedicated protocol, correct?
Unfortunately, there is no MQTT in https://embello.jeelabs.org/flib/ yet.
I have repeatedly searched the issues and all docs for some spec, but the only source I found is the
https://github.com/jeelabs/el-client/blob/master/ELClient/examples/mqtt/mqtt.ino
However, assembling test strings from reverse engineering this very error prone, too.
It would be great to find a couple of preassembled test strings to ensure that the whole setup is working.
So I could work down to more complicated functions by small changes.
Just some "hello" appearing on my mqtt would be great.
Details from my setup:
esp-link {"rssi":-60, "heap_free":19992}on my mosquitto_pub,so the ESP-link <-> mosquitto is working
I'd plan to rework the whole thing using string buffer (hope to find such in mecrisp) instead of blowing the stack.
Would be glad to share, of course.
The text was updated successfully, but these errors were encountered: