mqtt: support nested JSON values #1779
Conversation
JSON values from ebusd are published on MQTT with values "boxed" in a
format like: { name: xxx, value: xxx }
Support this structure by allowing deeper-nested JSON structures, and
unpacking the value and adding the name if present.
Ensure non-scalar values (nested JSON, etc) are not recorded, or logged
without being converted back to JSON.
| @@ -279,7 +279,7 @@ function message($message) | |||
| $time = $timestamp; | |||
| } | |||
| } else { | |||
| $log->warn("Time value not valid ".$inputtime); | |||
| $log->warn("Time value not valid ".json_encode($inputtime)); | |||
There was a problem hiding this comment.
What difference does the .json_encode make?
There was a problem hiding this comment.
Because $inputtime is derived from json_decoded values it can be a non-scalar type, an array or whatever, and attempting to print it was killing the whole script. Specifically because the source I'm using sometimes emits JSON that happens to contain "time": { "value": xxx } although this is the controllers current time of day, not the event time - discarding it is fine, exploding the whole script less great. :)
| $value = $value["value"]; | ||
| } | ||
|
|
||
| if (is_scalar($value)) { |
There was a problem hiding this comment.
I quite like this check, but might I suggest it should check for a number instead?
There was a problem hiding this comment.
That would be a behavioural change compared to currently, where strings (or whatever) are sent down the line and then presumably dropped by emoncms. :) This check was mostly just to "balance" the consequences of increasing the JSON depth because it could let in other JSON structures that were not { name: xxx, value: xxx } structured.
I don't want to start discarding strings here because some of the values reported by ebusd are truthy values (on / off / true / false / yes / no) which can be safely mapped to 1 and 0, which I'd like to add in another patch. However, here is not the right place to do it, as not all such values follow this JSON path, because it's predicated on JSON and an array - a bare "true" or "yes" value follows the other path. So I think a follow-on patch would need to add that truthy-string test somewhere lower down before they are recorded.
(Separately / additionally, there are also textual enums reported by ebusd - but can't be handled generically here so I don't quite know where I should be thinking about that.)
There was a problem hiding this comment.
Yes true and yes sort of dropped later, but often create bad inputs as a result so ensuring only numbers are sent on would be a really good thing (but could be proposed as a separate PR).
There was a problem hiding this comment.
Would be nice to know if there's a possible approach for keeping the strings in the pipeline, if we could add a feed function to do matching or mapping somehow...?
|
I am trying to think where the acceptable JSON structure is documented. It may not be, as it is currently the same for the HTTP interface as well as the MQTT interface. To be consistent, you may need to update the HTTP interface for I did some work on this a while back as originally, it did not accept valid JSON. |
|
Can we presume you have tested this with 2 level JSON so it doesn't break anything :) |
|
Yup, input API here emoncms/Modules/input/input_methods.php Line 98 in d9264ec |
ahem taptaptaptap Yes that appears to work :) |
My life resolves around "Assume nothing"... |
Opened an issue #1780 |
Happy to update this but (and perhaps giving us one place to decode "truthy" string values and rejecting others) these two close-but-no-cigar functions are ideally looking for a couple of utility functions for fetching time and numeric values from submitted strings. Is there somewhere sensible to factor those out to? |
@TrystanLea will need to comment on that. |
|
Ping @TrystanLea - looking at this again (trying to remember and make it actually work...!) - is there somewhere I can move the JSON-to-time-and-number functions to so they are not duplicated between the MQTT code and the HTTP input API? |
JSON values from ebusd are published on MQTT with values "boxed" in a
format like: { name: xxx, value: xxx }
Support this structure by allowing deeper-nested JSON structures, and
unpacking the value and adding the name if present.
Ensure non-scalar values (nested JSON, etc) are not recorded, or logged
without being converted back to JSON.