Configuration Specifications #104
Conversation
|
Hi,
thanks for working on this! I feel I should point out, that configuration
specification patches likely won't be merged without also having the
code availabled, which is using them. Otherwise things probably would
get horribly out of sync. After all people do submit patches changing
and introducing configuration items quite frequently (compared to our
over all development pace) and can't be bothered to update files that
aren't in use.
You might want to coordinate your efforts with the other people on your
team (I don't really know who they are) or maybe even pester them into
shareing some preliminary PRs like you did. :)
As for your questions:
+- [ ] Some ports have ranges like this: 0x200 - 0x400
+ - Is there also ABCDEF possible?
I'm not sure I understand this one. The prefix clearly indicates an
hexadecimal number, I do assume all hexadecimal digits are valid. Why
do you have doubts?
I think this refers to LPT port io addresses on x86 architecture. There
are only very few common values, 0x3BC is one of them.
+- [ ] DriverPath
+ - Where exactly are those? `server/drivers` is not available on my system.
Maybe /usr/lib/x86_64-linux-gnu/lcdproc?
"server/drivers" is a relativ path, used when testing in tree compiled
lcdproc without installing into a system location. Configuration files
installed into the system need to have the path updated to the proper
location for the system.
About questions related to keys:
Keys in lcdproc are (almost) free strings. You can define in the drivers
section an arbitrary key and use it as a value elsewhere. Also using a
key that doesn't exist or defining a key that nobody uses are not errors.
I can't help you with most of the driver specific questions. You will
have to either look it up in the code or use "git blame" and pester
the people who last changed the offending lines. But also note that
many of the drivers are quite obsolte and unmaintained and it might
not be worth putting lot's of effort into them.
Also I don't know how to answer many of your questions about valid
values. While there are often ranges that are (not) sensible, like a
timeout of a million years, it's not clear to me where to draw the line.
Also I'm not convinced that software should decide which timeout is okay
and what is ridiculous. Also which value is okay technically and which
might cause an interger overflow or something is platform dependent. I
think such information should be added when converting the source code
to libelektra, if at all.
An other problem with validation is, that many enumeration values have
synonyms. (I try to cut down on them, but for some reason people love them
and submit patches adding even more of them.) I guess sometimes we haven't
been consistent about where we document the synonyms and for some settings,
like hardware model, values, that in one version of lcdproc are exactly
the same, might become different in the future, when more features of the
hardware are supported. I have no idea how to handle this with elektra
or configuration validation in general. You are the experts here, but
if it is a hassle, then being very sloppy in validation is not a problem.
+## linux_input
+- [ ] `#key=1,Escape .. #key=28,Enter .. `
+ - I have no idea what this is and how it is set
Yes, this is a bit quirky because ini is not arbitrarily hierarchical.
If we were using libelektra from the beginning, this would be easier...
There are multiple configuration items, each with the key "key" and a
string value of the format ("%d,%s", linux_key_code, lcdproc_key_name).
I think this is somewhat similiar to an array in elektra?
+## lis
+- [ ] VendorID/ ProductID
+ - validation possible? eg regex?
Um, I guess you would need to look up the USB spec for valid values?
HTH,
Harald
|
|
Hi Harald, thank you for your quick answer!
I definitely will, but the colleague is currently on vacation ;)
I do not come from the C field of programming and do not have anything to do with hex numbers. I just wanted to ask to be sure.
Do you have a list of these drivers? If something is particularly complicated in an obsolete driver I will simply ignore a validation. Otherwise I try to support it :)
You are right, I added some validation to at least prohibit negative values / false types.
Can you provide an example of such a (not well documented) synonym usage?
Ok, so the value is clear that it always has this format. But the key is still unclear to me.
I could probably do that but I think the maintenance effort to maybe synchronize with new USB standards outweighs the possible benefits of users being to sloppy when typing in a wrong ID. I removed some questions now. Can you look over it for a second time and see if there is anything Otherwise I will use the git blame and pester your fellow contributors ;) Thank you for your help! Best regards, |
|
i can't add much here, but with regard to USB VID and PID, trying to somehow validate against a list of existing devices would be a fruitless exercise, however the USB standard allocates a 16 bit field to each of the VID and PID, as such the simplest and arguably most useful validation would be to ensure that it is a valid hexadecimal number (but maybe without the 0x prefix? not sure what the drivers look for, but would be clearer if the prefix is always present) between 0x0 and 0xFFFF |
|
Hi, With regards to some of your questions on I believe you'll also find some information regarding As for your limits, some protocols themselves impose a maximum size on the display (at least for serialPOS - see the CD5220 and AEDEX protocols), and other attributes. If you really want to go full ham you can do some research regarding that. Good luck! |
|
Piankero writes:
Do you have a list of these [obsolet] drivers?
Not really. My policy is to just disable drivers if they are broken in
some way and wait until somebody complains.
The important drivers are "curses" (default, useful for debugging) and
"hd44780" (the original lcdproc hardware and a defacto standard used
by many venders even today). These to are maintained in some kind of
group effort. Every other driver should have some kind of maintainer,
who looks after them and at least answers questions if directly contacted
by e-mail. If no maintainer is reachable anymore (or not listed at all)
it is reasonable to treat the driver as obsolete.
Can you provide an example of such a (not well documented) synonym usage?
I'm not sure, but your questions regarding hd44780 connectiontype or model
might fall in this category.
> There are multiple configuration items, each with the key "key" and a
> string value of the format ("%d,%s", linux_key_code, lcdproc_key_name).
Ok, so the value is clear that it always has this format. But the key is still unclear to me.
Yes, because it is too simple to see the pattern. The key is always "key".
It can occour multiple times.
In elektra an array looks like this: `somekey/#0, somekey/#1, somekey/#2` etc.
How does it look in your specific case. Can you give me an example?
See the default configuration file. There is an example, which is commented
out.
I removed some questions now. Can you look over it for a second time and see if there is anything
which maybe affects you(r code)?
I wrote the linux_input driver, that we are already discussing.
|
Thank you, I updated the specification and also added more text to Custom characters. I also updated some more specifications since I read through some of your doc files. The rest looks like to be pestering people. Can someone answer the last 2 questions from the "Server" part? I think that this is also something on which everybody works. Thanks for all of your help! |
|
Alright, I have some further questions and will now address it to the corresponding people: @marschap In glcdlib you provided a setting @marschap In hd44780 you made the setting "Speed" for the bitrate of the serial port. Is there any allowed range or enumerations like in all other settings for speed? @marschap In IOWorrior: Whats the default for ExtendedMode? @marschap In IOWorrior: Is there any validation possible for SerialNumber? Eg. 8 hex/decimal numbers @marschap In lirc: what does @marschap In xosd: Is there any range limit in Offset? @marschap In xosd: Is there any validation possible for Font or any random string allowed? @marschap In Server at Hello&Goodbye: Is there any validation possible, eg. a maximum number of allowed characters? @haraldg I cannot find the user @mmdolze for glcd/rawserial questions. Are these drivers still active? @haraldg In Server/FrameInterval: Are there any allowed enumerations or is just a positive number is required? Thanks for your help. |
|
On 01.09.2018 18:19, Piankero wrote:
I cannot find the user @mmdolze for glcd/rawserial
questions. Are these drivers still active?
Yeah, you won't have much luck with these. At least rawserial was a
relatively recent addition, so I guess somebody is using it.
Anyway I propose to not worry to much about it.
In Server/FrameInterval: Are there any allowed enumerations
or is just a positive number is required?
Yes, a positive number.
In practice very small or very large numbers probably don't make
sense, but there is no clear line, so just stick with a positive
number.
|
|
@marschap lcdproc client: OnTime/OffTime ... Can you provide any description for these values? Any validation possible? @marschap lcdproc client: Reportlevel: What is the range? 1-5? @marschap lcdproc client: Delay: Just a positive number required? 0 inlcuded? No Description is given. @marschap lcdproc client: ShowInvisible: No description is given @haraldg is marschap still active since he does not answer any of my questions? |
|
On 04.09.2018 11:29, Piankero wrote:
@marschap lcdproc client: Reportlevel: What is the range? 1-5?
The same as with the server. 0-5 IIRC, but I always get confused
because it is different from the official syslog priorities. (I
think offet by 2, but syslog documentation doesn't actually
specify numeric values, so this is more an implementation detail.)
@haraldg is marschap still active since he does not answer any of my
questions?
He contributed a patch some time ago, so I guess he is still somewhat
interested in lcdproc. But he used to be maintainer many years ago
(before we used git), so much of the code attributed to him, might
not be his and he might not now much about it.
Maybe also he isn't following the github discussions closely. We are
all volunteers.
|
|
Hi,
I am that guy :) Until now I was focused on implementing a version of lcdproc using the old low-level Elektra-API, which doesn't use a specification and will only be used for comparisons in my thesis. I will provide some code as soon as possible. For now I too have some questions:
Edit
|
|
kodebach writes:
Until now I was focused on implementing a version of lcdproc using the old low-level Elektra-API, which doesn't use a specification and will only be used for comparisons in my thesis.
Out of curiosity: What is the objective of that comparison?
Also I have a question of my own to both of you regarding the specification
thing: How is the validation going to work? Currently we have a bit of
validation coded in C directly into the drivers. Can we get rid of this
with the specification? Is the elektra validation only performed when
changing the configuration via elektra means or whenever loading the
configuration?
Background: lcdproc is used on embedded devices too where code size (and
memory foot print) are an issue. In a nice world we would have elektra
validation on updating the configuration and optional elektra validation
at runtime for non-embedded systems and be able to strip in code
validation completely.
(This is not a requirement from my side, just wondering where this is
going.)
I will provide some code as soon as possible.
Thanks. I think it would be very useful to first just work on a small
part of lcdproc and we discuss that, before putting effort into
adapting much of the code base. Also this way maybe some people are
encouraged to update their own drivers themselves...
Also we should probably agree on some order in which the various parts
of the code are worked on - to give people something to play with and
test early and in case you have to abandon your work, the most important
bits would have been done already.
1) Should the new version of lcdproc be able to use the existing configuration format, or would it be enough, if there was some kind of automated script for updating to the new libelektra format?
What do you mean with configuration format? ini vs. something or the key
hierarchy (and format of values)?
About the first: AFAIK nobody is especially fond of ini and people probably
wouldn't mind switching to something else. For lot's of people (me included)
it is very important that the configuration files are human readable /
human editable - so elektras old native format is not an option. As I
understand it, with elektra the choice of configuration format is up to the
user anyway? My suggestion is to just use elektras ini as default and if
there are any incompatibilities with our legacy ini parser, then let's
discuss them. Bug compatibility is not a goal I have for lcdproc ATM and
I break compatiblity in subtle ways often enough myself ...
About the second: I think new drivers will probably use the new hierarchical
features of elektra in their configuration - we should allow and encourage
that. If there is a good reason to switch to a more complex key hierarchy
for existing drivers, I'm open for that as well.
2) Is there some way of testing all the drivers, except for manually getting the hardware and checking for problems?
No. Some drivers probably wouldn't even initialize without the real hardware
present. Many probably can be operated with some kind of dummy device
(think of /dev/null).
We will encourage all people using lcdproc to test the new code before
the PR is merged (likely making an alpha release from a side branch)
and then give the release candidate lot's of time for testing too.
Anybody still ending up with a broken driver will be kindly invited to
join development of lcdproc ... ;)
3) Are the following configuration values the only ones that can occur repeatedly, i.e. are used like arrays?
I don't think anybody can answer this question reliably. Your check might
miss some cases where statements are split over several lines, but otherwise
looks good.
|
|
IIRC some of the serial drivers that just blit data to the display can be tested if you hook them up to a pty slave and record the output for validation. It's even better if you build a simulator, but I think that's really a hassle. |
That depends on the validation. Many things like range checks, regex checking strings, etc. can easily be checked by elektra. More advanced things like checking whether the .so file for a driver can be found or not will still have to be manually checked in code.
I did some tests, and as far as I can tell at least some checks (e.g. from the
What kind of embedded devices? The lcdproc documentation (which seems kind of outdated but still) mentions that an x86 system with a POSIX compliant OS is required. AFAIK elektra this should suffice for elektra to run. Maybe @markus2330 can also expand on this.
Yes, I agree. @markus2330 would this be possible somehow?
I will probably start with the the
I was mostly asking whether the new version should be able to directly work with an existing
I will try some more advanced checks, but thanks. The reason for my asking is that it we need to write the specification differently, if a key can occur repeatedly (it represents an array). |
Using the low-level API should be much easier to do but does not add type safety. The objective is to have some idea how expensive (in work hours) the different ways of elektrifying applications are.
Yes, the goal is to get rid of all the C code doing checks or transformations.
Within the low-level API validation only takes place when changing the configuration. This is safe as long Elektra is not bypassed (and config files do not vanish). The higher-level API then also add run-time and type-safety guarantees (not yet fully implemented). The goal is that the getter functions always return valid configuration (by using in-memory specs and exit() on memerrors).
The validation happens within plugins. So when someone does not install the plugins and we ignore some warnings, everything should be fine. We cannot provide any run-time guarantees then, tough. Maybe @kodebach can compare the code sizes so that we know if this reduction is noticeable at all? @haraldg Can you maybe give us some "acceptable" sizes for validation code? In the best case LCDproc without config code + Elektra + essential plugins is about the same size as LCDproc now. But that remains to be seen. @Piankero is thinking about having default value calculations in Python. They certainly would need to be optional. @haraldg any thoughts about which interpreter (if any at all) would be acceptable to calculate default values? Should we open a new issue for this? |
|
kodebach writes:
More advanced things like checking whether the .so file for a driver can be found or not will still have to be manually checked in code.
Of course. I was only speaking about checking static things (ie independent
from the actual system).
> lcdproc is used on embedded devices too where code size (and memory foot print) are an issue.
What kind of embedded devices? The lcdproc documentation (which seems kind of outdated but still) mentions that an x86 system with a POSIX compliant OS is required. AFAIK elektra this should suffice for elektra to run. Maybe @markus2330 can also expand on this.
Well x86 is not true. People use it successfully on mips and arm at least.
Yes, running elektra is not a problem. But if we already have elektra, it
would be nice to get rid of our in code validation to save some space.
Of course, saving some bytes is not the only reason this is desireable: Also
having to keep two validation steps in sync is not nice for all the
obvious reasons.
> Also we should probably agree on some order in which the various parts of the code are worked on
I will probably start with the the `lcdvc` and `lcdproc` clients. After that we will see...
lcdvc is pretty obscure. Not sure how many people use that. -> low priority
Also I'd suggest to not delay working on LCDd too much, to give people
something to test early. Also integration into our build system should
come early, to get as much testing coverage of that as possible. (I fear
nobody still active completely understands our build system.)
Maybe something like:
1) lcdproc client
2) autoconf integration
3) LCDd + curses driver
4) ...
would work for you?
I will try some more advanced checks, but thanks. The reason for my asking is that it we need to write the specification differently, if a key can occur repeatedly (it represents an array).
Well, I guess all these cases will surface during porting the code to
elektra anyway - the specification won't help much, when code itself
doesn't run with elektra. You can always fix the specification then, so
I wouldn't worry about it at this stage.
|
|
markus2330 writes:
The validation happens within plugins. So when someone does not install the plugins and we ignore some warnings, everything should be fine. We cannot provide any run-time guarantees then, tough.
Of course. The idea is, that on embedded systems the configuration couldn't
be changed (without going through some elektra tool, ie a webinterface) - so
the stored configuration should be guaranteed to be valid.
Maybe @kodebach can compare the code sizes so that we know if this reduction is noticeable at all?
Not a priority for me, but would be interesting.
@haraldg Can you maybe give us some "acceptable" sizes for validation code?
There is no hard limit. But saving 1kB of RAM (and again of flash) on
one million systems is still 1GB of RAM.
Now I don't think lcdproc has a million users, but elektra probably already
does in commercial applications and might have a lot more if it takes
off in FLOSS. Enough that even saving 1kB might add up to significant
costs. So ultimately this is more important for you then for me. (Remember
the case of Linksys switching it's firmwares from Linux to vxWorks because
of lower memory footprint.)
Also think about this from a green programming PoV.
In the best case LCDproc without config code + Elektra + essential plugins is about the same size as LCDproc now. But that remains to be seen.
I doubt it. I don't have the numbers ready ATM, but probably Elektra +
essential plugins is already larger then LCDproc now.
Anyway, there is no hard limit because this isn't fighting to fit on
some obscure device. It's just about getting rid of unnecessary code
if it is easily possible.
@Piankero is thinking about having default value calculations in Python. They certainly would need to be optional. @haraldg any thoughts about which interpreter (if any at all) would be acceptable to calculate default values? Should we open a new issue for this?
Build time, install time or run time?
run time: None. (Well, maybe /bin/sh would be acceptable, but I think
everybody agrees, this is not an option.)
install time: If it has to run on the target, pretty much same as above.
OpenWRT now use Lua, but there are already enough portability issues around
that, that I wouldn't suggest it. If it only has to run on the (build) host,
(ie when copying files into the binary package), then I suppose python or
any other common language is ok.
build time: As long as it doesn't make our build system more horrible
than it is already, I won't complain. (But I still compile lcdproc locally
on a system with only 256MB RAM. Python is not a problem there, but maybe
java might be?)
|
Works for me. I only mentioned
Couldn't we use the |
Well, there are so many use cases which a simple math + if plugin are not sufficient enough. After seeing through some configurations it has to be a scripting language. Of course if the calculation is too complicated it should get into the code itself and maybe not the specification but this is up to the spec + code maintainer. In this case its defenitely worth to be in the specification. @haraldg it is neither build/run/install time. In my proposed idea the python code just gets executed, once a user changes a setting. Take the Size setting as an example in your configuration. The python code would be executed only if someone would change the Model setting. It does not affect the run time at all. |
As far as i can see the Size setting is just: [/CwLnx/Size]
default = (../Model == 12232) ? 20x4 : ( (../Model == 12832) ? 21x4 : 16x2)So possible a kind of |
Your above example with the For LCDproc it is not needed to have a full scripting language because default value calculation is just a simple switch expression, thats right. But as far as it goes for other configurations it will not suffice very quickly. Imagine cases such as:
So before making small plugins which cover some specific needs for certain configurations I would suggest that we make a much more powerful plugin. Of course every user should keep code there as simple as possible or if possible do not use it at all but it would greatly relieve from the burden to make all these checks and calculations in the respective application in which they honestly should not belong to. |
|
Nonetheless the default value calculation plugin idea should be discussed in our repo :) Here we are interested if this approach when trying to set a value could trigger the change of another value (if not already present) via python is acceptable for the LCDproc community. This code would just execute once a user changes via |
I agree, but like you said yourself for LCDproc (and probably most other cases) no scripting language is needed. I propose we create a plugin that uses simple expressions and ifs that will be used for LCDproc. (the additional effort would be minimal because code already exists in Whether or not a plugin using python would be useful can then be discussed in the libelektra repo. |
|
I think I now understand what problem you are trying to solve with default value calculation. I haven't made up my mind yet, if I think it is worth solving on the elektra side. If the script only runs at configuration updates, then which data is changed and how is it made persistent? I think I don't understand what solution you are proposing yet. Anyway, you can't expect any scripting language other then /bin/sh to be available on the target where lcdproc is running. |
|
haraldg wrote:
If there is any way to change configuration via any Elektra tool, the validation plugins need to be present on the device to provide safety. (Btw. for me this is run-time, even if it happens in another process then LCDproc.)
Yes, easily possible. I thought there are more drivers or they are more complicated. We won't have much success with trying to reduce the size of a 138k binary. (Elektra's core alone is 64k. Both sizes from Debian Stretch binaries on amd64.)
We had questions for both build time and run time. The build-time dep can easily avoided (rewriting a small python script in C) but avoiding the run-time dep requires us to add more default value calculation plugins. This is a major thing but might be needed anyway. We need to discuss this internally first (ElektraInitiative/libelektra#2231) to not bother you too much. |
|
Duh, that's a point I missed in LCDd model for Config::Model. I'll add that soon... |
Dear LCDcommunity,
in the course of my Master Thesis I will write a specification for your configuration with the help of the tool libelektra. I already implemented most of LCDd but still have alot of questions concerning some configuration settings.
Specifications for your 3 clients will follow in the next week(s).
Please take a look at the added
Questions.md. Once everything is clear I will move out the specification from the created folder and delete the Questions.md file.