Replies: 8 comments 13 replies
-
|
I think a Python binding for libavrdude is good to have. My question is why using SWIG? The more popular ways now seem to be using ctypes or cython. Example from the projects I am involved in. From my testing, libftdi's built-in SWIG based binding is the most problematic. pyusb -- ctypes |
Beta Was this translation helpful? Give feedback.
-
|
Well, frankly, because I don't have even less of an idea about cython or ctypes. ;-) |
Beta Was this translation helpful? Give feedback.
-
|
I actually do not know much about ctypes and even python, but here is a simple introduction. I think the following shows how simple it can be. Cython is probably a bit more complicated. |
Beta Was this translation helpful? Give feedback.
-
|
The following is based on my own experiences building various Python bindings under different OS and toolchain. |
Beta Was this translation helpful? Give feedback.
-
|
Not so sure if you have some inputs here as well. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
I do have some ideas, but TBH, the time I can spend on avrdude is less than all the things I have ideas for would take to discuss and develop right now. I had therefore hoped to kick the API can down the road for a bit longer until I have concluded most of the build stuff, but I fear I have to take a look into API design next after all and reschedule the build stuff for later. |
Beta Was this translation helpful? Give feedback.
-
|
For the build stuff, there are things related to libavrdude. Before the ABI/API discussions conclude, it is probably better that avrdude uses static link with libavrdude. So at least some of the build improvement activities related to libavrdude can be delayed. |
Beta Was this translation helpful? Give feedback.
-
The comments by @ndim here may be related. Maybe now it is time to clean up libavrdude, probably @stefanrueger, @ndim and you can help on this front.
|
Beta Was this translation helpful? Give feedback.
-
I still think it is OK to fix the build system first and look at the API/ABI later.
If that means encapsulating some of the properties of parts by the use of get_...() functions and hiding access to components this is doable. If the end-goal is to not expose the internal structure of PROGRAMMER, AVRPART or AVRMEM in the API, that's also doable. This kind of means replacing What has happened in the past is that, for example, a new part came out with an "unusual" property that the page size for erasing is 4 times bigger than the page size for writing. AVRDUDE needed to find a way to describe a part as having this property and extended the AVRPART structure by a component But, and this is the crucial bit here, this should by and large be independent of the build process. So, I'd argue to get this [the build system] right first. |
Beta Was this translation helpful? Give feedback.
-
|
A libavrdude library user should not need to know the size or the the memory layout of libavrdude internal structures like PROGRAMMER or AVRPART or AVRMEM. A libavrdude user never needs to copy or free such a structure, and accesses to struct members can be done by functions with a stable function signature. As I understand it, libavrdude reads in the So libavrdude can change the size and memory layout of its data structures as it likes without breaking API or ABI. And if a function needs to return a string from one of those data structures, it does not even need to involve A very basic example for a fictitious "libad" library and a utility using "libad", which I have written without taking a look at libavrdude internals at all: /* libad.h */
typedef struct ad_programmer ad_programmer;
typedef struct ad_part ad_part;
typedef struct ad_context ad_context;
ad_context *ad_context_new(void);
void ad_context_free(ad_context *ctx);
int ad_write(ad_context *ctx, ad_programmer *prog, ad_part *part, void *buf, size_t sz);/* libad.c */
#include <stddef.h>
#include <libad.h>
struct ad_part {
...
size_t n_page_erase;
...
};
struct ad_programmer {
...
};
int ad_write(ad_context *ctx, ad_programmer *prog, ad_part *part, void *buf, size_t sz)
{
ctx->errno = 0;
...
... deal with n_page_erase ...
...
}
struct ad_context {
ad_programmer *programmers;
ad_parts *parts;
};
ad_context *ad_context_new(void)
{
ad_context *ctx = calloc(1, sizeof(*ctx));
... parse avrdude.conf into ctx->programmers and ctx->parts and ...
return ctx;
}
void ad_context_free(ad_context *ctx)
{
... free members of ctx ...
free(ctx);
}/* main.c */
#include <libad.h>
int main(void)
{
ad_context *ctx = ad_context_new();
ad_programmer *prog = ad_...(ctx);
ad_part *part = ad_...(ctx);
uint8_t buf[256];
if (ad_write(ctx, prog, part, buf, sizeof(buf))) {
/* deal with ad_write errors */
fprintf(stderr, "ad error %d\n", ctx->errno);
}
ad_context_free(ctx);
return 0;
}The library user As It should be a not insignificant amount of not very complex code to adapt such data structures and the functions operating on them to a Python or C++ or ... API wrapper. |
Beta Was this translation helpful? Give feedback.
-
|
The disadvantage is that it requires drafting a completely new library – without anyone known to be willing to use it right now. libavrdude was a simple quick way to just segregate the existing pieces of code into a library backend, and a UI (CLI) frontend. Even that has proven to already cause quite a bit of extra work – still, with no known users (yet). Basically, what you are describing is a bit of that what the existing pieces of my SWIG wrapper are doing, just that you don't have a single function for each any any possible access to struct fields, but instead wrap up what I believed to be relevant fields into a Python dict. That does not require spending more work in to the backend library, only into the interface definition. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that would be a completely new library, at least the parts directed to the outside world via its API and ABI. The programmer protocol code etc. would not be completely new. Still a lot of work. The existing libavrdude API shows that it was not particularly designed ("simple quick way of segregating existing pieces of code" is exactly what the API looks like), and that is why I would say it is unsuitable for building at least shared libraries of. Static libraries might be helpful for some scenarios. If I needed to incorporate parts of the FYI, a few weeks ago in one of the issues I was active in, there was a library user... here: #1683 (comment) I am unfamiliar with this space, but there might be projects which have been spawning As to the functions... you would need iteration functions over programmers and parts, lookup functions for ports, programmers, and parts, and many more. I just wanted to demonstrate how incomplete types can be used in C to allow a library to internally change its data structures without necessarily breaking API or even ABI. |
Beta Was this translation helpful? Give feedback.
-
|
Interestingly Homebrew installs dynamic linking avrdude binary and static linked |
Beta Was this translation helpful? Give feedback.
-
|
Without the trick of using |
Beta Was this translation helpful? Give feedback.
-
Actually, the SWIG wrapper (as I have it now) isn't using the library itself, but it recompiles everything statically – in order to eventually supply a shared lib for Python. I think it might even be able to use the existing shared lib to build upon though, it's just that the CMake stuff was easier for me to do that way (I'm absolutely unfamilar with CMake).
That's what many others did in the past (and still do), but I think it's really a crock rather than a solution. You'd have to parse all the messages shown, and unless you implement a kind of pseudo TTY wrapper (as e.g. Arduino does), there's no chance for things like a progress indication that really integrates into that wrapper. All that once eventually got us to the idea of segregating frontend and backend in a clearer manner than it used to be before.
The SWIG wrapper just wraps the list access functions (which came from Brian Dean, and have always been there since the start of AVRDUDE), and just offers a few helper functions (basically typecast), to access things like programmer, parts, or memory regions. That is kind of a compromise between effort (for the wrapper) and not having to add anything to the backend at all. The completely new abstraction library is certainly a good idea, but I wouldn't want to do it without a potential "customer" since I assume it is really quite some effort (and then still requires [some] additional effort for non-C wrappers). |
Beta Was this translation helpful? Give feedback.
-
Interesting approach and I kind of understand why you use SWIG now. However, SWIG based Python bindings are more difficult to build cross-platform based on my own experiences. We can already see that there are problems with macOS (should be able to be sorted out as only one step is mssing) and Windows (more difficult to get right for both MSYS2 mingw build and MSVC build). |
Beta Was this translation helpful? Give feedback.
-
Another thing, there could be a licensing concerns as well. For those choosing to spawning the For example, AVRDUESS is using GPL. But technically different license can be used. As of now, libavrdude will be covered by GPL without linking exception. Technically speaking, LGPL will be a more suitable license if we want to allow more users to use it. A new library may be able to use LGPL (or GPL with linking exception). As of now, this is a moot point though as there are not well known projects using libarvdude. |
Beta Was this translation helpful? Give feedback.
-
|
Naturally, with my BSD roots, I (and also Brian Dean) would have favored a BSD license. It was Ted Roth by that time who convinced Brian to put the project under GPL. I think it can just remain there, including derivatives. So far, I don't know of anyone who'd have complained about that. |
Beta Was this translation helpful? Give feedback.
-
This commit is a typical example: Using the |
Beta Was this translation helpful? Give feedback.
-
|
Btw., just for a test, I turned the CMake library computation into a shared lib, and I could successfully use that shared lib in the SWIG wrapper from Python. I don't think this is really the way to go right now, but it works. If someone is interested: |
Beta Was this translation helpful? Give feedback.
-
I just opened a WIP (work in progress) pull request for a SWIG Python wrapper for libavrdude.
I've been toying with that idea all the way back when we started segregating libavrdude from the actual frontend code, years ago. However, I never found enough time to really materialize the idea in any way. Now that there are so many nice developers driving the progress of AVRDUDE (thanks!), I could lean back a bit, and get back to that idea.
I confess: I imagined it would be easier … there's a lot of legwork needed, to wrap everything that would be useful from the Python level. Fortunately, a lot of things are basically opaque stuff that is only required inside libavrdude itself, so it does not need to be wrapped.
Everything is now in a state were I would consider it to be "hello world": config files can be read, the list of AVR devices can be walked, and for each device, the possible memory regions can be examined.
I think SWIG might also benefit a lot from turning as many
#defines intoenumliterals as possible, as they are typed, and can therefor be wrapped automatically.Any suggestions or even code contributions (to my branch) are welcome!
Beta Was this translation helpful? Give feedback.
All reactions