HomeSpan 1.9.0 take more flash than 1.8.0 #785
Comments
|
For further check you can try ESP IDF tool: https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/performance/size.html I user idf.py size to check the binary size, but you can use more check. Arduino have the same feature when it finish build. You can check same sample project with Arduino IDE with sample 1.8 and same sample project with 1.9 version. The project I am using use Arduino as IDF component and build it using ESP IDF, not Arduino, you can check more about it with my PR here: #717, you can also try using my change to make it building with ESP IDF with 1.8 version and same sample project with 1.9 version. I know that you are not familiar with ESP IDF but it is the professional tool. You can still using all your beloved Arduino libs and feature in ESP IDF with more advance feature. Ping me if you want to setup the test project and got any problem. |
|
I am not fully read all the code but in version 1.9.0 you are Enable C++ exceptions, this alone add many code to the final binary. A code using exception: https://github.com/HomeSpan/HomeSpan/blob/42e174e3cdbf2c943de5185e8c11d1eea861497b/src/PSRAM.h#L48C65-L48C73 |
|
Thanks. I tried removing there noexcept from all lines in the code but compile size is the same under the Arduino IDE. I also removed throwing exceptions entirely but still same size. I'm wondering if it is the use of a custom allocator, or perhaps the inclusion of streams. I'll look into it, though please feel free to experiment as well and let me know if you identify a primary driver of the increase that can be eliminated. |
|
I bet it is c++ Streams. I noticed this when I was trying to buffer Serial to a different output, and also replacing Strings, or std::string to stringbuffers. |
|
I have confirmed that the size increase of the flash binary is completely unrelated to the use of the custom allocators (regardless of whether C++ exceptions are used). As @frankonski correctly surmised, the significant increase in size of the codebase we see in 1.9.0 is solely due to the use of the streams library. Simply adding In fact, if you create a basic ESP32 "Hello World" sketch (nothing to do with HomeSpan) and compile both with and without adding Since the inclusion of streams in 1.9.0 is a critical innovation that allows for much more scalable use of RAM, I think the increase in size is more than warranted. For base ESP32 chips that have 4MB of Flash, RAM has always been a more limiting factor than sketch size since there is plenty of Flash if you use 1.9MB OTA partitions (or eliminate OTA altogether). That said, if anyone knows a way of including whatever subset of iostreams is needed to implement the changes in 1.9.0, that could possibly help reduce the overall footprint a bit. |
|
The use of stream allows much better RAM usage, at the expense of more flash usage. |
|
Since I'm not using input streams, I'll see if including ostream instead of iostream shrinks the binary. |
|
|
|
I've reviewed the stream classes I used in HomeSpan and think they are already as small as possible. The key is NOT to include |
|
@HomeSpan Because when refactor codes you are not mention in every commit, it is very hard to know what cause the problem, the only way now is analyses the final binary. If I have spare time I will setup two test project with 1.8.0 and 1.9.0 and compare it using ESP IDF tool. But what I said before, in the ESP IDF by default they are not enable exception handle for++: https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32/api-guides/error-handling.html?#c-exceptions, ESP32 Arduino 2.x using ESP IDF 4.4. I do not believe add more code to handle exception will not take any size in the final binary. I must enable the CONFIG_COMPILER_CXX_EXCEPTIONS option to build 1.9.0. |
|
Note that all of the individual commits can be seen in the release-specific branches. Please see "release-1.9.0" for details of each commit. Under the Arduino-ESP32 library, there is no change in the size of the code when I added the new malloc code that handles exceptions. You can instead see a significant jump in code size in commit f3e9860 when I first started testing the use of streams. If I'm understanding you correctly, in order to compile 1.9.0 under the IDF you need to enable CXX_EXCEPTIONS. However, Espressif compiled the Arduino-ESP32 board manager with CXX_EXCEPTIONS already enabled, so there is no change to the code size under the Arduino-ESP32 library when adding exception handling to the code. One of the limitations with using the Arduino-ESP32 library is that you are stuck with the decisions that Espressif has made with regard to all the sdkconfig choices. The only way to change this is to recompile your own version of the Arduino-ESP32 library from the IDF itself (which you may be doing if you are using the IDF). One thing I noted on the link you indicated above is that Espressif says adding exceptions only increases the size of the file by a few kB, but we are seeing a jump of about 200 kB, which is consistent with adding streams. Once I have a chance to start working with the IDF directly I will be able to test more thoroughly, but right now all testing is done only with the pre-compiled Arduino-ESP32 library since nearly all HomeSpan users make use of that library from either within the Arduino IDE, Platform I/O, or Visual Studio. Note that even if I use the IDF directly, I will only be able to deploy code that everyone can generically use if I set the sdkconfig parameters to exactly match those that Espressif chooses in their version of the Arduino-ESP32 pre-compiled library. I can of course deploy a custom library but that needs to be target-specific, and I would need to create different versions for different chips and configurations. With regard to the addition of exceptions to the custom allocators, I pulled that code snippet from the web - it seems to be widely used, and it included exception handling. There is likely no problem if you remove the exception handling from the code if it solves the issue when compiling under the IDF. One other question --- are you pushing up against the 2MB OTA partition limit? If so, how? A typical HomeSpan sketch is only about 1.4MB, and that includes all the underlying WiFi libraries, Streams, etc. What else is in your program that takes up another 600MB? Long term my goal is to make HomeSpan independent of the Arduino-ESP32 library, but still fully compatible with it. In other words, I'd like to migrate all the functions to only those that are native to the IDF so HomeSpan can be used either with or without the Arduino-ESP32 library, though as noted above, I need to make sure I don't create any conflicts with the pre-compiled versions. This may be a very long-term project... |
|
In my project I use some lib like mqtt, async webserver, sensor driver, heatpump driver so it now 1.85MB. As I understand you want to make your lib chip independant? Mean it can run on Ti, Esp, X86, STM32 ... chip? But if we are on the ESP32 their Arduino lib is base on ESP IDF and custom Rtos. I know some lib is chip independant like bacnet-stack but the codebase very complex. About the ESP IDF I can use their https://github.com/espressif/esp-homekit-sdk with some wraper for Arduino, but it is not easy to use like your lib, but it is small size and very fast. |
|
@HomeSpan Today I build my project with some lib disable to test with HomeSpan 1.8.0 and 1.9.0 with command:
This is detail size for 1.9.0: 1.8.0 No exception: 1.8.0 with Exception enable: There are same problem with the libstdc++ here: https://esp32.com/viewtopic.php?t=3741 seem that PSRAM code will use this lib and the iostream too. |
|
Thanks - the link at the end regarding libstdc++ is quite informative. One thing to note is that the compile process under the Arduino IDE is not readily configurable, and Espressif invokes -fexceptions. That means that under the Arudino IDE there is no difference in the size of the binary between using exceptions in a sketch and not using exceptions. For example, the following code produces a binary of 261,169 bytes (under the Arduino IDE using Arduino-ESP32 2.0.14): #include <vector>
void setup() {
Serial.begin(115200);
delay(1000);
Serial.printf("\nReady\n\n");
std::vector<int> myvector;
for (int i=1; i<=5; i++)
myvector.push_back(i);
for (auto it = myvector.begin() ; it != myvector.end(); ++it)
Serial.printf("%d ",(*it));
Serial.printf("\n");
}
///////////////////
void loop(){
}Adding in the the custom allocator code with exceptions produces a binary size of 261,109 bytes (ironically, 60 bytes smaller): #include <vector>
template <class T>
struct Mallocator {
typedef T value_type;
Mallocator() = default;
template <class U> constexpr Mallocator(const Mallocator<U>&) noexcept {}
[[nodiscard]] T* allocate(std::size_t n) {
if(n > std::size_t(-1) / sizeof(T)) throw std::bad_alloc();
if(auto p = static_cast<T*>(malloc(n*sizeof(T)))) return p;
throw std::bad_alloc();
}
void deallocate(T* p, std::size_t) noexcept { std::free(p); }
};
template <class T, class U>
bool operator==(const Mallocator<T>&, const Mallocator<U>&) { return true; }
template <class T, class U>
bool operator!=(const Mallocator<T>&, const Mallocator<U>&) { return false; }
void setup() {
Serial.begin(115200);
delay(1000);
Serial.printf("\nReady\n\n");
// std::vector<int> myvector;
std::vector<int, Mallocator<int>> myvector;
for (int i=1; i<=5; i++)
myvector.push_back(i);
for (auto it = myvector.begin() ; it != myvector.end(); ++it)
Serial.printf("%d ",(*it));
Serial.printf("\n");
}
///////////////////
void loop(){
}This result is independent of adding and using With the full IDF you likely have more flexibility and can compile with -fno-exceptions, which should automatically convert the exceptions in code to if/else statements (https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_exceptions.html). |
Yes Arduino they enable by default but in version 1.8.0 you are not using Exception, so it it not require. |
I am testing 1.9.0 with one of my project before (20%) free, and with 1.9.0 it only free 6% (0x1e0000 ~2MB app partition).
The text was updated successfully, but these errors were encountered: