Migrate from doing OTA with esp_https_ota in ESP-IDF v4.3

[monkeytronics]

Describe the issue
In my old code, ( ESP-IDF v4.3 ), OTA is done by subscribing to jobs etc and when a new ota_url arrives, executing this function:

/*
 * Https request & Flash Erase & Write code only
 */
esp_err_t do_firmware_upgrade(const char *url)
{
    if (!url) {
        return ESP_FAIL;
    }

    esp_http_client_config_t config = {
        .url = url,
        .cert_pem = (char *)root_ca,
        .timeout_ms = 60000,
        // .buffer_size = 1024 // default is 279 bytes! Default is best... 
    };


/* Update MODEM power saver mode to NONE - Full Power during OTA. */
    wifi_ps_type_t wifi_ps;
    esp_err_t ret = esp_wifi_get_ps(&wifi_ps);
    if (ESP_OK == ret) {
        if (wifi_ps != WIFI_PS_NONE) {
            ESP_ERROR_CHECK( esp_wifi_set_ps(WIFI_PS_NONE) );
            ESP_LOGW(TAG, "Updating WiFi PS from %s to %s.", wifiPsModem[wifi_ps], wifiPsModem[WIFI_PS_NONE]);
        } else {
            ESP_LOGW(TAG, "Leaving WiFi PS as is: %s.", wifiPsModem[wifi_ps]);
        }
    }


/* Console Debug Stuff */
    ESP_LOGW(TAG, "************************************");
    ESP_LOGW(TAG, "        Starting OTA Update         ");
    ESP_LOGW(TAG, "************************************\n\n");

    #ifdef TEST_HIGH_WATERMARK_OTA
        UBaseType_t uxHighWaterMark;
        uxHighWaterMark = uxTaskGetStackHighWaterMark( NULL );
        ESP_LOGE(TAG, "OTA Water Mrk Before = %d\n", uxHighWaterMark);
    #endif  


/* Run Actual OTA task */
    esp_err_t rc = esp_https_ota(&config);
    ESP_LOGW("********", "esp_https_ota Result = %d", rc);


/* Console Debug Stuff */
    ESP_LOGW(TAG, "Free Heap After OTA: %d", xPortGetFreeHeapSize());
    #ifdef TEST_HIGH_WATERMARK_OTA
        // UBaseType_t uxHighWaterMark;
        uxHighWaterMark = uxTaskGetStackHighWaterMark( NULL );
        ESP_LOGE(TAG, "OTA Water Mrk After = %d", uxHighWaterMark);
    #endif  

    return rc;
}

On the AWS side, I create a custom job and drop the firmware binary and job file at suitable locations. The above code works fine with both signed and unsigned binaries. The unsigned being an important part of the test strategy before rolling it out to production devices which have encrypted firmware.

Is there any documentation that would describe how to migrate from this method of OTA to use the code in this library for MQTT and HTTP OTA? Some issues that spring to mind:

notes
I am using an ESP32, not the C3 device. But I have made the necessary changes so that these demos work fine.

Specific Road Blocks

1. The supplied code looks to have code signing hard baked in with AWS code signing certificate. Can this code be used without code signing enabled? Sorry it it seems obvious, but it's a big library and I don't know where to look.
2. We do not use AWS code signing or AWS OTA jobs directly in the console. We create a custom job and skip out code signing, since it is done during the build step. The location of the code signing certificate is provided from menu config. And appears to be seamlessly picked up when we use esp_https_ota( ). It neatly handles the case when the certificate is not provided, and knows to skip over that step (hence handling signed / unsigned with no effort on our part). Could you provide some guidance as to how this can be achieved with your library?
3. Advice on RSA versus ECDSA? In our previous secure boot, I believe the format was RSA. It appears ECDSA is now mandatory for secure boot??? Does that mean that there is a compatibility boundary here that actually prevents this upgrade path?
4. One more stupid question while on the subject - how do the above questions impact upon the decision between OTA over MQTT versus OTA over HTTPS? I haven't found anything that really explains how the OTA over MQTT is driven.

[paulbartell]

The supplied code looks to have code signing hard baked in with AWS code signing certificate. Can this code be used without code signing enabled? Sorry it it seems obvious, but it's a big library and I don't know where to look.

Yes. You will need to modify the ota_pal.c file to skip code signing verification. Simply modify otaPal_CheckFileSignature to return OTA_PAL_COMBINE_ERR( OtaPalSuccess, 0 ) ).

We do not use AWS code signing or AWS OTA jobs directly in the console. We create a custom job and skip out code signing, since it is done during the build step. The location of the code signing certificate is provided from menu config. And appears to be seamlessly picked up when we use esp_https_ota( ). It neatly handles the case when the certificate is not provided, and knows to skip over that step (hence handling signed / unsigned with no effort on our part). Could you provide some guidance as to how this can be achieved with your library?

The certificate is provided to ota_pal.c via the otaPal_SetCodeSigningCertificate function called in ota_demo_core_mqtt.c. Since you're not using code signing, you can ignore this part and remove the certificate portions which are not necessary for your use case.

Advice on RSA versus ECDSA? In our previous secure boot, I believe the format was RSA. It appears ECDSA is now mandatory for secure boot??? Does that mean that there is a compatibility boundary here that actually prevents this upgrade path?

Generally, ECDSA is preferred because it is faster to compute and more secure for a given key size. That said, RSA is widely regarded as acceptable. In the OTA example in this repository, code is signed twice:

• Once with an ECDSA key by the AWS signing service. This signature is verified prior to attempting to boot the new image.
• Again with an RSA key for ESP Secure Boot V2 where the image is verified on each boot.

One more stupid question while on the subject - how do the above questions impact upon the decision between OTA over MQTT versus OTA over HTTPS? I haven't found anything that really explains how the OTA over MQTT is driven.

OTA over MQTT is driven by the MQTT File Delivery Service (aka mqtt streams). The best explanation available is probably in the MQTT File Streams Library which is part of our newer approach to OTA and partially replaces our existing OTA library. We plan to update this reference integration with our new set of OTA libraries in the coming months.

I would suggest using the http method because it avoids unnecessary costs due to a large number of OTA messages.

[monkeytronics]

Thanks @paulbartell. Mind if I double check.

1. I'd remove the call to otaPal_SetCodeSigningCertificate.
2. Inside ota_pal.c, looks like otaPal_ReadAndAssumeCertificate is never called, so can ignore it. Seems odd.
3. otaPal_CheckFileSignature is called from otaPal_CloseFile, which is one of the handlers attached. Do you reckon commenting out the line where it is called will do the job?
   mainErr = OTA_PAL_MAIN_ERR( otaPal_CheckFileSignature( pFileContext ) );

I'll obviously go and test it now too, but I wanted to check that my understanding was ok?

[paulbartell]

@monkeytronics Yes. That sounds correct.

[monkeytronics]

Yip, that seems to work. Had to comment out much of otaPal_CloseFile in the end. But works fine.

[monkeytronics]

Appreciate that this is not encouraged, so no need to do anything...