You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- [x] bug report -> please search issues before submitting
- [ ] feature request
- [X] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
Minimal steps to reproduce
Run the following command
docker run -it -e "IotHubConnectionString=[redacted]" -e DeviceCount=1000 mcr.microsoft.com/oss/azure-samples/azureiot-simulatordeviceprovisioning
Provisioning crashes with a ThrottlingBacklogTimeout that causes the script to stop running.
Any log messages given by the failure
PS C:\Users\kyichii>
Starting device provisioning
2021-10-12T07:17:21.6511837Z: 100 devices have been created @ 63.20/sec
2021-10-12T07:17:21.8319529Z: 200 devices have been created @ 553.18/sec
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:55f0784980964ecf83ec9bf4cce2bc7c-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:3ea3022e253e4d7b95ae629f77c7c403-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:474cf77e1c214b409f7429e4ed439825-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:a6b0b0fd21ab40c0a341fbe65a366d54-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func2 isMappedToException, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken)
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func3 modifyRequestMessageAsync, Func3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163
Expected/desired behavior
This should not crash when throttling. Ideally it would allow us to set the SKU rate for the IoT Hub to avoid the throttling all together and test provisioning at the maximum limits of a specific iot hub
OS and Version?
Windows 10
Versions
Latest
Mention any other details that might be useful
This is behavior that inconsistently crashes but consistently throttles on lower end IoT Hub due to the registry operations. I am also attempting to get some additional logs from customers that are encountering the same issuje.
The text was updated successfully, but these errors were encountered:
This issue is for a: (mark with an
x
)Minimal steps to reproduce
Provisioning crashes with a ThrottlingBacklogTimeout that causes the script to stop running.
Any log messages given by the failure
PS C:\Users\kyichii>
Starting device provisioning
2021-10-12T07:17:21.6511837Z: 100 devices have been created @ 63.20/sec
2021-10-12T07:17:21.8319529Z: 200 devices have been created @ 553.18/sec
Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:55f0784980964ecf83ec9bf4cce2bc7c-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
2 isMappedToException, Func3 processResponseMessageAsync, IDictionary
2 errorMappingOverrides, CancellationToken cancellationToken)at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary
2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List
1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:3ea3022e253e4d7b95ae629f77c7c403-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
2 isMappedToException, Func3 processResponseMessageAsync, IDictionary
2 errorMappingOverrides, CancellationToken cancellationToken)at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary
2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List
1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:474cf77e1c214b409f7429e4ed439825-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
2 isMappedToException, Func3 processResponseMessageAsync, IDictionary
2 errorMappingOverrides, CancellationToken cancellationToken)at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary
2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List
1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163Microsoft.Azure.Devices.Common.Exceptions.ThrottlingException: {"Message":"ErrorCode:ThrottlingBacklogTimeout;The request has been throttled. Wait 10 seconds and try again. Operation type: CRUD","ExceptionMessage":"Tracking ID:a6b0b0fd21ab40c0a341fbe65a366d54-G:0-TimeStamp:10/12/2021 07:18:02"}
at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpClient httpClient, HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
2 isMappedToException, Func3 processResponseMessageAsync, IDictionary
2 errorMappingOverrides, CancellationToken cancellationToken)at Microsoft.Azure.Devices.HttpClientHelper.ExecuteAsync(HttpMethod httpMethod, Uri requestUri, Func
3 modifyRequestMessageAsync, Func
3 processResponseMessageAsync, IDictionary2 errorMappingOverrides, CancellationToken cancellationToken) at Microsoft.Azure.Devices.HttpClientHelper.PostAsync[T1,T2](Uri requestUri, T1 entity, IDictionary
2 errorMappingOverrides, IDictionary2 customHeaders, CancellationToken cancellationToken) at IotSimulatorDeviceProvisioning.Program.BulkCreateDevicesAsync(List
1 devices, RegistryManager registryManager, DeviceProvisionStats stats) in /src/src/IotSimulatorDeviceProvisioning/Program.cs:line 163Expected/desired behavior
OS and Version?
Versions
Mention any other details that might be useful
This is behavior that inconsistently crashes but consistently throttles on lower end IoT Hub due to the registry operations. I am also attempting to get some additional logs from customers that are encountering the same issuje.
The text was updated successfully, but these errors were encountered: