GWC makes millions of requests per minute on Azure Blob Api when stuck in bulk deletion

[danduk82]

We found the hard way that GWC was stuck in a corrupted status for bulk deletion and it was generating MILLIONS OF REQUESTS PER MINUTE on Azure blob api, resulting in several thousands of dollars per day of useless costs. The pending_deletes.properties file is present in the blob storage, but the layers are already deleted (not clear how this can be reproduced for the moment).

Early analysis seems to show that this is already present in the default GWC implementation, so fixes might need to go in the upstream geoserver project and gs-cloud should possibly just update the dependency.

Storage account log sample:

2.0;2023-05-16T13:54:20.2764292Z;ListBlobs;Success;200;5;4;authenticated;redacted;redacted;blob;https://redacted.blob.core.windows.net/geoserver-blobs-cache?prefix=LayerGroupInfoImpl--23fa92c9%3a16d2b008a54%3a-765e/EPSG%3a900913/&marker=&maxresults=1000&include=&restype=container&comp=list;"/redacted/geoserver-blobs-cache";662a6913-d01e-0066-41fd-877f50000000;0;10.224.0.9:34598;2018-11-09;653;0;152;12864;0;;;;;;"Azure-Storage/11.0.0 (JavaJRE 17.0.7; Linux 5.15.0-1035-azure)";;"419a0a32-c73e-4d89-9936-e3aa2040bcf1";;;;;;;;

Pod logs:

2023-05-16 13:54:12.607  INFO 1 --- [           main] c.m.azure.storage.blob.LoggingFactory    : 'https://redacted.blob.core.windows.net/geoserver-blobs-cache/%2F_pending_deletes.properties'==> OUTGOING REQUEST (Try number='1')
2023-05-16 13:54:12.628  INFO 1 --- [ntLoopGroup-1-1] c.m.azure.storage.blob.LoggingFactory    : Successfully Received Response
Request try:'1', request duration:'21' ms, operation duration:'21' ms
GET: https://redacted.blob.core.windows.net/geoserver-blobs-cache/%2F_pending_deletes.properties
2023-05-16 13:54:12.642  INFO 1 --- [           main] org.geowebcache.azure.AzureBlobStore     : Restarting pending bulk delete on 'sr-geoserver-trn-blobs-cache/LayerGroupInfoImpl-408b678:1749c11e33e:-7f67/EPSG:900913/':1680611350000

Contents of pending_deletes.properties:

#
#Thu Apr 20 05:17:56 UTC 2023
LayerGroupInfoImpl-408b678\:1749c11e33e\:-7f67/EPSG\:900913/=1680611350000
LayerGroupInfoImpl--23fa92c9\:16d2b008a54\:-765e/EPSG\:900913/=1681967876000