Caching of README files is potentially not working

[daveverwer]

Needs investigation, but something isn't right.

We have code that appears to check the ETAG before fetching/storing it a README file to our cache, but at least locally on my machine, it’s not working:

[ INFO ] Ingesting (limit: 10) ... [component: ingest]
[ INFO ] Candidate count: 10 [component: ingest]
[ INFO ] Copying readme to s3://spi-dev-readmes/sdggiesbrecht/swift-driver/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/shareup/json-apple/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/ctreffs/swiftsimctl/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/swhitty/swiftdraw/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/serverdriven/screendataui-ios/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/adamayoung/police-data-kit/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/mcrich23/mcrich23-toolkit/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/ptsochantaris/can-proceed/readme.html ... [component: server]
[ INFO ] Copying readme to s3://spi-dev-readmes/purelayout/purelayout/readme.html ... [component: server]

Both with --limit and --id, it fetches and caches all README files, every time.

I deployed a quick bit of logging to staging to see if it was happening there, and it's much less clear that there's a bug:

2024-03-12 11:13:40 | [ INFO ] Copying readme to s3://spi-dev-readmes/william-weng/wwtoast/readme.html ... [component: server]
2024-03-12 11:13:41 | [ INFO ] Copying readme to s3://spi-dev-readmes/goldenplan/tvoskeyboard/readme.html ... [component: server]
2024-03-12 11:13:41 | [ INFO ] Copying readme to s3://spi-dev-readmes/naftaly/threadcrumb/readme.html ... [component: server]
2024-03-12 11:13:41 | [ INFO ] Copying readme to s3://spi-dev-readmes/mtj0928/slidekit/readme.html ... [component: server]
2024-03-12 11:18:43 | [ INFO ] Copying readme to s3://spi-dev-readmes/william-weng/wwsigninwith3rd_line/readme.html ... [component: server]
2024-03-12 11:18:43 | [ INFO ] Copying readme to s3://spi-dev-readmes/gohanlon/swift-memberwise-init-macro/readme.html ... [component: server]
2024-03-12 11:23:45 | [ INFO ] Copying readme to s3://spi-dev-readmes/cheekyghost-labs/oslogclient/readme.html ... [component: server]
2024-03-12 11:23:45 | [ INFO ] Copying readme to s3://spi-dev-readmes/polpielladev/chatty-cli/readme.html ... [component: server]
2024-03-12 11:23:45 | [ INFO ] Copying readme to s3://spi-dev-readmes/swift-server/swift-service-lifecycle/readme.html ... [component: server]
2024-03-12 11:28:47 | [ INFO ] Copying readme to s3://spi-dev-readmes/kishikawakatsumi/swift-power-assert/readme.html ... [component: server]
2024-03-12 11:28:47 | [ INFO ] Copying readme to s3://spi-dev-readmes/insub4067/swiftuinavigator/readme.html ... [component: server]
2024-03-12 11:28:47 | [ INFO ] Copying readme to s3://spi-dev-readmes/jrsaruo/mediaviewer/readme.html ... [component: server]
2024-03-12 11:28:47 | [ INFO ] Copying readme to s3://spi-dev-readmes/krzysztofzablocki/sourcery/readme.html ... [component: server]
2024-03-12 11:33:49 | [ INFO ] Copying readme to s3://spi-dev-readmes/william-weng/wwexpandablecell/readme.html ... [component: server]
2024-03-12 11:33:49 | [ INFO ] Copying readme to s3://spi-dev-readmes/artsabintsev/freedom/readme.html ... [component: server]
2024-03-12 11:38:51 | [ INFO ] Copying readme to s3://spi-dev-readmes/pusher/pusher-websocket-swift/readme.html ... [component: server]
2024-03-12 11:38:51 | [ INFO ] Copying readme to s3://spi-dev-readmes/hackiftekhar/iqkeyboardmanager/readme.html ... [component: server]
2024-03-12 11:43:53 | [ INFO ] Copying readme to s3://spi-dev-readmes/andreamazz/bubbletransition/readme.html ... [component: server]
2024-03-12 11:43:53 | [ INFO ] Copying readme to s3://spi-dev-readmes/simonweniger/superagent-swift/readme.html ... [component: server]
2024-03-12 11:43:53 | [ INFO ] Copying readme to s3://spi-dev-readmes/ladislas/swifteventcenter/readme.html ... [component: server]
2024-03-12 11:48:54 | [ INFO ] Copying readme to s3://spi-dev-readmes/quickbirdeng/datakit/readme.html ... [component: server]
2024-03-12 11:48:54 | [ INFO ] Copying readme to s3://spi-dev-readmes/mailslurp/mailslurp-client-swift/readme.html ... [component: server]

In 30 minutes, it uploaded 22 README files to S3. That's not the total number of packages that were ingested during that time, but I'm confident it's more README files than were edited.

Needs investigation