Seat\Eveapi\Jobs\Assets\Character\Names and ESI for invalid ids

[recursivetree]

• Problem: The Seat\Eveapi\Jobs\Assets\Character\Names job fails if a single item id is invalid. All other, possibly valid ids won't get processed if this happens. The code queries the id in batches of 1000. If one of these 1000 items doesn't exist, ESI fails the request for all 1000 items. Additionally, the exception caused by this is not handled, meaning following batches won't get processed too.
• Expected: The job shouldn't crash and optimally manage to query all valid ids
• Logs / Screenshots / Proof: Error log of the job:

Seat\Eveapi\Exception\TemporaryEsiOutageException: Undefined 404 response. Original message: Invalid IDs in the request in /var/www/seat/vendor/eveseat/eveapi/src/Jobs/EsiBase.php:488
--
Stack trace:
#0 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/EsiBase.php(291): Seat\Eveapi\Jobs\EsiBase->handleEsiFailedCall(Object(Seat\Eseye\Exceptions\RequestFailedException))
#1 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Assets/Character/Names.php(101): Seat\Eveapi\Jobs\EsiBase->retrieve(Array)
#2 /var/www/seat/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php(39): Seat\Eveapi\Jobs\Assets\Character\Names->Seat\Eveapi\Jobs\Assets\Character\{closure}(Object(Illuminate\Database\Eloquent\Collection), 1)
#3 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Assets/Character/Names.php(114): Illuminate\Database\Eloquent\Builder->chunk(1000, Object(Closure))
#4 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Seat\Eveapi\Jobs\Assets\Character\Names->handle()
#5 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/Util.php(37): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#6 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#7 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#8 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/Container.php(590): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#9 /var/www/seat/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\Container\Container->call(Array)
#10 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#11 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#12 /var/www/seat/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#13 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(83): Illuminate\Bus\Dispatcher->dispatchNow(Object(Seat\Eveapi\Jobs\Assets\Character\Names), false)
#14 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#15 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Middleware/WithoutOverlapping.php(102): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#16 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Seat\Eveapi\Jobs\Middleware\WithoutOverlapping->handle(Object(Seat\Eveapi\Jobs\Assets\Character\Names), Object(Closure))
#17 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Middleware/CheckTokenVersion.php(57): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#18 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Seat\Eveapi\Jobs\Middleware\CheckTokenVersion->handle(Object(Seat\Eveapi\Jobs\Assets\Character\Names), Object(Closure))
#19 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Middleware/CheckTokenScope.php(49): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#20 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Seat\Eveapi\Jobs\Middleware\CheckTokenScope->handle(Object(Seat\Eveapi\Jobs\Assets\Character\Names), Object(Closure))
#21 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Middleware/CheckEsiRouteStatus.php(62): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#22 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Seat\Eveapi\Jobs\Middleware\CheckEsiRouteStatus->handle(Object(Seat\Eveapi\Jobs\Assets\Character\Names), Object(Closure))
#23 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Middleware/CheckServerStatus.php(51): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#24 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Seat\Eveapi\Jobs\Middleware\CheckServerStatus->handle(Object(Seat\Eveapi\Jobs\Assets\Character\Names), Object(Closure))
#25 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Middleware/CheckEsiRateLimit.php(55): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#26 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Seat\Eveapi\Jobs\Middleware\CheckEsiRateLimit->handle(Object(Seat\Eveapi\Jobs\Assets\Character\Names), Object(Closure))
#27 /var/www/seat/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#28 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(85): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#29 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(59): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware(Object(Illuminate\Queue\Jobs\RedisJob), Object(Seat\Eveapi\Jobs\Assets\Character\Names))
#30 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(88): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\RedisJob), Array)
#31 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(368): Illuminate\Queue\Jobs\Job->fire()
#32 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(314): Illuminate\Queue\Worker->process('redis', Object(Illuminate\Queue\Jobs\RedisJob), Object(Illuminate\Queue\WorkerOptions))
#33 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(134): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\RedisJob), 'redis', Object(Illuminate\Queue\WorkerOptions))
#34 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(112): Illuminate\Queue\Worker->daemon('redis', 'characters', Object(Illuminate\Queue\WorkerOptions))
#35 /var/www/seat/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(96): Illuminate\Queue\Console\WorkCommand->runWorker('redis', 'characters')
#36 /var/www/seat/vendor/laravel/horizon/src/Console/WorkCommand.php(46): Illuminate\Queue\Console\WorkCommand->handle()
#37 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\Horizon\Console\WorkCommand->handle()
#38 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/Util.php(37): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#39 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure(Object(Closure))
#40 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#41 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/Container.php(590): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#42 /var/www/seat/vendor/laravel/framework/src/Illuminate/Console/Command.php(134): Illuminate\Container\Container->call(Array)
#43 /var/www/seat/vendor/symfony/console/Command/Command.php(255): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#44 /var/www/seat/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#45 /var/www/seat/vendor/symfony/console/Application.php(1009): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#46 /var/www/seat/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(Laravel\Horizon\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#47 /var/www/seat/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#48 /var/www/seat/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#49 /var/www/seat/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(131): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#50 /var/www/seat/artisan(37): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#51 {main}

• Version Info: latest seat 4
• Proposed solution: Catch the exception so that the next batch can be processed, maybe randomize the order of the ids so that they don't always end up in the same batch. We could also reduce the batch size slightly. More advanced, we could also implement a system that narrows down which id is invalid

[veteranmina]

This issue is mainly occurring due to how assets are handled between esi and SeAT. Currently we are checking if the etag is the same on page one and the expiry is not passed. If conditions are met to not continue to the next page we do not update assets even though say on page 4 there is an asset change.

Esi is sending the same etags for the pages not effected, however one page of many could have a changed etag.