Retrospective on a problem that occurred after installing this package

[SebastianAwatramani]

I ran into an interesting problem after installing this package that took me about a year to figure out, but which ended up having some fairly serious consequences.

In short, after installing this package, failed jobs were no longer created on job failure.

The longer version: a project I inherited has a highly customized markdown mail template that references a number of variables that are expected to be passed in when creating notifications. However, there are very few notifications in the project, so much so that I didn't even realize there were any.

At some point I started to get these random exceptions in Sentry referencing undefined variables in the default markdown template. They always had to do with one queued email or another (note: email, not notification), but the exception always occurred in the markdown template, not any of the mail templates that the emails were supposed to be using. Moreover, looking through everything related to the emails that were failing never led me anywhere, because everything seemed to be doing what it was supposed to do.

So I got the idea in my head that Laravel was somehow accidentally using the wrong template when rendering emails, and being that this makes no sense, my repeated visits down the rabbit hole never actually led me to a rabbit.

What was actually going on was that whenever there was a problem that would lead to a failed job (generally a connection error), the application would throw an exception when this package attempted to use notification system to send the message because it wasn't passing the necessary variables (which, of course it wasn't because how could it know?).

I finally figured this out by monitoring my queue and noticing that on the third try it never went from running to fail.

In hindsight, there was some definite blindness on my part. In Sentry, the full stack is collapsed by default and instead shows you project files since 99.9% of the time you don't need to look at internals. In this case, had I taken a closer look I would have seen the pattern.