Milestone Reminder GH Action doesn't always work

[nemanjaglumac]

Describe the bug

We have a very useful milestone reminder GH action, but it doesn't always work as expected.
There are two scenarios we need to fix:

1. The action sometimes literally fails because of an exception (example)
2. The action posting a comment on PRs that don't even have the closing keyword (example)

To Reproduce

1. Close a PR that contains this body: Closes #12345 and #23456

or

2. Close a PR that doesn't have a closing keyword at all.

Expected behavior

1. Action should never fail
2. Action should not post comments on PRs that didn't have the closing keyword and a linked issue to begin with

Logs

No response

Information about your Metabase installation

.

Severity

p3

Additional context

Internal.

[darksciencebase]

I'm probably missing something, but if a PR doesn't have a closing keyword, that means it's not connected to an issue, which in turn means the PR itself should have a milestone, so I'm lost about the #2 in the description.

[nemanjaglumac]

which in turn means the PR itself should have a milestone

Not always. There are a ton of tech debt PRs that don't really need a milestone. I'm not aware that we're enforcing a milestone in 100% of cases?

[darksciencebase]

Not always. There are a ton of tech debt PRs that don't really need a milestone. I'm not aware that we're enforcing a milestone in 100% of cases?

not 100% but (1) better too many PRs than too few, (2) many tech debt PRs should have a milestons, (3) the reminder doesn't know a difference. let's continue this discussion on the notion doc explaining our usage of milestones