slack.crossplane.io is down

[jbw976]

What happened?

Navigating to https://slack.crossplane.io/ shows an error page with:

Application error
An error occurred in the application and your page could not be served. If you are the application owner, [check your logs for details](https://devcenter.heroku.com/articles/logging#view-logs). You can do this from the Heroku CLI with the command
heroku logs --tail

The application logs from https://dashboard.heroku.com/apps/crossplane-slackin/logs show:

2023-03-15T20:42:31.567428+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/socket.io/?EIO=3&transport=polling&t=ORd0fHb" host=slack.crossplane.io request_id=3aa8cb43-43f5-47db-aefd-8d7712e3fb4b fwd="24.191.209.129" dyno= connect= service= status=503 bytes= protocol=https
2023-03-15T20:42:31.658673+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=slack.crossplane.io request_id=08881835-7b6d-448a-9ab1-9d6c7cf221e0 fwd="50.46.239.117" dyno= connect= service= status=503 bytes= protocol=https

This same issue is being seen at slack.rook.io and slack.openinfra.mx - wonder if others are seeing it too?

Temporary Workaround

For folks that want to join the Crossplane Slack workspace, this link should work in the meantime:
https://join.slack.com/t/crossplane/shared_invite/zt-1rkkaeelc-WKrorO7PAJllVmYTNyodiA

[jbw976]

Restarted all dynos and still seeing this behavior

[jbw976]

Updated to the heroku-22 stack of node.js with no improvement

[jbw976]

Aha, we may be getting somehwere now - connecting via the heroku console and trying to run the command from Procfile manually says that the certificate has expired:

/app/dist/slack.js:54
          throw err;
          ^

Error: certificate has expired
    at Error (native)
    at TLSSocket.<anonymous> (_tls_wrap.js:1092:38)
    at emitNone (events.js:86:13)
    at TLSSocket.emit (events.js:185:7)
    at TLSSocket._finishInit (_tls_wrap.js:610:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:440:38)

[jbw976]

Which doesn't actually look to be the case from the heroku CLI's point of view 🤔

❯ heroku certs:auto --app=crossplane-slackin
 ›   Warning: heroku update available from 7.68.2 to 7.69.1.
=== Automatic Certificate Management is enabled on crossplane-slackin

Certificate details:
Common Name(s): slack.crossplane.io
Domain(s):      cd258fe0-8812-4f36-bc71-546e6b9e7763
Expires At:     2023-04-21 07:21 UTC
Issuer:         /C=US/O=Let's Encrypt/CN=R3
Starts At:      2023-01-21 07:21 UTC
Subject:        /CN=slack.crossplane.io
SSL certificate is verified by a root authority.

Domain               Status       Last Updated
───────────────────  ───────────  ────────────
slack.crossplane.io  Cert issued  23 days

[jbw976]

i'm also frequently seeing this error when running the app manually in the heroku console too, in addition to the certificate expiration error above - wonder if something on the slack API side changed recently?

/app/dist/slack.js:62
        var team = res.body.team;
                      ^

TypeError: Cannot read property 'body' of undefined
    at /app/dist/slack.js:62:23
    at Request.callback (/app/node_modules/superagent/lib/node/index.js:699:3)
    at ClientRequest.<anonymous> (/app/node_modules/superagent/lib/node/index.js:636:10)
    at ClientRequest.g (events.js:292:16)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at TLSSocket.socketErrorListener (_http_client.js:309:9)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:188:7)
    at emitErrorNT (net.js:1277:8)

[jbw976]

I'm able to test making Slack API calls using curl from inside the dyno. Looks like there are 3 API calls that https://github.com/bassam/slackin/blob/master/lib/slack.js makes during initialization: channels.list, team.info, users.list.

I'm able to get valid responses with real data for team.info and users.list, but I get an error response from channels.list:

curl -X GET -H 'Authorization: Bearer <redacted>' -H 'Content-type: application/json' https://crossplane.slack.com/api/channels.list
{"ok":false,"error":"unknown_method","req_method":"channels.list"}

rauchg/slackin#417 indicates that this API was deprecated, but looks like that should have happened back in 2021, so I'm not sure that's what could all of a sudden start causing this issue.

[jbw976]

If we can't figure this out, it may be reasonable to move to Slack's invite link functionality, e.g. https://slack.com/help/articles/201330256-Invite-new-members-to-your-workspace#share-an-invitation-link

They can be set to "never expire", but are still limited with "Each invite link can be used by up to 400 people", so we would have to refresh the link every so often 🤔

[jbw976]

This issue has been resolved with a fairly straightforward solution, implemented in https://github.com/crossplane/slack.crossplane.io. That repo is essentially a static HTML page that redirects to an official Slack provided invitation link. This allows people to still navigate to slack.crossplane.io, but then be taken to a full invite page with a longer URL path on slack.com - something that simple DNS doesn't allow you to do.

I am closing this issue for now since the immediate problem is solved, but will open a new issue to track a possible longer term solution and follow-up items to consider.