403 returned on comment delivery from streams

[macgirvin]

NodeBB version

No response

NodeBB git hash

No response

NodeJS version

No response

Installed NodeBB plugins

No response

Database type

No response

Database version

No response

Exact steps to cause this issue

Reply was sent to julian at community.nodebb.org from mikedev@fediversity.site to a conversation initiated by julian. I believe julian is following this account.

2024-04-27T20:38:35Z
https://fediversity.site/item/7ac903a0-efcf-46ee-a818-bab85c01240f

Account was rejected with error 403 on multiple attempts. Message had http signature which works on a very wide range of fediverse implementations. The actor record was subsequently fetched by nodebb before return the 403.

{
  "@context":[
    "https://www.w3.org/ns/activitystreams",
    "https://w3id.org/security/v1",
    "https://www.w3.org/ns/did/v1",
    "https://w3id.org/security/multikey/v1",
    "https://w3id.org/security/data-integrity/v1",
    "https://purl.archive.org/socialweb/webfinger",
    {
      "fep":"https://w3id.org/fep/ef61#",
      "aliases":"fep:aliases"
    },
    {
      "nomad":"https://fediversity.site/apschema#",
      "toot":"http://joinmastodon.org/ns#",
      "manuallyApprovesFollowers":"as:manuallyApprovesFollowers",
      "oauthRegistrationEndpoint":"nomad:oauthRegistrationEndpoint",
      "sensitive":"as:sensitive",
      "movedTo":"as:movedTo",
      "discoverable":"toot:discoverable",
      "indexable":"toot:indexable",
      "Hashtag":"as:Hashtag",
      "canReply":"toot:canReply",
      "canSearch":"nomad:canSearch",
      "expires":"nomad:expires",
      "directMessage":"nomad:directMessage",
      "Category":"nomad:Category",
      "copiedTo":"nomad:copiedTo",
      "searchContent":"nomad:searchContent",
      "searchTags":"nomad:searchTags"
    }
  ],
  "type":"Create",
  "id":"https://fediversity.site/activity/7ac903a0-efcf-46ee-a818-bab85c01240f",
  "published":"2024-04-27T19:27:28Z",
  "context":"https://community.nodebb.org/post/99385",
  "actor":"https://fediversity.site/channel/mikedev",
  "url":"https://fediversity.site/activity/7ac903a0-efcf-46ee-a818-bab85c01240f",
  "object":{
    "type":"Note",
    "id":"https://fediversity.site/item/7ac903a0-efcf-46ee-a818-bab85c01240f",
    "published":"2024-04-27T19:27:28Z",
    "attributedTo":"https://fediversity.site/channel/mikedev",
    "inReplyTo":"https://mitra.social/objects/018f1f0d-9391-44ec-6a4d-f64553095da8",
    "context":"https://community.nodebb.org/post/99385",
    "content":"@<span class=\"h-card\"><a class=\"u-url mention\"  href=\"https://community.nodebb.org/user/julian\"  target=\"_blank\"  rel=\"nofollow noopener\" >julian@community.nodebb.org</a></span> @<span class=\"h-card\"><a class=\"u-url mention\"  href=\"https://mastodon.social/@pfefferle\"  target=\"_blank\"  rel=\"nofollow noopener\" >pfefferle@mastodon.social</a></span> @<span class=\"h-card\"><a class=\"u-url mention\"  href=\"https://community.nodebb.org/user/pfefferle\"  target=\"_blank\"  rel=\"nofollow noopener\" >pfefferle@community.nodebb.org</a></span> @<span class=\"h-card\"><a class=\"u-url mention\"  href=\"https://lemmy.ml/u/nutomic\"  target=\"_blank\"  rel=\"nofollow noopener\" >nutomic@lemmy.ml</a></span> @<span class=\"h-card\"><a class=\"u-url mention\"  href=\"https://mitra.social/users/silverpill\"  target=\"_blank\"  rel=\"nofollow noopener\" >silverpill@mitra.social</a></span> @<span class=\"h-card\"><a class=\"u-url mention\"  href=\"https://oisaur.com/@renchap\"  target=\"_blank\"  rel=\"nofollow noopener\" >renchap@oisaur.com</a></span> <br><br>Here's the reason Article became a second class citizen... <br><br><a href=\"https://github.com/mastodon/mastodon/issues/5022\"  target=\"_blank\"  rel=\"nofollow noopener\" >https://github.com/mastodon/mastodon/issues/5022</a><br><br>In this issue I raised against Mastodon in 2017 (on a now defunct github account), Mastodon at the time treated Note and Article identically. In particular, it removed all the HTML except for 'a' tags - even from Article. This made federation with the elephant impossible for us. At this time the ActivityPub fediverse consisted of Hubzilla and Mastodon. Period. The specification wasn't even final yet. Hubzilla provides long-form multi-media content, just like a blog. This content was completely destroyed by Mastodon's HTML sanitizer, especially blockquotes, which displayed everything we quoted as original text and mis-attributed. <br><br>My proposal to the Mastodon team (which was basically Eugen) was to relax the input sanitisation on the Article type a bit , and Mastodon could have their plaintext Note and we could have our multi-media and the fediverse be one happy family. Regardless of the fact that HTML is specified as the default content-type for all content in ActivityPub.<br><br>The response from Eugen was to turn Article into a link, meaning our content wouldn't be shown inline at all - and closing the issue. I believe this is the last time I ever communicated with Eugen and I will never, ever file another issue against Mastodon. <br><br>We started using Note instead, so that our messages would federate at all and knowing that Article would have been the most sensible choice.<br><br>We also need to strip all the images out of our perfectly renderable content and add them back in as attachments - otherwise they won't be displayed on Mastodon. As it turns out, Mastodon only adds back 4 images and reverses the order. This is less than satisfactory because the source content lets us position text around each image, and it forces anybody with multi-media content to not only perform this unnecessary step, but also to check every attachment on import and see if it was already included in the HTML - or it will be displayed twice. <br><br>As far as I'm concerned, Mastodon should be taken to the mountain-top and cast into the volcano. But it appears we're stuck with the infernal thing.",
    "source":{
      "content":"@[url=https://community.nodebb.org/user/julian]julian@community.nodebb.org[/url] @[url=https://mastodon.social/@pfefferle]pfefferle@mastodon.social[/url] @[url=https://community.nodebb.org/user/pfefferle]pfefferle@community.nodebb.org[/url] @[url=https://lemmy.ml/u/nutomic]nutomic@lemmy.ml[/url] @[url=https://mitra.social/users/silverpill]silverpill@mitra.social[/url] @[url=https://oisaur.com/@renchap]renchap@oisaur.com[/url] \r\n\r\nHere's the reason Article became a second class citizen... \r\n\r\n[url=https://github.com/mastodon/mastodon/issues/5022]https://github.com/mastodon/mastodon/issues/5022[/url]\r\n\r\nIn this issue I raised against Mastodon in 2017 (on a now defunct github account), Mastodon at the time treated Note and Article identically. In particular, it removed all the HTML except for 'a' tags - even from Article. This made federation with the elephant impossible for us. At this time the ActivityPub fediverse consisted of Hubzilla and Mastodon. Period. The specification wasn't even final yet. Hubzilla provides long-form multi-media content, just like a blog. This content was completely destroyed by Mastodon's HTML sanitizer, especially blockquotes, which displayed everything we quoted as original text and mis-attributed. \r\n\r\nMy proposal to the Mastodon team (which was basically Eugen) was to relax the input sanitisation on the Article type a bit , and Mastodon could have their plaintext Note and we could have our multi-media and the fediverse be one happy family. Regardless of the fact that HTML is specified as the default content-type for all content in ActivityPub.\r\n\r\nThe response from Eugen was to turn Article into a link, meaning our content wouldn't be shown inline at all - and closing the issue. I believe this is the last time I ever communicated with Eugen and I will never, ever file another issue against Mastodon. \r\n\r\nWe started using Note instead, so that our messages would federate at all and knowing that Article would have been the most sensible choice.\r\n\r\nWe also need to strip all the images out of our perfectly renderable content and add them back in as attachments - otherwise they won't be displayed on Mastodon. As it turns out, Mastodon only adds back 4 images and reverses the order. This is less than satisfactory because the source content lets us position text around each image, and it forces anybody with multi-media content to not only perform this unnecessary step, but also to check every attachment on import and see if it was already included in the HTML - or it will be displayed twice. \r\n\r\nAs far as I'm concerned, Mastodon should be taken to the mountain-top and cast into the volcano. But it appears we're stuck with the infernal thing.",
      "mediaType":"text/x-multicode"
    },
    "url":"https://fediversity.site/item/7ac903a0-efcf-46ee-a818-bab85c01240f",
    "tag":[
      {
        "type":"Mention",
        "href":"https://community.nodebb.org/user/pfefferle",
        "name":"@pfefferle@community.nodebb.org"
      },
      {
        "type":"Mention",
        "href":"https://community.nodebb.org/user/julian",
        "name":"@julian@community.nodebb.org"
      },
      {
        "type":"Mention",
        "href":"https://mastodon.social/@pfefferle",
        "name":"@pfefferle@mastodon.social"
      },
      {
        "type":"Mention",
        "href":"https://mitra.social/users/silverpill",
        "name":"@silverpill@mitra.social"
      },
      {
        "type":"Mention",
        "href":"https://oisaur.com/@renchap",
        "name":"@renchap@oisaur.com"
      },
     {
        "type":"Mention",
        "href":"https://lemmy.ml/u/nutomic",
        "name":"@nutomic@lemmy.ml"
      }
    ],
    "to":[
      "https://www.w3.org/ns/activitystreams#Public",
      "https://community.nodebb.org/uid/27655",
      "https://community.nodebb.org/uid/2",
      "https://mastodon.social/users/pfefferle",
      "https://mitra.social/users/silverpill",
      "https://oisaur.com/users/renchap",
      "https://lemmy.ml/u/nutomic"
    ],
    "cc":[
      "https://fediversity.site/followers/mikedev",
      "https://community.nodebb.org/uid/2/followers",
      "https://socialhub.activitypub.rocks/ap/actor/30d35c07698335c03f2bb89d3a51a02f"
    ]
  },
  "tag":[
    {
      "type":"Mention",
      "href":"https://community.nodebb.org/user/pfefferle",
      "name":"@pfefferle@community.nodebb.org"
    },
    {
      "type":"Mention",
      "href":"https://community.nodebb.org/user/julian",
      "name":"@julian@community.nodebb.org"
    },
    {
      "type":"Mention",
      "href":"https://mastodon.social/@pfefferle",
      "name":"@pfefferle@mastodon.social"
    },
    {
      "type":"Mention",
      "href":"https://mitra.social/users/silverpill",
      "name":"@silverpill@mitra.social"
    },
    {
      "type":"Mention",
      "href":"https://oisaur.com/@renchap",
      "name":"@renchap@oisaur.com"
    },
    {
      "type":"Mention",
      "href":"https://lemmy.ml/u/nutomic",
      "name":"@nutomic@lemmy.ml"
    }
  ],
  "to":[
    "https://www.w3.org/ns/activitystreams#Public",
    "https://community.nodebb.org/uid/27655",
    "https://community.nodebb.org/uid/2",
    "https://mastodon.social/users/pfefferle",
    "https://mitra.social/users/silverpill",
    "https://oisaur.com/users/renchap",
    "https://lemmy.ml/u/nutomic"
  ],
  "cc":[
    "https://fediversity.site/followers/mikedev",
    "https://community.nodebb.org/uid/2/followers",
    "https://socialhub.activitypub.rocks/ap/actor/30d35c07698335c03f2bb89d3a51a02f"
  ],
  "proof":{
    "type":"DataIntegrityProof",
    "cryptosuite":"eddsa-jcs-2022",
    "created":"2024-04-27T20:38:34Z",
    "verificationMethod":"https://fediversity.site/channel/mikedev#z6MkhvhVi1Fikiad6TxqEHmRmyePZJiyvuNa3KPJbpr3BgPb",
    "proofPurpose":"assertionMethod",
    "proofValue":"z4y6XJPPjz3GFkhU4J6ek3CrnPEysX4czYpX3Xdt8tyqcaE6mF9rAfveMedx2nu9P5iFDbb9hMDAQxFjLwYHf2CKY"
  }
}

What you expected

No response

What happened instead

No response

Anything else?

Not urgent, just trying to sort potential interop issues.

[macgirvin]

It appears julian is not actually following this account, even though it appears that way from my software. Closing.

[julianlam]

One sec, that shouldn't stop it from being accepted.

[julianlam]

We do a check for a pre-existing relationship, and one of those is whether the comment resolves back to an existing topic. In your case, even without a follow from me, the condition should've been satisfied.

This sounds like a bug we need to look into.

[julianlam]

@macgirvin I see the reply actually made it through. Did something change in the interim?

[macgirvin]

I still see nothing but 403 returns here. But there was a reply to my comment from silverpill with the same mentions and perhaps that resulted in pulling it in. TBH, I'm not certain exactly what happened.

[julianlam]

But there was a reply to my comment from silverpill

Ah that's a good point, that might be why. I'll take a closer look.

Thank you for reporting!

[macgirvin]

After some digging, this appears to be my own bug. We have some quite extensive permissions. The fetch permissions on that comment ended up being my current default - followers only. So even though it was posted to your site and the conversation is public, your site couldn't actually fetch the activity unless it used your credentials. I think this is what happened. The permissions on my activity should be public because it was part of a public conversation, regardless of my personal preference. I'll try and get this sorted.

[julianlam]

For what it's worth I've actually tried to follow you, but I'm not sure why it doesn't complete (might be my follow isn't accepted)

[macgirvin]

I show you as following and accepted. So much for a quiet Sunday.... looks like I'm going to be tracking weird bugs. I might try deleting the connection and starting over. Couldn't hurt at this point.

[julianlam]

In that case it may suggest that the accept from you just wasn't properly handled 🤷 likely something for us to address

[macgirvin]

Deleted my side of the connection and started over (I've sent a follow). We'll see how that goes. If that works we can try it going the other way. I might give you a less public account to test against since the logs rotate pretty fast on my primary site. You can try following slosh@unfediverse.com ... though I'm about to get called away for chores. I'll approve it when I get a chance and let you know here.

[julianlam]

Sadly did not receive. NodeBB doesn't have a concept of follow approvals so an Accept should've been sent back immediately.

Will check my logs soon. Also have chores to do 😑

It seems like whatever content is being sent my way from fediversity is rejected for whatever reason, but if requested from my end, is ok (e.g. I was able to successfully retrieve your post)

[macgirvin]

Yeah, didn't see any Accept here. But I've been called away. Will have to take up at a later time. I might need to give you a test account here so you can check your side on your own schedule.

[julianlam]

Sure, we'll try again another time. Happy to test with a local account on your service if you'd like. I sent a follow from my dev instance (bb.devnull.land) hoping to see something come back but I got nothing, I guess the follow needs to be approved?

[macgirvin]

S'rry - approved this around 12-13 hours ago, but Microsoft's SMS 2FA service was borked so I couldn't login here and let you know.

[macgirvin]

Just accepted dragonfruit, which I assume is yours (correctly or incorrectly assumed)

[julianlam]

It is, but I didn't realize it was a different account. Now that you've accepted both will subsequent follows automatically bounce an approve back?

Edit: The answer is yes, here's what I see on my end:

24-04-29T01:12:21.457Z [4567/35216] - verbose: [middleware/activitypub] Validating incoming payload...
2024-04-29T01:12:21.458Z [4567/35216] - verbose: [activitypub/verify] Starting signature verification...
2024-04-29T01:12:21.459Z [4567/35216] - verbose: [activitypub/verify] Retrieving pubkey for https://fediversity.site/channel/mikedev?operation=rsakey
2024-04-29T01:12:21.463Z [4567/35216] - verbose: [activitypub/get] https://fediversity.site/channel/mikedev?operation=rsakey
2024-04-29T01:12:21.795Z [4567/35216] - verbose: [activitypub/verify] Attempting signed string verification
2024-04-29T01:12:21.798Z [4567/35216] - verbose: [middleware/activitypub] HTTP signature verification passed.
2024-04-29T01:12:21.798Z [4567/35216] - verbose: [middleware/activitypub] Request body check passed.
2024-04-29T01:12:21.799Z [4567/35216] - verbose: [middleware/activitypub] Origin check failed, stripping object down to id.
2024-04-29T01:12:21.800Z [4567/35216] - verbose: [middleware/activitypub] Origin check passed.
2024-04-29T01:12:21.802Z [4567/35216] - verbose: [middleware/activitypub] Key ownership cross-check failed.

The "key ownership cross-check" ensures that the claimed actor in the received payload actually controls the keyId received in the signature. I'll have to check to see what's up.

[julianlam]

Got it, it was naive logic in how I broke apart the signature string. I wasn't accounting for values that contained equal signs, of which yours uses (?operation=)

[macgirvin]

Cool. Thanks. I've been waffling on using fragments for these things like everybody else does, but the webserver never sees fragments on inbound urls - and I kind of think it's important for the webserver to have knowledge of what exactly was requested of it.