Stale or published elsewhere `ApplicationMetadataMessage_COMMUNITY_DESCRIPTION` public messages #4922

felicio · 2024-03-14T10:03:56Z

Problem

In Store nodes, last ApplicationMetadataMessage_COMMUNITY_DESCRIPTION for zQ3shZeEJqTC1xhGUjxuS4rtHSrhJ8vUYp64v6qWkLpvdy9L9 (Status CC Community) is from around 2024-02-28T14:57:13.580Z
Another Community (zQ3shgz9XWZerfqsSpR2XKjfyiwtyiRNV6whj6BXREwnuLo5P) gets recent updates

Summary from comments below

don't have ProtocolMessage#public_message set
have ProtocolMessage#encrypted_message['none'].payload set but without any private material needed to decode it
aren't signed by the community's private key so cannot be verified validate metadata for communities with owner token status-web#541
while CommunityDescription#chats are empty the CommunityDescription#categories are set Categories should be encrypted for closed communities #4943
while CommunityDescription#members are empty the CommunityDescription#active_members_count isn't active_members_count should be encrypted for closed communities #4944
possible COMMUNITY_DESCRIPTION messages larger than 7MB Lower the size of maximum waku message to 125KiB #4955

Implementation

Known steps towards feature implementation.
What needs further specifying and investigating.

Acceptance Criteria

Rules for the future PR to be accepted.

Notes

pubsub topic: /waku/2/rs/16/32
content topic: /waku/1/0x13267e91/rfc26
fleet: shards.test

Future Steps

Steps which should be taken after this issue has been resolved.

The text was updated successfully, but these errors were encountered:

felicio · 2024-03-14T10:04:17Z

Cc @osmaczko

felicio · 2024-03-14T10:05:05Z

Is it possible that a content topic format would change for certain Communities?

jrainville · 2024-03-14T14:19:46Z

We changed fleet to use the new one called shards-test, so that might be it.

Also, that community is not really used anymore, so it's possible that Iuri shut down the control node for it cc @iurimatias

felicio · 2024-03-14T14:25:03Z

@jrainville the description mentions shards.test.

Which community is used then, please?

jrainville · 2024-03-14T14:26:25Z

Which community is used then, please?

@felicio this is the new community https://status.app/c/G4IAAMSIuW0ORJLY43bdG0nzRKzhlAOHMJAkwRaAHZJ7yqYUxajs35U_WeFesRAnUVV96woIFyIvCSUFsiKN4L-2ESkypLDwYw-S5xztCs8IqQXB99foPIShRGs8H6mTpoTgr1fieI3s7HWt0-cH-_Px_DjNaXrUAA==#zQ3shZeEJqTC1xhGUjxuS4rtHSrhJ8vUYp64v6qWkLpvdy9L9

felicio · 2024-03-14T14:31:41Z

@jrainville again, that's what the description references.

Could you please have a closer look at the issue?

jrainville · 2024-03-14T15:04:23Z

@jrainville again, that's what the description references.

Could you please have a closer look at the issue?

Sorry for the confusion, you named the community Status CC Community, which is the name of the old community, so I thought you were talking about that one.

I'm not sure what's going on here then. We clearly receive the latest updates in peer to peer, since I can see all the new channels.

Is it possible that a content topic format would change for certain Communities?

If we shard the community, yes that would be the case. I don't recall that we sharded this one however.

@iurimatias can you check the shard popup which topic it is using?

felicio · 2024-03-15T01:00:42Z

@jrainville thank you. One more thing to reiterate, however, this issue focuses primarily on Store nodes.

osmaczko · 2024-03-15T15:27:33Z

The shard is not set, the pubSubTopic we fetch community info from is /waku/2/rs/16/64

status-go/protocol/common/shard/shard.go

Lines 58 to 63 in 38183ab

    
           func DefaultShard() *Shard { 
        
           	return &Shard{ 
        
           		Cluster: MainStatusShardCluster, 
        
           		Index:   NonProtectedShardIndex, 
        
           	} 
        
           }

status-go/protocol/common/shard/shard.go

Line 51 in 38183ab

const NonProtectedShardIndex = 64

jrainville · 2024-03-15T16:37:25Z

pubsub topic: "/waku/2/rs/16/32"
we fetch community info from is /waku/2/rs/16/64

@felicio can you check if you update the topic if it works?

osmaczko · 2024-03-15T18:22:07Z

Sorry, I might have misinterpreted the code (the code is misleading tbh - which one is the default? 😵 ). I just double-checked, and we publish CommunityDescription on /waku/2/rs/16/32. I got it mixed up with where we share ApplicationMetadataMessage_COMMUNITY_PUBLIC_SHARD_INFO.

osmaczko · 2024-03-15T18:59:15Z

@felicio I am able to fetch CommunityDescription from today:

store node: store-01.do-ams3.shards.test
description envelope: 0xc06f05cb5f15eab1424c57acb8f05bb11484786cffdba97a4c79d73565a1e4cf
timestamp: 1710508493775973252 (2024-03-15 13:14:53.775973252 +0000 UTC)
size: 530204 bytes
contentTopic: /waku/1/0x13267e91/rfc26
pubsubTopic: /waku/2/rs/16/32

felicio · 2024-03-17T01:59:56Z

@osmaczko I saw some recent messages of similar sizes for that community, but again since around 2024-02-28T14:57:13.580Z they

don't have ProtocolMessage#public_message set
have ProtocolMessage#encrypted_message['none'].payload set but without any private material needed to decode it
aren't signed by the community's private key so cannot be verified

And as a side note

while CommunityDescription#chats are empty the CommunityDescription#categories are set
while CommunityDescription#members are empty the CommunityDescription#active_members_count isn't

osmaczko · 2024-03-17T11:46:35Z

don't have ProtocolMessage#public_message set

have ProtocolMessage#encrypted_message['none'].payload set but without any private material needed to decode it

We don't wrap CommunityDescription into ProtocolMessage when we publish the org 🤔 Not sure what is the logic in status-web, but I would expect ProtocolMessage to be skipped altogether due to protobuf unmarshaling fail.

However, if it used to work, I am pretty sure, it is not working now because of messages segmentation.

aren't signed by the community's private key so cannot be verified

Oh, I see. The behavior changed a few months ago. Once the owner-token is minted, the signer of the community is no longer the community's private key. The signer becomes the owner of the token. I'll send you a link to the presentation about that offline.

while CommunityDescription#chats are empty the CommunityDescription#categories are set

while CommunityDescription#members are empty the CommunityDescription#active_members_count isn't

Could you share the CommunityDescription protobuf you are referring to, please?

felicio · 2024-03-17T15:27:06Z

@osmaczko

We don't wrap CommunityDescription into ProtocolMessage

Step into

status-go/protocol/messenger_communities.go

Line 134 in 38183ab

_, err = m.sender.SendPublic(context.Background(), org.IDString(), rawMessage)

if you can see the wrapping there.

However, if it used to work,

It still works the same for zQ3shgz9XWZerfqsSpR2XKjfyiwtyiRNV6whj6BXREwnuLo5P, but works unexpectedly, as listed above, for zQ3shZeEJqTC1xhGUjxuS4rtHSrhJ8vUYp64v6qWkLpvdy9L9.

The behavior changed a few months ago.

How do you validate the message then?

Could you share the CommunityDescription protobuf you are referring to, please?

Attaching the top-most waku (e.g. "ProtocolMessage > ApplicationMetadataMessage > CommunityDescription" message payload.txt – there's nothing specific about "status-web".

osmaczko · 2024-03-17T17:48:14Z

Step into ... if you can see the wrapping there.

Message is wrapped only into ApplicationMetadataMessage. ProtocolMessage layer is skipped.

It still works the same for zQ3shgz9XWZerfqsSpR2XKjfyiwtyiRNV6whj6BXREwnuLo5P, but works unexpectedly, as listed above, for zQ3shZeEJqTC1xhGUjxuS4rtHSrhJ8vUYp64v6qWkLpvdy9L9.

That's most likely because zQ3shgz9XWZerfqsSpR2XKjfyiwtyiRNV6whj6BXREwnuLo5P is still small enough to stay unsegmented.

How do you validate the message then?

I sent you the link to the video with detailed explanations. Long story short, we queue CommunityDescription messages until we know the owner, then we validate the message against that known owner. Code paths:

queue:

status-go/protocol/communities/manager.go

Line 1774 in 38183ab

return nil, m.Queue(signer, community, processedDescription.Clock, payload)

status-go/protocol/communities/manager.go

Line 1782 in 38183ab

return nil, m.Queue(signer, community, processedDescription.Clock, payload)
owner verification loop:

status-go/protocol/communities/manager.go

Line 475 in 38183ab

func (m *Manager) runOwnerVerificationLoop() {
community validation against current owner:

status-go/protocol/communities/manager.go

Line 515 in 38183ab

func (m *Manager) validateCommunity(communityToValidateData []communityToValidate) (*CommunityResponse, error) {

felicio · 2024-03-17T23:43:45Z

@osmaczko

Message is wrapped only into ApplicationMetadataMessage.

So this

status-go/protocol/common/message_sender.go

Line 662 in 9a133d5

messageSpec, err := s.protocol.BuildPublicMessage(s.identity, wrappedMessage)

is not run?

ProtocolMessage layer is skipped.

So it would not fall into

status-go/protocol/common/message_sender.go

Line 668 in 9a133d5

if len(rawMessage.HashRatchetGroupID) != 0 {

because of

status-go/protocol/messenger_communities.go

Line 131 in 38183ab

rawMessage.HashRatchetGroupID = org.ID()

?

is still small enough to stay unsegmented.

Isn't the limit for segmentation like 7 MB?
Wasn't the message you talked about like 500 KB?
Why would those messages be split?

That's most likely because

Let me point out again, even those types of messages for zQ3shZeEJqTC1xhGUjxuS4rtHSrhJ8vUYp64v6qWkLpvdy9L9 aren't split.

Also, based on what you've shared so far, I believe you might have got the description from COMMUNITY_EVENTS_MESSAGE_REJECTED instead of COMMUNITY_DESCRIPTION.

Let me know if you'd prefer a short meeting.

osmaczko · 2024-03-18T10:46:48Z

I double checked and indeed we wrap the Description into ProtocolMessage 😓 It has changed recently for encrypted communities, we didn't do that before (public communities are not wrapped into ProtocolMessage I believe). The wrapping of ApplicationMetadataMessage happens here:

status-go/protocol/common/message_sender.go

Line 684 in 224103d

    
           spec, err := s.protocol.BuildHashRatchetReKeyGroupMessage(s.identity, rawMessage.Recipients, rawMessage.HashRatchetGroupID, wrappedMessage, ratchet)

and that's how it is constructed exactly:

status-go/protocol/encryption/protocol.go

Lines 322 to 333 in 224103d

    
           Message: &ProtocolMessage{ 
        
           	InstallationId: p.encryptor.config.InstallationID, 
        
           	EncryptedMessage: map[string]*EncryptedMessageProtocol{noInstallationID: &EncryptedMessageProtocol{ 
        
           		HRHeader: &HRHeader{ 
        
           			SeqNo:   0, 
        
           			GroupId: groupID, 
        
           			Keys:    keys, 
        
           		}, 
        
           		Payload: payload, 
        
           	}, 
        
           	}, 
        
           },

I also checked how we exactly process the message and indeed it is not segmented.
Handling of description envelope 1: 0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12, timestamp: 1710530403844026612 (2024-03-15 19:20:03.844026612 +0000 UTC), size: 532508 bytes, contentTopic: /waku/1/0x13267e91/rfc26, pubsubTopic: /waku/2/rs/16/32:

2024-03-18T10:13:24.530+0100	INFO	user-waku-0	wakuv2/waku.go:1103	received waku2 store message	{"envelopeHash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12", "pubsubTopic": "/waku/2/rs/16/32", "timestamp": 1710530403844026612}
2024-03-18T10:13:24.530+0100	DEBUG	user-waku-0	wakuv2/waku.go:1327	received new envelope	{"messageType": 1, "envelopeHash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12", "contentTopic": "/waku/1/0x13267e91/rfc26", "timestamp": 1710530403844026612}

2024-03-18T10:13:24.568+0100	DEBUG	user-messenger-0	transport/transport.go:262	message not cached	{"Transport": {"site": "retrieveRawAll", "hash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12"}}
2024-03-18T10:13:24.568+0100	DEBUG	user-messenger-0	common/message_sender.go:1362	handling message segment	{"site": "handleSegmentationLayer", "hash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12", "EntireMessageHash": "0x", "Index": 0, "SegmentsCount": 0}
2024-03-18T10:13:24.568+0100	DEBUG	user-messenger-0	common/message_sender.go:902	failed to handle segmentation layer message	{"site": "handleMessage", "hash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12", "error": "invalid segments count", "errorVerbose": "invalid segments count"}
2024-03-18T10:13:24.569+0100	DEBUG	user-messenger-0	encryption/protocol.go:714	received a protocol message	{"Protocol": {"site": "HandleMessage", "sender-public-key": "0x04953f5f0d355b37c39d1d6460a31ed1114455f8263b3fd1b84406c5f12c9eb7dfb76ba7513b92186010928254984fe98aee069b4c7e20f9ea3da497c3ae769477", "my-installation-id": "844ba546-7516-4c2e-b067-5d24f7a162df", "messageID": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12"}}
2024-03-18T10:13:24.569+0100	DEBUG	user-messenger-0	common/message_sender.go:927	failed to handle datasync message	{"site": "handleMessage", "hash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12", "error": "handling non-datasync message"}

2024-03-18T10:13:24.577+0100	DEBUG	user-messenger-0	protocol/messenger.go:3904	processing messages further	{"site": "RetrieveAll", "hash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12", "count": 1}
2024-03-18T10:13:24.577+0100	INFO	user-messenger-0	protocol/messenger.go:3908	processing message	{"site": "RetrieveAll", "hash": "0xf8e170716e3caa9d49e12b5ec1aa1ae680933dc5c973d6f23faa97c315c4cb12", "message-id": "0xe242e775379544c62a196b1b315fe8a9926eeb34eaf7997bab514aea3a55d52d"}
2024-03-18T10:13:24.577+0100	INFO	user-messenger-0	protocol/messenger.go:3920	processing message	{"type": "COMMUNITY_DESCRIPTION", "senderID": "0x04953f5f0d355b37c39d1d6460a31ed1114455f8263b3fd1b84406c5f12c9eb7dfb76ba7513b92186010928254984fe98aee069b4c7e20f9ea3da497c3ae769477"}
2024-03-18T10:13:24.578+0100	INFO	user-messenger-0	protocol/messenger_handlers.go:668	handling CommunityDescription
2024-03-18T10:13:24.579+0100	DEBUG	user-messenger-0	communities/manager.go:1708	HandleCommunityDescriptionMessage	{"communityID": "0x02b5bdaf5a25fcfe2ee14c501fab1836b8de57f61621080c3d52073d16de0d98d6", "clock": 1710529833209}
2024-03-18T10:13:24.579+0100	DEBUG	user-messenger-0	communities/community_description_encryption.go:83	failed to decrypt community private data	{"keyIDSeqNo": "618891d4ace09777ffd614ad998e91d5c5b37123c2b7b61fe3886afd1d0bd1d1316", "error": "no ratchet key for given keyID", "errorVerbose": "no ratchet key for given keyID"}
2024-03-18T10:13:24.580+0100	DEBUG	user-messenger-0	communities/community_description_encryption.go:83	failed to decrypt community private data	{"keyIDSeqNo": "14cb119293e26251202d6036aebf6f870dfc92ff1e6d97adb40f751478be653e879", "error": "no ratchet key for given keyID", "errorVerbose": "no ratchet key for given keyID"}
2024-03-18T10:13:24.582+0100	INFO	user-messenger-0	communities/manager.go:1689	queuing community	{"id": "0x02b5bdaf5a25fcfe2ee14c501fab1836b8de57f61621080c3d52073d16de0d98d6", "name": "Status", "clock": 1710529833209, "signer": "0x04953f5f0d355b37c39d1d6460a31ed1114455f8263b3fd1b84406c5f12c9eb7dfb76ba7513b92186010928254984fe98aee069b4c7e20f9ea3da497c3ae769477"}

So going back to your points:

don't have ProtocolMessage#public_message set

Yes, that's the case. We don't set it.

have ProtocolMessage#encrypted_message['none'].payload set but without any private material needed to decode it

Paylod is unencrypted ApplicationMetadataMessage in that case. Since, we have HRHeader with seqNo==0 we return payload as is when decrypting:

status-go/protocol/encryption/encryptor.go

Lines 715 to 718 in 224103d

    
           // Key exchange message, nothing to decrypt 
        
           if seqNo == 0 { 
        
           	return payload, nil 
        
           }

Hope this explains the missing bits, if not, please let me know, we'll do a short meeting.

jrainville · 2024-03-18T16:18:22Z

Closing as the issue is on the Web side

felicio · 2024-03-19T02:36:45Z

@jrainville

don't have ProtocolMessage#public_message set

have ProtocolMessage#encrypted_message['none'].payload set but without any private material needed to decode it

@osmaczko confirmed that's the behavior now, but don't you guys think these public messages that don't carry any keys nor require any other private information to be decoded should have the same data structure as those for public communities?

while CommunityDescription#chats are empty the CommunityDescription#categories are set

while CommunityDescription#members are empty the CommunityDescription#active_members_count isn't

Did you also confirm these? And are you going to create follow-up issues?

possible COMMUNITY_DESCRIPTION messages larger than 7MB

Given fast lookups, why would a message with public community metadata ever be larger than 7MB? And if some of it is not public, shouldn't it be sent on another content topic?

osmaczko · 2024-03-19T10:03:05Z

Given fast lookups, why would a message with public community metadata ever be larger than 7MB?

Waku message limit defined in status-go is 1MiB. Message payload is limited to 0.75MiB. We reached this limit in the past already: status-im/status-desktop#12188. Also please note that in the future, the maximum size of a message might be reduced dramatically from 1MiB to 150KiB: status-im/status-desktop#12188 (comment). Currently, the size of Status' CommunityDescription is ~0.5MiB - we are close to the limit.

And if some of it is not public, shouldn't it be sent on another content topic?

We wanted to avoid the complexity of reconstructing the Community state from data being spread across multiple messages/topics. I suppose we could consider sending only the public data on another content topic if needed.

osmaczko · 2024-03-19T10:20:29Z

Did you also confirm these? And are you going to create follow-up issues?

Indeed. Thanks for heads-up. Created issues: #4943, #4944

felicio · 2024-03-20T07:53:48Z

Thanks, clarified the data structure and if you decide not to create an issue for that too, I'll close this.

Also, this issue is not blocking.

felicio mentioned this issue Mar 14, 2024

handle larger messages status-im/status-web#531

Merged

jrainville assigned osmaczko Mar 15, 2024

felicio changed the title ~~Stale or published elsewhere ApplicationMetadataMessage_COMMUNITY_DESCRIPTION updates~~ Stale or published elsewhere ApplicationMetadataMessage_COMMUNITY_DESCRIPTION public updates Mar 17, 2024

jrainville closed this as completed Mar 18, 2024

felicio changed the title ~~Stale or published elsewhere ApplicationMetadataMessage_COMMUNITY_DESCRIPTION public updates~~ Stale or published elsewhere ApplicationMetadataMessage_COMMUNITY_DESCRIPTION public messages Mar 19, 2024

felicio reopened this Mar 19, 2024

felicio closed this as completed Mar 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stale or published elsewhere `ApplicationMetadataMessage_COMMUNITY_DESCRIPTION` public messages #4922

Stale or published elsewhere `ApplicationMetadataMessage_COMMUNITY_DESCRIPTION` public messages #4922

felicio commented Mar 14, 2024 •

edited

felicio commented Mar 14, 2024

felicio commented Mar 14, 2024 •

edited

jrainville commented Mar 14, 2024

felicio commented Mar 14, 2024

jrainville commented Mar 14, 2024

felicio commented Mar 14, 2024

jrainville commented Mar 14, 2024

felicio commented Mar 15, 2024 •

edited

osmaczko commented Mar 15, 2024

jrainville commented Mar 15, 2024

osmaczko commented Mar 15, 2024

osmaczko commented Mar 15, 2024

felicio commented Mar 17, 2024 •

edited

osmaczko commented Mar 17, 2024 •

edited

felicio commented Mar 17, 2024 •

edited

osmaczko commented Mar 17, 2024

felicio commented Mar 17, 2024 •

edited

osmaczko commented Mar 18, 2024

jrainville commented Mar 18, 2024

felicio commented Mar 19, 2024 •

edited

osmaczko commented Mar 19, 2024

osmaczko commented Mar 19, 2024 •

edited

felicio commented Mar 20, 2024

Stale or published elsewhere ApplicationMetadataMessage_COMMUNITY_DESCRIPTION public messages #4922

Stale or published elsewhere ApplicationMetadataMessage_COMMUNITY_DESCRIPTION public messages #4922

Comments

felicio commented Mar 14, 2024 • edited

Problem

Summary from comments below

Implementation

Acceptance Criteria

Notes

Future Steps

felicio commented Mar 14, 2024

felicio commented Mar 14, 2024 • edited

jrainville commented Mar 14, 2024

felicio commented Mar 14, 2024

jrainville commented Mar 14, 2024

felicio commented Mar 14, 2024

jrainville commented Mar 14, 2024

felicio commented Mar 15, 2024 • edited

osmaczko commented Mar 15, 2024

jrainville commented Mar 15, 2024

osmaczko commented Mar 15, 2024

osmaczko commented Mar 15, 2024

felicio commented Mar 17, 2024 • edited

osmaczko commented Mar 17, 2024 • edited

felicio commented Mar 17, 2024 • edited

osmaczko commented Mar 17, 2024

felicio commented Mar 17, 2024 • edited

osmaczko commented Mar 18, 2024

jrainville commented Mar 18, 2024

felicio commented Mar 19, 2024 • edited

osmaczko commented Mar 19, 2024

osmaczko commented Mar 19, 2024 • edited

felicio commented Mar 20, 2024

Stale or published elsewhere `ApplicationMetadataMessage_COMMUNITY_DESCRIPTION` public messages #4922

Stale or published elsewhere `ApplicationMetadataMessage_COMMUNITY_DESCRIPTION` public messages #4922

felicio commented Mar 14, 2024 •

edited

felicio commented Mar 14, 2024 •

edited

felicio commented Mar 15, 2024 •

edited

felicio commented Mar 17, 2024 •

edited

osmaczko commented Mar 17, 2024 •

edited

felicio commented Mar 17, 2024 •

edited

felicio commented Mar 17, 2024 •

edited

felicio commented Mar 19, 2024 •

edited

osmaczko commented Mar 19, 2024 •

edited