Let devices request keys to decrypt past history for a room.

[ara4n]

If I'm in a room as device A, and join it later as device B, i need a way to request the old room key data from device A.

Similarly if I'm invited into a room by someone and they want to share past history with me, they need a way of transferring key data.

[apexo]

Could a possible solution to this be:

• create a special (maybe hidden per default) per-user channel on the user's home server
• each new device sends an (encrypted) poll message to all other devices (via this room)
• all other devices reply with the encrypted e2e state (sans identity keypair), or just all keys required for decrypting room history

[ara4n]

@apexo this is sort of the plan, except we now have a dedicated "to device" store-and-forward API for sharing data like this, which is encrypted in turn by olm. Also, we'd probably request the historical ratchet state from a specific device rather than all of them, and require the target user to approve/deny the request. e.g. "Matthew has added a new device 'iPad' and is requesting room history since Sept 18th. Do you want to share history?" or something.

[0kto]

@ara4n So I am currently in a room listed with seven devices from three machines and as many browsers. well, I guess vector --> riot transition helped to increase the number. But with only one other person in the room, this is cumbersome.
I would prefer if a client-side certificate could be recognized as a user, and verification becomes an automated process that does not rely on devices or interaction of members in the room and I just see the history. Would work on private devices (i.e. not the library computer).
To understand the process, how are the IDs generated, and how long are they kept / stored? Do they change with version change of kernel, browser or if I purge cookies? I am pretty sure I am missing discussions on the same device I logged in at the time...

PS: I haven't seen another issue / discussion on this matter, so I hope this is the right place to put this :)

[madduck]

Maybe look at Bitcoin wallets, e.g. http://wallet.counterwallet.io, which ask a 12 dictionary word passphrase from the user. That passphrase is the private key.

Now when I log in to a device, I could be asked if I wanted to create a device-specific key, or if I wanted to instate the user's key from such a passphrase.

[csmart]

Can I check my understanding please, does the following represents current expected behaviour?

Say I'm in a room which has encryption turned on and we're all happily chatting away and everyone has verified keys. Then I sign out of riot and back in to see that all previous conversations are encrypted again and that there's no way to get the original key to you, so history remains unavailable?

Thanks!

[madduck]

It doesn't matter whether keys are verified. The point is that the encryption algorithm (megolm/ld double ratchet) encrypts to the current keys of all the people present.

Now if you log out of Riot, your key is discarded from local storage, and upon rejoining, a new key is made. Future messages will now be encrypted (also) to that key, but since you no longer have access to the previous key, the old messages aren't accessible any longer.

The solution is IMHO to allow one to export/safe/backup one's key and restore it upon rejoin.

[csmart]

Right, thanks @madduck

[ghost]

I've been having major usability problems with end to end encryption in riot, and I didn't realize that it was because logging out deletes your session key! riot-android/#869 would be nice, because I had no idea!

Now that I understand why I've been losing my keys -- and being locked out of my chat history -- I have to say I'm really annoyed. I'm sure everyone agrees you shouldn't have to stay logged in to your chat program continuously in order to be able to use it reliably. It's one thing to hide past messages from newcomers to a channel, but this same issue means that I can't share decryption keys with myself! In my opinion, E2E should not have been rolled out as a feature before this problem was solved. It's been a trainwreck for everyone I know who's tried turning it on.

[csmart]

@kebertx Aside from this issue (which I now understand) e2e has been awesome for me and those I've been chatting with.

IIRC, it was advertised as beta and at least in the Android app it says it's that e2e is an "experimental labs feature which may break in unexpected ways." So I think the warning has been fair enough.

Having said that, I do think things can be improved (a simple warning about losing keys when signing out, for example would be handy for new users).

IMO, it's been great to have the feature early and be able to test it.

[madduck]

Hey @kebertx, sorry you've had a bad experience. It is true that e2e at the moment is highly beta, and things certainly need to be improved, which is, in part, why this issue exists.

FWIW, I see no need to ever log out. I do have riot.im exempted from my cookie munger, and so I don't have this issue. Similarly, I don't log out the Android app, but the problem will come when I need to switch to a new phone, and by then I hope they've got a fix.

[ara4n]

(fwiw the warning was added)

[irunonayran]

I don't see a need to log out either, if the app supports multiple identities, which riot seems to not do, so logging in and out would be a workaround for me. Alternatively I could use a second android user, but that drains my battery more.

[pik]

@madduck You could also go the other-way an export an existing private key as a mnemonic - e.g. BIP39 uses a 2048 word list (https://github.com/bitcoin/bips/blob/master/bip-0039/english.txt) which is sufficient to store 128-bit keys (2048^12). Although one issue is that ed25519 and curve25519 keys are I believe independently derived in Olm? So you would need mnemonics for both, or support deriving one from the other.

@ara4n

Similarly if I'm invited into a room by someone and they want to share past history with me, they need a way of transferring key data.

I'm not sure I understand the original issue very well, can't an existing member just key-share all of the session_keys they have received from other members in that room via. the same sendToDevice api using an Olm Session as is normally done when a client shares their outbound session key?

[ara4n]

having read through prior comments - most of them don't relate to this bug, but are general UISI concerns (#2996) or key management concerns (#3611); please upvote and/or comment on the right bugs rather than using this one as a dumping ground O:-)

As per element-hq/element-web#2996 (comment) we're moving on and implementing this asap, both to provide an escape hatch for cleaning up remaining UISIs as well as to provide the feature of syncing history to new devices without having to manually export/re-import megolm session keys. @richvdh, @NegativeMjark and I just went through the discussion of how this should work, resulting in the plan:

• If a client receives a message which it doesn't have megolm session keys for (i.e. a UISI), it should send to_device messages to all devices (i.e. "*") belonging to themselves and the sender, requesting the missing key.
• By default senders will automatically re-share megolm keys to devices they have verified (if they sent the message to that device, or it's their own device).
• If the device isn't verified, they have the option to verify it before sharing, or to ignore the request (and/or blacklist the device?)
• We'll also provide a setting called "Do not automatically share E2E keys with new devices" in settings, so that paranoid users can explicitly authorize this.

Separately, we will obviously need to improve the verification UX, e.g. by cross-signing devices to automatically trust them, QR codes or mnemonics for verifying fingerprints, etc. This is all element-hq/element-web#2142 however and orthogonal to the problem of actually sharing keys.

We also discussed whether we should store megolm keys on the server, to try to both improve clientside key storage capacity problems (#3660) or to provide an easier way of backing up keys and/or sharing them - this is element-hq/element-web#3661. We concluded that this is basically an optimisation for key storage and wouldn't change the semantics of the crypto at all (i.e. if you're running out of clientside storage, you're welcome to use an encrypted cloud storage service to store data, but that's not going to change our E2E behaviour).

Finally, we considered the edge case that Alice and Bob are both pretty paranoid, and only ever use Matrix by firing up an incognito window, logging in, sending a message, and closing the tab (not implausible). As a result, at any given point they probably wouldn't have any devices live. Therefore when Alice sends a message to Bob, she won't know how to encrypt for him, and when Bob gets a UISI, there won't be any devices of Alice he can query for the megolm keys (unless Alice happens to be logged in and having re-imported her megolm keys at the moment that Bob requests history, or perhaps later). We concluded that there were two possible solutions to this:

1. Introduce the concept of 'virtual devices' so that Alice & Bob can maintain a virtual device which is stored encrypted on the server, and is passed between physical logins, letting them pick up conversations when desired - this would almost act as an 'identity key'.
2. Introduce the concept of deliberately 'dehydrating' and 'rehydrating' devices, so that when Bob stops using his Incognito window, he could explicitly export that device intact onto a USB stick or whatever, and subsequently rehydrate it next time he opens an incognito tab - effectively deliberately backing up & unbacking up the device.

In the end, both of these are tantamount to the same idea, whether the device is stored encrypted on the server or on a USB stick. Both suffer from element-hq/element-web#3822 - that if two identical devices ever exist concurrently Olm will completely wedge and E2E will break. So the conclusion there was to educate users that they need to keep at least one device active if they expect E2E to work, unless we ever get as far as dehydrating/rehydrating devices.

@pik: hopefully this resolves your confusion a bit :)

[uhoreg]

Will there also be a button somewhere to request all the keys from one of your own devices, in order to aid migration from one device to another? Without this, it sounds like when people want to replace one device with another, people would either need to manually export/import the keys (a bit less convenient than just clicking a button to request keys), or else go to each encrypted room and backfill to the beginning so that the device automatically requests keys (a lot less convenient).

[ara4n]

@uhoreg possibly. i'm a bit scared at how powerful such a button would be; at least an attacker needs to know the IDs of the megolm sessions they're trying to get keys for in the "send_missing_keys" to_device event outlined above. Whereas an "send_all_keys_ever" feels like a terrible thing to fall into the wrong hands...

[dkasak]

This very ornate and fragile solution is our exact use case :)

Just to add, so that it doesn't leave the wrong impression, I think this is a problem that should be solved. I just don't think it's a good idea to double down by solving a special case in a flaky way, which will only serve to increase the perception that the feature is broken and with no way to extend it to be a general solution.

So we need to think of a better, general solution. For your use case, uhoreg's bot idea could work extremely well.

[HarHarLinks]

I have hit either this issue or element-hq/element-web#16983, I find it hard to believe that none of the other users in the room is using an element web that would share their keys, see my comments there.
I have hit it again(?) just now due to a room upgrade. The room list message preview says A: ** Unable to decrypt: The sender... while the room's timeline itself is blank apart from my join event.

[t3chguy]

I find it hard to believe that none of the other users in the room is using an element web that would share their keys, see my comments there.

Only the client which invited you matters. If someone invited you from their Element Android even though they have an Element Web online at the same time, that won't do anything.
It also matters how many of the megolm sessions in the room were created by Element Web (how much do the users of that room use Element Web to send the messages rather than read them) - only sessions created by Web & Desktop correctly flag the megolm sessions as sharable at this time.

[richvdh]

Didn't this get fixed?

[pmaier1]

Can we close this in favor of #749? To my knowledge element-hq/element-web#749 will be the way to go here and we want to schedule it soon(tm).

[richvdh]

element-hq/element-web#749 is an epic describing a desired end goal rather than a particular solution, whereas this was more an idea for a particular implementation.

Nevertheless I still think this has been done and should be closed; if anyone disagrees, we can reopen with more context.

[karolyi]

Pardon my intrusion but in which version did this get solved? I'm using 1.71 and it still doesn't work there.

[richvdh]

Pardon my intrusion but in which version did this get solved? I'm using 1.71 and it still doesn't work there.

Version 1.71 of what? Note that this is the element bug tracker.

The fix landed in Element Web in matrix-org/matrix-js-sdk#454. You're likely seeing a different problem; feel free to open a new issue and send bug reports from both the sender and recipient applications (see also #245)

[karolyi]

Oh sorry, the ecosystem around matrix is a bit convoluted. I'm using synapse 1.71 and element-web 1.1.17 and we're still seeing this problem.

[HarHarLinks]

Based on the problem statement in OP (without reading the whole thread) and the explanation in #647 (comment), I don't think this is "solved", merely worked around in one specific setting. To me the wording of both implies that element-hq/element-web#749 duplicates this issue (but is phrased more clearly).

[uhoreg]

No, this issue is for getting keys for messages sent while you were not in the room. For messages sent while you were in the room, and that other devices of yours can decrypt, it should be handled by key backup.