How to delete duplicate entries?

[Serif-7]

Somehow, several entries in my jrnl have been duplicated, so now there are three of each entry, all write next to each other with the same timestamp.

Is there a simple way to select and delete duplicate entries via --delete or some other method like sed?

[Serif-7]

awk '!seen[$0]++' file.txt > file_new.txt works well I've found, though it deletes the blank lines between entries. For fixing that that I've found
sed "s#\[#\n[#g" file.txt > file_spaced.txt works.

[wren]

Although we don't have an explicit command to find dupes (maybe we should?), you can use jrnl's import command for this.

1. Move your journal file (let's say jrnl.txt) to something else (like jrnl-backup.txt or anything you want)
2. Import the backup into the (now empty) journal: cat jrnl-backup.txt | jrnl --import
3. jrnl will recreate jrnl.txt and dedupe the entries as they're being imported

Afterward, you can diff jrnl.txt and jrnl-backup.txt to be sure you got what you expected.

[griffrawk]

I presume the input file has to be unencrypted for this to work? I will probably need to dedupe after I've finalised a DayOne JSON import one-liner I'm working on.

[wren]

@griffrawk If you're importing from a file, then yes, it'll have to be unencrypted to work. If you have an encrypted journal that you want to do this with, the newer --config-override flag to help.

Instead of cat jrnl-backup.txt | jrnl --import, you can do something like jrnl --config-override journals.default.journal jrnl-backup.txt --format text | jrnl --import. By using jrnl instead of cat you'll be able to read the encrypted journal.

[griffrawk]

Ah, thanks.. I hadn't caught up to the overrides in the docs yet.

[Serif-7]

@wren alright, I'm running into weird behavior now. I... don't know when this happened, but I checked my journal and some entries were duplicated, even after I explicitly removed them using the commands I listed. Then I tried your methods, and it did remove some dupes, but the ones that were duplicated still had a single duplicated entry alongside them.

This one might require some investigation on my part.