Info about snap update

[xator91]

Hello @xet7,

I came across information that you're working on an update for Snap. Currently, I have a fresh installation of Wekan on my Ubuntu 22.04 LTS. I'm developing an HTML integration app with Wekan to enhance the production line at my workplace. This leads me to inquire whether you're close to releasing a Snap update, or if I should consider an alternative installation method to get the latest version immediately. If so, what method do you recommend for keeping up with updates? Perhaps Docker, or something else?

[xet7]

@xator91

Before releasing newest WeKan to Snap Stable, there needs to be these additions:

1. Make working migrating from MongoDB 3.x to MongoDB 6.x or FerretDB here https://github.com/wekan/wekan/blob/main/snap-src/bin/mongodb-control#L21-L77
2. Look that attachments are migrated correctly, with a) updating database structure or b) export files from MongoDB 3.x to filesystem like this https://github.com/wekan/minio-metadata and look is it possible to use attachments directly from filesystem with Meteor-Files . Also, fixing attachments port number Attachment cannot be handled correctly, because port number for attachment is missing #5281
3. Add checking, is there enough free disk space for migration in Snap

I'm not yet sure how close it is to update Snap. All help is very welcome.

Other way would be for you to just install newest WeKan with Snap Candidate, like this: https://github.com/wekan/wekan/wiki/OpenSuse

[xet7]

@xator91

To get translations visible, I use these settings: https://github.com/wekan/wekan/wiki/Caddy-Webserver-Config#caddy-2

[xet7]

Sure it's possible to use Apache, Nginx etc, there are examples at right menu of wiki.

[e-gaulue]

@xet7 All the elements describe above are not of the same level.

Point 1 looks not so bad for Mongo (to me). For FerretDB, I don't know.
Point 2: I'll try to have a look. Does it means that #4891 and internal migration from CollectionFS to ostrio-files/Meteor-Files is stuck? As of today, it looks using the filesystem way is preferred. Where to look at to give it a try on both direction? Attachment port number looks like a smaller trouble. I suppose, it could be handled by a documented DB query. Moreover, you don't change your port number every week.
Point 3: As a sysadmin I would say space is not a trouble, you've nowadays plenty of way to expand your FS. And specifically for short time, if it's just a question of migration. If you think you may lack of space make mongo => USB SSD => mongo. If USB SSD is not enough, use a NAS. If NAS id not enough, rent space. It's more a sysadmin trouble than a wekan one. All applications are limited by their allocated resources.

[xet7]

@e-gaulue

Newest WeKan is using Meteor-Files.

At database, there are text fields (like filename, etc). That database structure has changed from CollectionFS to Meteor-Files.

Then, at database, there are chunks, where actual files are. One file can be in one or more chunks. So it is possible to install WeKan Snap Stable and Snap Candidate, and compare database structure, after adding one attachment.

For migrating database structure, there is this incomplete migration code, currently commented out:

https://github.com/wekan/wekan/blob/main/server/migrations.js#L1248-L1408

Alternatively, with mongofiles command (for MongoDB 3.x and 6.x), it is possible to export files etc, like at https://github.com/wekan/minio-metadata

There is info how to build WeKan at https://github.com/wekan/wekan/wiki/Emoji

[e-gaulue]

@xet7
So I've been the third proposed way: make cfs-grid suitable with Mongo 6.

1. meteor npm remove/install mongodb, that comes with bson...
2. All var1 ??= var2; have been turned into var1 = var1 ?? var2; to be OK with node 14
3. meteor npm install abort-controller and add import AbortController from "abort-controller" where necessary due to node 14

I have no more compilation error on screen, but just nothing happening, no 2000 port gets open. I just have:

i18next: languageChanged en
i18next: initialized {
  debug: 'true',
  initImmediate: true,
  ns: [ 'translation' ],
  defaultNS: 'translation',
  fallbackLng: [ 'en' ],
  fallbackNS: false,
  supportedLngs: [
    'af',    'af_ZA', 'en_ZA',   'ar-DZ', 'ar-EG',  'ar',
    'ary',   'az-AZ', 'az-LA',   'az',    'bg',     'br',
    'ca',    'ca-ES', 'cnm',     'cs',    'cs-CZ',  'cy-GB',
    'cy',    'da',    'de-AT',   'de-CH', 'de-DE',  'de',
    'el-GR', 'el',    'en-BR',   'en-DE', 'en-GB',  'en-IT',
    'en-MY', 'en-YS', 'en',      'eo',    'ast-ES', 'es-AR',
    'es-CL', 'es-CO', 'es-LA',   'es-MX', 'es-PE',  'es-PY',
    'es',    'et-EE', 'eu',      'fa-IR', 'fa',     'fi',
    'fr-BE', 'fr-CA', 'fr-CH',   'fr',    'fy-NL',  'fy',
    'gl-ES', 'gl',    'gu-IN',   'he-IL', 'he',     'hi-IN',
    'hi',    'hr',    'hu',      'hy',    'id',     'ig',
    'it',    'ja',    'ja-Hira', 'ja-JP', 'ka',     'km',
    'ko-KR', 'ko',    'lt',      'lv',    'mk',     'mn',
    'ms',    'ms-MY', 'nb',      'nl-NL', 'nl',     'oc',
    'or-IN', 'pa',    'pl-PL',   'pl',    'pt-BR',  'pt',
    'pt-PT', 'ro',    'ro-RO',   'ru-UA', 'ru',     'sk',
    'sl',    'sr',    'sv',      'sw',
    ... 37 more items
  ],
  nonExplicitSupportedLngs: false,
  load: 'all',
  preload: false,
  simplifyPluralSuffix: true,
  keySeparator: '.',
  nsSeparator: ':',
  pluralSeparator: '_',
  contextSeparator: '_',
  partialBundledLanguages: false,
  saveMissing: false,
  updateMissing: false,
  saveMissingTo: 'fallback',
  saveMissingPlurals: true,
  missingKeyHandler: false,
  missingInterpolationHandler: false,
  postProcess: [ 'sprintf' ],
  postProcessPassResolved: false,
  returnNull: true,
  returnEmptyString: true,
  returnObjects: false,
  joinArrays: false,
  returnedObjectHandler: false,
  parseMissingKeyHandler: false,
  appendNamespaceToMissingKey: false,
  appendNamespaceToCIMode: false,
  overloadTranslationOptionHandler: [Function: handle],
  interpolation: {
    escapeValue: false,
    format: [Function: bound format],
    prefix: '__',
    suffix: '__',
    formatSeparator: ',',
    unescapePrefix: '-',
    nestingPrefix: '$t(',
    nestingSuffix: ')',
    nestingOptionsSeparator: ',',
    maxReplaces: 1000,
    skipOnVariables: true
  },
  cleanCode: true,
  defaultNs: 'translation',
  resources: {},
  ignoreJSONStructure: true,
  lng: 'en'

I suppose we get in a forever loop somewhere, maybe due to ??= rewriting. If you have any idea to find where, tell.

Looking at Minio, at first sight I'm not really convinced. As sys admin I observe:

• require XFS, possible on ext4, btrfs... but without warranty. By the way, no NFS.
• not able to move file location once done
• it looks oversized for the need: why not just keeping metadata in DB and files through MeteorFS?

[xet7]

@e-gaulue

One of my customers uses minio at internal network. They can not use AWS S3. There could be some other alternative file servers. The point is to use Node.js drivers to access files at minio, and not have files directly accessible via URL.

I presume metadata about files will be at DB anyway, because minio has limited amount of metadata fields, and for other storage there is no metadata features I think.

[xet7]

@e-gaulue

That customer would like to have some metadata at minio for searching, but only limited amount of metadata does fit to metadata fields. I did notice it, when getting errors about too much metadata. With metadata, I mean details like filename, size, card where attachment is, etc, I was trying to store too much data.

[e-gaulue]

@xet7
For reproductibility, here is what I did:

cd ~/wekan-git
git fetch origin
git rebase -Xtheirs -i 3b936ff6e7ed733a65488f1384f868e17b8ab751~1  # and remove first line
./rebuild-wekan.sh # and option 2
cd ~/wekan-git/.build/bundle/programs/server/npm/node_modules/meteor/cfs_gridfs
meteor npm remove mongodb
meteor npm install mongodb
grep -Rl '??=' ~/wekan-git/.build/bundle/programs/server/npm/node_modules/meteor/cfs_gridfs | xargs perl -pi -e "s/([\w.]+) \?\?=/\1 = \1 ??/g"
meteor npm install abort-controller
sed -i "15iimport { AbortController } from 'abort-controller';" ~/wekan-git/.build/bundle/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/utils.js
cd ~/wekan-git
./strart-wekan

I did more tests and something loop without trigging anything. Is this in babel? I look to find a good place to put a breakpoint. I now I passed i18next, that is all.

[e-gaulue]

@xet7

As the trouble was linked with cfs:gridfs support of Mongo 6. Tried downgrade one by one.
meteor npm install mongodb@5 : give same result out of the box. No need to change anything, but ./start-wekan is stuck.
meteor npm install mongodb@4 : works, but throwing really fast a TypeError: grid.mongo.GridStore is not a constructor.

So, the more I look at it and the more I'm convinced it's going to get really hard to move that way.

I'm thinking about something with 2 mongo connections. One with Mongo 6 maybe the one of the snap and the other one with the old Mongo database each one on different port. And then a script that would look like the one you worked on in migration.js.

I'm a bit afraid it's above my skills. I should manage to set two mongos with different versions on the same machine. But, externalizing the code in migration.js looks hard to me. I have never work with meteor (and really few with mongo by the way).

[e-gaulue]

There are so much place where this migration is discussed that I've got the feeling I've lost my time. I think I understand better what @xet7 tried to say and #5226 also helped me.

I was focused on reusing what was written in migration.js, but we have to forget about it. There is no way to use cfs:gridfs with something else than the current DB present in 6.09. That is Mongo 3.

The good news is that cfs:gridfs rely on GridFS that is part of Mongo and is not linked to Meteor. So the files in the Mongo 3 database are still there in Mongo 6 (for the moment I just tested on 4.4). The tool mongofiles proves it. So yes, it looks more like a question of renaming tables and recreating some metadata.

I've consume almost all the time I could spend on Wekan last week and I'll have just a few this week, after what I'll have to work on something else. But, if anyone gets here, is interested in moving from Wekan 6.09 and knows well Mongo, the trick is really to:

1. Restore a backup from 6.09 to a newer version using a mongorestore corresponding the the mongo version you want to test.
2. Create avatars and upload images to this new version from Wekan UI. It will be rendered where old files will be shown with interrogation marks).
3. Analyze the differences in Mongo between old CollectionFS metadata and Meteor-Files ones with a tool like NoSQLBooster or anything else you prefer.
4. Propose Mongo requests to move from one situation to the other.

You needn't to know Meteor, just Mongo.

Certainly after, there will be exceptions to focus on, but that's the first way to go.

I think it's what @xet7 tried to explain last week (and in different places), and I never managed to get it.

[xet7]

@e-gaulue

No problem. I'll try to fix it anyway.

[e-gaulue]

@xet7
Don't worry the week isn't finished yet. I'm going to try a script tonight. It's just I code with an IA, because I have no MongoDb skills. In pure SQL, it would already be done. I know what I want, it's more a question of traduction, IA is good for that...

[xator91]

@xet7

Don't worry the week isn't finished yet. I'm going to try a script tonight. It's just I code with an IA, because I have no MongoDb skills. In pure SQL, it would already be done. I know what I want, it's more a question of traduction, IA is good for that...

Which IA are you trying?

[e-gaulue]

@xator91
Codegeex for the moment. Said to be open-source...

[e-gaulue]

For attachments this nodejs script works for me on snap RC after mongorestore

const { exit } = require('process');
const MongoClient = require('mongodb').MongoClient;
const ObjectId = require('mongodb').ObjectId;

async function main() {
    const client = new MongoClient('mongodb://localhost:27019');
    const db = client.db('wekan');

    //Delete mongodb 'attachments' collections in case it's present (need this for rename)
    const allcollection = await db.listCollections().toArray();
    if (allcollection.find(c => c.name === 'attachments')) {
        await db.collection("attachments").drop();
    }
    if (allcollection.find(c => c.name === 'attachments.files')) {
        await db.collection("attachments.files").drop();
    }
    if (allcollection.find(c => c.name === 'attachments.chunks')) {
        await db.collection("attachments.chunks").drop();
    }        

    //Rename old cfs attchment collections to the attended ones
    await db.collection("cfs.attachments.filerecord").rename("attachments");
    await db.collection("cfs_gridfs.attachments.files").rename("attachments.files");
    await db.collection("cfs_gridfs.attachments.chunks").rename("attachments.chunks");

    //For each file (file metadata)
    for await (const file of db.collection("attachments").find()) {
        console.log(file._id + ' : ' + file.cardId);
        if (file.copies && file.copies.attachments && file.copies.attachments.key) {
            //For each card in collection cards where string _id is the same than string file.cardId (should be just one) 
            for await (const card of db.collection('cards').find({"_id": file.cardId})) {
                let metadata = {};
                metadata.boardId = card.boardId;
                metadata.swimlaneId = card.swimlaneId;
                metadata.listId = card.listId;
                metadata.cardId = card._id;
                metadata.versionName = 'original';
                metadata.fileId = file._id;

                //Search an element in collection cfs_gridfs.attachments.files where _id=file.copies.attachments.key and update it its metadata field
                console.log("Change file metadata of " + file.copies.attachments.key);
                await db.collection("attachments.files").updateOne({"_id": new ObjectId(file.copies.attachments.key)}, {$set: {"metadata": metadata}});

                let meta = {};
                meta.boardId = card.boardId;
                meta.swimlaneId = card.swimlaneId;
                meta.listId = card.listId;
                meta.cardId = card._id;
                //Add in var ext the extension of the filename file.original.name
                let ext = file.original.name.split('.').pop();

                //Add new fields to file (metadata) document
                console.log("Change filerecord metadata of " + file._id);
                await db.collection("attachments").updateOne({_id: file._id}, {$set: {
                    "size": file.original.size,
                    "type": file.original.type,
                    "name": file.original.name,
                    "meta": meta,
                    "ext": ext,
                    "extension": ext,
                    "extensionWithDot": "." + ext,
                    "mime": file.original.type,
                    'mime-type': file.original.type,
                    "path": '/var/snap/wekan/common/files/attachments/' + file._id + '.' + ext,
                    "versions": {
                        "original" : {
                            "path" : '/var/snap/wekan/common/files/attachments/' + file._id + '-original-' + file.original.name,
                            "size" : file.original.size,
                            "type" : file.original.type,
                            "extension" : ext,
                            "storage" : "gridfs",
                            "meta" : {
                                "gridFsFileId" : file.copies.attachments.key
                            }
                        }
                    },
                    "_downloadRoute" : "/cdn/storage",
                    "_collectionName" : "attachments",
                    "isVideo" : file.original.type.startsWith('video/'),
                    "isAudio" : file.original.type.startsWith('audio/'),
                    "isImage" : file.original.type.startsWith('image/'),
                    "isText" : file.original.type.startsWith('text/'),
                    "isJSON" : file.original.type.startsWith('json/'),
                    "isPDF" : file.original.type.startsWith('pdf/'),
                    "_storagePath" : "/var/snap/wekan/common/files/attachments",
                    "public" : false
                }});

                //Remove all obsolete fields
                console.log("Remove filerecord old field of " + file._id);
                await db.collection("attachments").updateOne({_id: file._id}, {$unset: {
                    "original": "",
                    "boardId": "",
                    "cardId": "",
                    "uploadedAt": "",
                    "copies": "",
                    "synchronizedAt": ""
                }});
            }
        } else {
            //delete this file (metadata) from collection
            console.log("Remove filerecord with no corresponding file of " + file._id);
            await db.collection("attachments").deleteOne({ _id: file._id }, (err, result) => {
                if (err) throw err;
                console.log(`Deleted document with _id: ${file._id}`);
            });
        } 
    }
    await db.collection("attachments").createIndex(
        {
            "meta.cardId": 1
        },
        {
            name: "meta.cardId_1"
        }
    );
    exit(0);
}

main();

For avatars it shouldn't be so different but we won't work on it as we have just 8 to change

Other things that could be done:

• clean old collections if not needed anymore: _tempstore? _pre? cfs*?
• clean all attachments (files/chunks) that do not have a metadata counter part anymore
• clean md5 field in the attachments (file) collection

[e-gaulue]

Any new about your plan regarding upgrade?

Looking at #5226 and https://github.com/wekan/wekan/blob/main/snap-src/bin/mongodb-control#L21-L66, I may understand:

• you want multiple versions of mongo in snap. Is this just to deal with mongobackup/mongorestore? Or to upgrade mongodb and the wekan db inplace with successive version upgrade to avoid mongobackup/mongorestore and the kB,mB,gB or tB it would need on disk (about the same than the running database that can be huge for some)?
• you also look for a way to use SQLlite? Would it be for long term support wekan? Should it be done at the same time?

It looks there is a lot of snap instances worldwide. Do you think a migration script could handle all use cases? I've got the feeling, users will have to backup their instance anyway. As an admin sys, I wonder if I wouldn't prefer a script, than an automation. The typical risk is, if wekan is big, you can't check everything everywhere, so you believe everything is all right. But, after few months, if ever you discover it is not, users are using the new version and have created datas. So it may get really hard to reconciliate. It happened to me sometimes. I'm working with Debian for more than 20 years now and migration are never "simple" (even when praised it is).

I worry about a full automation. Moving from mongoDB 3 to mongoDB 6 + Meteor version change + CollectionFS to MeteorFiles + one year of wekan improvements is a big move. I have no crystal ball, but I suppose, some may fail the snap/stable migration even with a good automation. Couldn't we release a kind of recipes like in #4780. So, we would have 2 wekan/stable, like wekan/stable and wekan/new-stable or wekan/old-stable and wekan/stable (or simply wekan/stable and wekan/candidate).

Is this even possible through a simple snap/stable upgrade, to send a message/email to the administrator (if ever he has filled it) about what is going to happen?

I have also this feeling, after migration, somes may be impacted by changes that happened since their stable version, raising a lot of question/issues (in my case default on board cover images has changed). This also praise for little by little move. All instances who have an admin sys or anyone with enough skills to try #4780, should try to migrate and report whatever they observe wrong. The process could be permanently optimised in a wiki page.

This is a good place for discussion. But we are only three, right now. Anything we could do to test/help/document?

[e-gaulue]

@xet7
Do you know if GitHub wiki is changeable through PR? It may be easier? I know some projects like readthedoc use GitHub repos this way, but it's not the wiki part of GitHub.

[e-gaulue]

I mean, you clone, you change, you commit, you PR...

[xet7]

@e-gaulue @C0rn3j

With GitHub wiki, it is possible to clone, but not PR, it is limited.

But, to make PRs possible, a moment ago I copied wikis to wekan repo, so now you can create PRs:

https://github.com/wekan/wekan/tree/main/docs

[xet7]

@e-gaulue @C0rn3j

With GitHub PRs, it's possible to fork/edit/PR at GitHub webpages. Without using CLI commands.

[xet7]

@e-gaulue @C0rn3j

I updated info about contributing to wiki:

https://github.com/wekan/wekan/blob/main/CONTRIBUTING.md#contributing-to-documentation-wiki