Skip to content

arminfelder/gridfsmigrate

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

46 Commits
 
 
 
 
 
 
 
 

Repository files navigation

RocketChat GridFS to FileSystem/AmazonS3 Migration

This is a script for migrating files uploaded to RocketChat from the default GridFS upload store to FileSystem/AmazonS3.

migrate -c [command] -d [s3 bucket|output directory] -r [dbname] -t [target]

Help

Run ./migrate.py -h to see all available options

Commands

  • dump : dumps the GridFs stored files into the given folder/s3 bucket and writes a log files
  • updatedb : changes the database entries to point to the new store instead of GridFS
  • removeblobs : removes migrated files from GridFS

Requirements

Dependencies

  • python3 (e.g. apt install python3 python3-pip)
  • packages (pip3 install ...):
    • pymongo
    • boto3

Environment

Steps

  1. Backup your MongoDB database so that you won't loose any data in case of any issues. (MongoDB Backup Methods)

  2. Change Storage Type in RocketChat under Administration> File Upload to FileSystem or AmazonS3. Update the relevant configuration under the corresponding head in configuration page.

  3. Start copying files to the new store

    • File System

         ./migrate.py -c dump -r rocketchat -t FileSystem -d ./uploads
      
    • S3

         ./migrate.py -c dump -r rocketchat -t AmazonS3 -d S3bucket_name
      
  4. Update the database to use new store (use -t AmazonS3 if you are migrating to S3)

     ./migrate.py -c updatedb -d /app/uploads -r rocketchat -t FileSystem
    
  5. Check if everything is working correctly. Ensure that there are no files missing.

  6. Remove obsolete data from GridFS

     ./migrate.py -c removeblobs -d /app/uploads -r rocketchat
    

Troubleshooting

On some configurations, it might help to add the parameters "directconnection=True and connect=False" to the MongoClient constructor, such as:

    MongoClient(..., retryWrites=False, directconnection=True, connect=False)[self.db]

So that the connection happens in Single topology.