Memory usage during db migration

[eeveelution1313]

I'm in the middle of migrating from sqlite to mariadb, in hopes of speeding some things up, and i'm seeing extremely high memory utilisation. I'm not sure if it could be related to it not being done with scheduled tasks before trying to switch.

screenshot of stats before start of migration

i followed the guide from here https://github.com/meeb/tubesync/blob/main/docs/other-database-backends.md

Not sure if it matters but it's running using podman-compose with the following

  TubeSync:
    image: ghcr.io/meeb/tubesync:latest
    container_name: TubeSync
    ports:
      - 4848:4848
    volumes:
      - ./Managers/TubeSync:/config
      - /mnt/HDD/Media/Youtube/:/downloads
    environment:
      - TZ=Europe/Copenhagen
      - PUID=1000
      - PGID=1907200513
      - TUBESYNC_WORKERS=1
      - DATABASE_CONNECTION=mysql://user:pass@ip:3306/tubesync
    restart: unless-stopped

Very new to github, sorry if i didn't include something needed

[eeveelution1313]

additionally after hitting the max memory it can use it seems to process scheduling tasks in patches slowing down significantly

sorry if it comes off as me complaining, that is not my intention i'm just not sure if it could be valuable information

[meeb]

This is the Django dumpdata / loaddata process. This can use a lot of memory. Your usage does seem extreme however. How big is the JSON file you're trying to import into MySQL?

[eeveelution1313]

i just started using this, and pretty much just added all of my subscriptions in one go, as i'm getting tired of youtube's interface. The json is 1.1 gig, the sqlite is also roughly the same size