-
|
Hello all, Also, how can I use the external Postgres Server that we have in production? Ok, confusing, lets drill down this: 1 - We need to export / Migrate the data that we already have in the DD PROD environment, with some tests and findings and so on, to a Postgres external DB. 2 - We need to install the DD using the external Postgres DB. I have found already some info about this but its a bit confusing where all the info should be configured.
3 - Can we streamline the docker-compose file so we use only the configuration that we need (Postgres and so on), instead all teh confusion of the profiles?
Thanks in advance for your support. Jorge Gomes |
Beta Was this translation helpful? Give feedback.
Replies: 7 comments 13 replies
-
|
@quirinziessler , maybe you can help here |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jasgggit I actually had nearly the same situation as you a few months ago. I will try to answer your question. Regarding 1.: Regarding 2: Regarding 3: Hope this helps you a bit. |
Beta Was this translation helpful? Give feedback.
-
|
Hello and thanj you, I'll try your suggestions and I'll report back ASAP. |
Beta Was this translation helpful? Give feedback.
-
|
+1 on PostgreSQL & Redis We will shortly be deprecating MySQL and RabbitMQ since those appear to be the best combination of supported tech. |
Beta Was this translation helpful? Give feedback.
-
|
Hi and sorry for this ... How can I import the DB dumped with: docker exec -it django-defectdojo-uwsgi-1 python manage.py dumpdata > dumpfile.json ? This is 75MB file so is not huge ... but can't figure how, tried with: docker exec -it django-defectdojo-uwsgi-1 python manage.py loaddata --format=json -v 3 - dumpfile.json But it stalls saying: [19/Feb/2024 18:23:06] INFO [dojo.models:4505] enabling audit logging And nothing more ... Thanks in advance. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, the loaddata command is the right one. Why did you add "--format=json -v 3 -" to it? The "-" at the end defines you want to load the "-" Django fixture but you want to load all the data in the dumpfile. So simply "python manage.py loaddata dumpfile.json" should work for you. |
Beta Was this translation helpful? Give feedback.
-
|
@quirinziessler Hi, I did, the '-v3' should be verbose 3 ... this is what it outputs: /opt/containers/django-DefectDojo # docker exec -it django-defectdojo-uwsgi-1 python manage.py loaddata -v 3 dumpfile.json Can't figure out what the issue is .... |
Beta Was this translation helpful? Give feedback.
-
|
About this:
An example of that is available in the Community Contribs repo: If you navigate into the dojo directory in that part of Community Contribs, you can see a 'simplified' version of compose at: |
Beta Was this translation helpful? Give feedback.
-
|
@mtesauro Hi and thank you for this insight, although there are substantial changes on DD for this to work properly I managed to adapt a version to work on our DEV env. It keeps giving the error of the fixtures missing ...
Can you share some light on this ? Thanks in advance. |
Beta Was this translation helpful? Give feedback.
-
|
I haven't had consistent results using the Django fixtures. If you're going from one DB to another e.g. PostgreSQL to Postgres (just running in different places), then I've had luck with just using the 'normal' PostgreSQL tools. Something like: to create the backup. Then move that .sql file over to the new location and do something like: Just make sure and stop the nginx container or otherwise make sure users don't try to change data during the backup and restore operations. If you Google for pg_backup, there's loads of resources and different ways to use the 'native' PostgreSQL tools to backup and relocate DB data. HTH |
Beta Was this translation helpful? Give feedback.
-
|
@mtesauro the issue here is that I'm trying to export from a Mysql to import on a Postgres ... |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jasgggit I just tried to reproduce your case. However dumping and loading the file works fine for me. Could you please check if the dumpfile is loaded inside the container? Just a side note: The command I sent you saved the file in the current directory (not inside the docker container/volume). In order to have it inside the container, move it to one of the defectdojo volumes e.g. media. Another option would be to login to the uwsgi container and drop it directly to media. After configuring with Postgres login again and upload from media. |
Beta Was this translation helpful? Give feedback.
-
|
@quirinziessler Actually I was following the same principle that if the dump was written to the local FS it can also be loaded from it. So to answer the question no, I wasnt placing the file inside the container, although I kept remembering that it should be ... |
Beta Was this translation helpful? Give feedback.
-
|
So ...
Outputs: The above exception was the direct cause of the following exception: Traceback (most recent call last): Does it need to be empty? like a new DB with no initialization? |
Beta Was this translation helpful? Give feedback.
-
|
I just tested it and it worked even I had the exact values already present. Here are the steps I was sucessfully going trough. Maybe this helps you a bit.
If watson is the only foreign key issue comming up you may can try to run |
Beta Was this translation helpful? Give feedback.
-
|
@quirinziessler Thank you for your good steps but unfortunately, they seem only to work on the same instance. BTW, I flush the DB prior to the load, and this is to be a MySQL to PostGRES dump and load. f0ae910a19fe:/app$ python manage.py loaddata media/dumpdata.json The above exception was the direct cause of the following exception: Traceback (most recent call last): Thanks for the support. |
Beta Was this translation helpful? Give feedback.
-
|
@mtesauro @quirinziessler Hello, So, for the past 2 days, I've been trying to migrate my small tests DB from a setup made in 2022 using docker and MySQL-RabbitMQ. So far I wasn't able to import the JSON dump file, either to a Mysql or Postgres deployed DB. It keeps giving the above errors and nothing is imported. File error:"mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces". 2- After the exported DB, I redeployed the DEV setup, using the profiles, and a MySQL-Redis setup, and this time I took note of the 'root' pwd generated during the process, imported the MySQL dump file, and all my products and engagements with the tests and findings are there. So far so good. 3- Then I exported that DB, this time using the 'root' user, with no errors on the dump, and tried to migrate that to the version of DD with the Postgres DB, using the above-mentioned utility, with no errors on the migration process, and when I open the DD all the products and engagements and tests are there, with the exception o the contents of the findings in the tests!. This is the part I need the most because using this allows me to compare the findings with the new tests added after, and compare them, DD is useless for me without this, and unfortunately, I have no original tests to reimport because we lost them when we lost the Nessus server. Comparing both deployments using the original MySQL dump file, used to recreate to new MySQL DB on DEV:
Can you help me finish this? Any ideas on how I might see this through? Best regards. |
Beta Was this translation helpful? Give feedback.
-
|
Hello all, the problem has been solved by one of my colleagues, we know more about Postgres than I do, and he found the issue of the import of the DUMP JSON file was related to the 'Sequences', on the 'auditlog_logentry_id_seq'. This way all is imported. Thanks for the support. Regards. JG |
Beta Was this translation helpful? Give feedback.
-
|
For future readers of this discussion, there's some good info in OWASP's Slack instance in the #defectdojo channel. I'm summarizing greatly here but one suggestion I found interesting was to use mysqldump to get the schema/data from MySQL then use a tool like pgLoader to convert the mysqldump to Postgres. Anyway, hope that helps those that want/need to migrate. |
Beta Was this translation helpful? Give feedback.
-
|
@mtesauro, thank you |
Beta Was this translation helpful? Give feedback.
Hello all, the problem has been solved by one of my colleagues, we know more about Postgres than I do, and he found the issue of the import of the DUMP JSON file was related to the 'Sequences', on the 'auditlog_logentry_id_seq'.
It needs to be changed from the default '1' to something greater than the one on the JSON file.
Then we can use the commands to load the JSON after we flush it first.
This way all is imported.
Thanks for the support.
Regards.
JG