Prepopulate on fresh docker install

[cpol0]

Hello,
I've followed docker install instruction and I have an issue to prepopulate with python manage.py app:prepopulate. So this is on a very clean installation.

ERROR:eve:batch op errors occurred, full error: {'writeErrors': [{'index': 0, 'code': 11000, 'keyPattern': {'name': 1}, 'keyValue': {'name': 'Editor'}, 'errmsg': 'E11000 duplicate key error collection: superdesk.roles index: name_1 dup key: { name: "Editor" }', 'op': {'name': 'Editor', 'privileges': {'ingest': {'read': 1}, 'publisher_dashboard': 1, 'planning_assignments_view': 1, 'monitoring_view': 1, 'spike_read': 1, 'highlights_read': 1, 'use_global_saved_searches': 1, 'dashboard': 1}, '_created': datetime.datetime(2022, 1, 24, 16, 39, 53, tzinfo=<UTC>), '_updated': datetime.datetime(2022, 1, 24, 16, 39, 53, tzinfo=<UTC>), '_etag': 'd8c8aed92ea76780a56b1ce5fa8b27d6e5aa47bc', '_id': ObjectId('61eed6595ac0697e11ed7f8b')}}], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 0, 'nModified': 0, 'nRemoved': 0, 'upserted': []}
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/eve/io/mongo/mongo.py", line 462, in insert
    return coll.insert_many(doc_or_docs, ordered=True).inserted_ids
  File "/opt/venv/lib/python3.8/site-packages/pymongo/collection.py", line 761, in insert_many
    blk.execute(write_concern, session=session)
  File "/opt/venv/lib/python3.8/site-packages/pymongo/bulk.py", line 528, in execute
    return self.execute_command(generator, write_concern, session)
  File "/opt/venv/lib/python3.8/site-packages/pymongo/bulk.py", line 363, in execute_command
    _raise_bulk_write_error(full_result)
  File "/opt/venv/lib/python3.8/site-packages/pymongo/bulk.py", line 140, in _raise_bulk_write_error
    raise BulkWriteError(full_result)
pymongo.errors.BulkWriteError: batch op errors occurred, full error: {'writeErrors': [{'index': 0, 'code': 11000, 'keyPattern': {'name': 1}, 'keyValue': {'name': 'Editor'}, 'errmsg': 'E11000 duplicate key error collection: superdesk.roles index: name_1 dup key: { name: "Editor" }', 'op': {'name': 'Editor', 'privileges': {'ingest': {'read': 1}, 'publisher_dashboard': 1, 'planning_assignments_view': 1, 'monitoring_view': 1, 'spike_read': 1, 'highlights_read': 1, 'use_global_saved_searches': 1, 'dashboard': 1}, '_created': datetime.datetime(2022, 1, 24, 16, 39, 53, tzinfo=<UTC>), '_updated': datetime.datetime(2022, 1, 24, 16, 39, 53, tzinfo=<UTC>), '_etag': 'd8c8aed92ea76780a56b1ce5fa8b27d6e5aa47bc', '_id': ObjectId('61eed6595ac0697e11ed7f8b')}}], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 0, 'nModified': 0, 'nRemoved': 0, 'upserted': []}
INFO:superdesk:Command finished with: None

It seems there is an issue with the editor role (duplicate key). I've tried to edit/delete some users to only have 1 editor (by default there are 2) but change nothing.
I've tried to prepoulate before and after app:initialize_data command (by deleting docker volume to always have fresh data).

Am I missing some steps? Thanks
Pol

[petrjasek]

hi,

I see only 1 Editor role in the prepopulate data, so this could be the case if you run this script for second time or if you created such role manually.

This editing of users you did via UI? Was there any error in the server console?

[cpol0]

Hi,
by default after initialize_data I agree there is only 1 editor (checked in the UI)

I swear I didn't start the script 2 times, and this is why I cleanup all the docker volumes to be sure there are no remainders.
It seems that launch app:prepopulate try to add a 2nd editor before the exception:

However, even by changing the editor role to contributor (in order to not have editor role at all), there is still the same exception.

Regarding the server console, are you talking about "system message"? If yes, this console is empty.

[petrjasek]

the error you get is not due to having second user with editor role, but the script is trying to add second role with the same name Editor which is not allowed. but I wonder what is your issue with that, seems like you have the role in place already.

[cpol0]

First, do you see the issue on your side? Or do am I missing some step in the configuration?

Then, if it is confirmed that's an issue, (sorry to ask that but I'm not at all a pyhon developer): once in the docker container (superdesk_superdesk-server_1) , I can't find how can I change the app_prepopulate_data.json file? (seems that the content is not the same than https://github.com/superdesk/superdesk-core)
Basically, it seems that this file try to add a 2nd editor, which could be the issue because we already have 1 at the superdesk initialization. Maybe I can try to remove this one and see what's append.

[petrjasek]

yep you have an Editor already, so it won't let you add another one, that is all good imo. I understand that error you get in console is rather cryptic, but all it really says is that there is user role with name Editor already and you can't add another one.
is there anything wrong with the system? are you expecting additional entities to be populated?

[cpol0]

Yes, regarding the documentation https://superdesk.readthedocs.io/en/latest/cli.html#app-prepopulate it seems this command is very helpfull to have a kind of demo with fake datas. Because we (www.reporterre.net) are looking for a new CMS, it will be very convenient if I can try this way.
Furthermore, yes I think something is wrong with the system: I should not have an error with a clean docker image, it should works perfectly so probably there is an issue in the prepoulate script.
Also, even if I delete the editor I have still an issue as I mentionned 2 posts ago. It would be great to have a workaround :)

Thanks for your support,
Pol

[petrjasek]

do you use docker-compose to run it? it has another set of demo data, you shouldn't need to run the prepopulate command with that, we use that one rather for testing

[cpol0]

yes, I run docker-compose and followed instructions from the documentation. Appart the 3 defaults users, I didn't see demo datas. Is there another command to run insteaf of prepopulate?
Thanks a lot!

[petrjasek]

are you using master branch?

[cpol0]

Yes I am.
I did few more tests today, unfortunately all failed.

• I've tried to launch prepopulate with DEMO_DATA=0, this time there is still the same issue because in docker/start.sh an admin is created which leads to duplicate admin.
• Then, I've removed the admin creation in the script, commit a new docker image and retry: still the same issue but later in the preopulate script
• Finally, I've removed also the app:initalize comand (just in case of...), commit again and retry: still an error...

I give up to try to use this prepopulate command, it will probably takes less time to add manually some articles than try to fix it. It's a pitty because it could be very helpfull to have an overview and play with API with a lot of fakes data insie superdesk.

@petrjasek , I'm open if you have another idea, otherwise thank you for the support.
Pol