Check use of updated IATI codelists

[emmajclegg]

For awareness - some IATI codelists have recently been updated, as explained here: Notice: Nonembedded codelist updates

Can we check the most up-to-date lists are being used in Publisher? (and that we have an easy way to update them when needed!)

[emmajclegg]

Sharing this for awareness @praweshsth - let me know if you want to keep it as a GH issue or check it elsewhere as part of general maintenance of IATI Publisher

[Sanilblank]

@emmajclegg The IATI codelists used in the IATI Publisher are all maintained as static files within the system which have not been updated in quite some time and so, the most up-to-date lists are not being used.
Currently, there is no way to be able to update them in an easy manner. We will have to copy the contents of the updated json files present in the registry and place them in the static files present in the IATI Publisher system and then push the changes. To make this process easier, we could write a command that is responsible for checking if the contents of any required codelist file has been updated and then change the contents of the static files if required. If we will need to run the command everyday as an automated process, other extra work will be required which will take more time.

CONCERNS

• When a certain code gets deprecated/withdrawn by IATI, how should IATI Publisher handle this?
  Currently, IATI Publisher is not keeping a list of any of the withdrawn codes and hence, users do not see such codes in any lists and cannot select them. However, if some codes which are currently not withdrawn are used by the user, and then they are withdrawn in the future, Publisher currently has no way to deal with them.
  For example: Suppose a user is using the code '10000' as the CRS channel code for their data. Since the code is not yet withdrawn from IATI, the code can be selected by the user for using in their activity. However, if the code is withdrawn in the future, we are not sure about how IATI Publisher should react to this.
  If such codes are removed from the codelists files maintained within the system, they will not be shown in any select options but since the codes may already be used by users, it may result in some issues. Maybe, we should develop a way to keep the withdrawn codes in the system, but show the users information that the code is withdrawn and should not be used anymore.

cc. @praweshsth @PG-Momik @sarinasindurakar

[robredpath]

@Sanilblank We expect that codelists within the IATI Standard will update fairly regularly (every few weeks to months) so I think it's worthwhile automating the import process. The formats and locations of the codelists won't change without notice, so you can rely on those.

We're still figuring out the best way to communicate about code list updates without excessive notifications, so either a regular process (e.g. once a month) or a daily check for changes that triggers a manual process to update the codelists would be appropriate.

I don't know how often codes get deprecated or removed, or what the process is for those, so I'll ask one of my colleagues to reply on that question.

[emmajclegg]

@Sanilblank - to answer your concerns question, if a code gets deprecated/withdrawn by IATI, it should no longer be possible to select this from drop-down menus within IATI Publisher (i.e. it should not be possible to use the withdrawn code when adding new or updating existing data)

The code remains valid however in historic (i.e. previously published) data, so no changes should be made to existing data in IATI Publisher when a codelist is updated.

I wouldn't recommend keeping withdrawn codes in the drop-down menus (even with an appropriate label) as some users are likely to ignore this and use them anyway, with an impact on data quality.

If you can see issues to existing users with this approach, please elaborate and we can try to help.

[Sanilblank]

@robredpath

We expect that codelists within the IATI Standard will update fairly regularly (every few weeks to months) so I think it's worthwhile automating the import process. The formats and locations of the codelists won't change without notice, so you can rely on those.

Currently in IATI Publisher, we are facing an issue where the CRON job (the one responsible for running tasks periodically automatically) is unable to update any files present within the public directory (the place where all our json files are currently placed). We had also faced this issue for the Organisation Registration Agency data retrieval command and had worked around the problem by using AWS to host the files. IATI Publisher would read the data from the file (in AWS) the first time and place it in cache for use every other time. We could employ a similar mechanism for all other codelists as well.
We could also try to solve the issue so that the CRON job would be able to update the files present in the public directory but quite a lot of time has been invested in this without much output to be seen. So, the earlier approach may be preferred.
In both cases, we would need to write a command that is responsible for maintaining the codelist data in AWS and create cache for the first time when the data present in AWS has been updated.

@emmajclegg

to answer your concerns question, if a code gets deprecated/withdrawn by IATI, it should no longer be possible to select this from drop-down menus within IATI Publisher (i.e. it should not be possible to use the withdrawn code when adding new or updating existing data)

To implement this, with the current codebase, we will need to remove the code and its data entirely from the json data file which will result in some issues in the activity detail page and the edit page.
If a user is using a code which has been deprecated and removed from IATI Publisher itself, the detail page will not be able to show the proper information relating to the code. Also, the select field when opening the form will be seen as empty even if the data is actually present in the database.
A similar case has been explained by @PG-Momik in this issue #1409.

We will need to update the codebase so that we can inform the system that certain codes have been deprecated and so they should not be shown on the drop down options. In this update, the deprecated data will not be shown in the drop down options but will still be present in the json data file. This will solve the issue for the detail page and so correct data will be seen even if the code has been deprecated. When user tries to edit the data, the edit option will be seen as empty and if the user saves the form without making any changes, the deprecated code will be replaced by empty value.
Please confirm whether this approach would be desirable.

cc. @praweshsth @PG-Momik @sarinasindurakar @A4family

[emmajclegg]

@Sanilblank - can you provide more detail on what the CRON job issue is? I can pass it to one of the developers in our team for advice.

I'll discuss your comments on the interface and drop-downs with the wider Support team this week and get back to you to confirm.

[Sanilblank]

@emmajclegg - I just had a talk with our devops team member and this is what he has said:

Our server is being served by a user "www-data". Everyday a log file is created but, by user "root". According to Linux ACLs (simply, user group permissions policy) the user www-data cannot access files or directories by "root" user. To solve this we implemented "cronjobs" to change the permissions. But the working is not going as expected. We are currently looking into solving this.

In a similar manner, the CRON job is unable to modify any files present in our code's directory.

[emmajclegg]

Thanks for the extra info @Sanilblank. Having checked with our developers, we think your AWS workaround is likely the better way of the two to update these codelists going forward (i.e. the same solution as used for the Organisation Registration Agency). The alternative may have been storing the codelists in database tables and drawing from that, though we're not sure what the work required for that set-up would be at this point.

I am still discussing with IATI Support colleagues what the expected behaviour should be on data already existing in IATI Publisher when codelists change (there's a few edge cases to think through as you've described above). I hope to share these expected behaviours for the interface by the end of this week

[emmajclegg]

Hi @Sanilblank @praweshsth - we've discussed and I've summarised below what we think the expected behaviour should be around codelists in IATI Publisher. This is slightly more complicated than originally thought as it does look like we need some record of deprecated codes kept within the system. Have a read and let me know if this is clear enough to go on?

One question we had - in your first comment on this thread @Sanilblank, you said "We will have to copy the contents of the updated json files present in the registry". We don't believe you can get these from the Registry so did you mean the IATI website?

---

Codelist updates

When an IATI codelist changes, the relevant page of the IATI website will be updated and changes should be reflected in IATI Publisher as soon as possible. The IATI website should remain synched with the IATI GitHub repo codelists, so either can be used by YI to find the most recent codelists. We expect codelists to change fairly frequently (every few weeks to a month), so it is worth automating the update process for IATI Publisher.

Expected user interface behaviour for IATI Publisher

How IATI Publisher data is treated will depend on whether it’s newly entered, already published or in draft state. It will also depend on whether IATI Publisher is displaying data that has already been entered or presenting a user with dropdown menu options to choose from.

Activity Data

Already published activity data from IATI Publisher should remain unchanged. Draft (unpublished) data in IATI Publisher should be treated differently depending on whether the activity has previously been published.

• If creating a new activity, the user should see updated drop-down menus on the data entry screens, preventing them from seeing or using deprecated codes.
• If publishing a draft activity for the first time, the user should be prompted to update any deprecated codes that they have used while creating it.
• If updating an already published activity, the user should not be forced to update deprecated codes they have previously entered. They should be able to see their data with the deprecated code in the IATI Publisher interface and IATI Publisher should let them edit and save sections containing a deprecated code.

Organisation Data

Already published organisation data from IATI Publisher should remain unchanged.

• If entering organisation data for the first time, the user should see updated drop-down codelist menus, preventing them from seeing or using deprecated codes.
• If updating previously published organisation data, the user should see deprecated codes in the interface if they have previously used them. They should be able to edit these sections of the Organisation data page without being forced to update the deprecated code.

Other considerations

Migrated organisations: If an organisation is migrated from AidStream to IATI Publisher, their data should be treated according to whether it is published or in draft state, as described in the sections above.

[PG-Momik]

TODO

• Upload all JSON files to AWS
• Update existing system to use files in AWS instead of the public folder
• Create a command to call the IATI API, check data, and update files in AWS
• Update the system to take deprecated codes into consideration in dropdown menus
• Changes to the publish workflow to show activities using deprecated codes
• In the detail page, show information about the deprecated codes when present

[sarinasindurakar]

• Issue 1
  The form for the default aid type is not being saved when the user uses the "3-Earmarking Modality" vocabulary code list.(in transaction too)

• Issue 2
  The error message shown in dot format in currency

• Issue 3
  Even when deprecated code is not used in the element, an error message still appears in activity detail page.

• Issue 5
  Show the level of element message in user readable form.

• Issue 6
  Server error is shown when user edits the organisation details.

• Issue 7
  If the existing activity has used deprecated code then form is not being saved.

• Issue 8
  The error message of deprecated code while publishing the activity is not shown.

• Issue 9
  The error message not shown for currency when user adds the deprecated code in the csv file .

[PG-Momik]

@emmajclegg This has been deployed to staging.

Since deprecated codelist do not show up on the dropdown unless they are used, I've added activity : "Activity with deprecated code" in the testing organisation: "My org's name", belonging to username: "emma_clegg_90"