bayanat v1.8

• Upgrade to Flask 2.0.
• Upgrade to Flask Security Too 4.0.
  • Update all system components to use username instead of email field.
  • Several changes to user management to accommodate changes in this new version.
• Upgrade multiple other requirements to major new versions.
• Geolocation feature:
  • Geo Marker component: allowing user to store geolocation info about important objects in videos.
  • Global map in Bulletin to visualize Geo markers as well as locations in the Bulletin and events.
• Refactoring location children search
• Search for children locations in Actors and Incidents.
• Accurate date search in all date fields.
• Add search box in Locations dashboard.
• Add assigned user and peer reviewer to preview cards.
• Add a setting to disable deduplication dashboard.
• Command to export Bulletins in JSON format.
• Refactoring event type display, fixing a bug that prevented event types for Actors from appearing in the list.
• Fix for a bug in media thumbnail rendering order.
• Fix for a bug that prevented adding related items to Incidents.
• Fix for a bug that affected video player dialog in rare circumstances.
• Fix for a bug in coordinates fields in Locations form.

This is a major release that brings new geolocation features to the Bulletin section of Bayanat, allowing the user to store, catalog and visualize coordinates for objects in videos. This new release also contains major changes to accommodate changes in the new Flask framework 2.0 version, Flask Security Too 4.0 version, as well as other requirements. It's important to upgrade to this version as soon as possible.

Ensure a backup of the database is taken before attempting to upgrade. Users should also clear their browsers' caches after any update.

To upgrade to this version, PostGIS needs to be installed:

sudo apt install postgis

The Postgres server should be restarted after installing this package.

After that, all requirements needs to be upgraded to the new versions. You should run in the directory of the Bayanat installation::

virtualenv env -p python3
source env/bin/activate
pip install -r requirements.txt

flask create-db can now be run to create the new geo markers component.

Then, there are a few migration steps in the Postgres database terminal for Flask Security Too upgrade. First, we need to create the new fs_uniquifier field and populate it:

alter table "user" add column fs_uniquifier varchar not null default md5(random()::text);

Secondly, as we switched the main identity to usersnames (instead of the email column), we need to generate usersnames. Our solution is to simply copy existing content of the email column:

update "user" set username = email;

Then, optionally, emails can be removed. This is especially a good idea if non-emails were used in that field as it might cause problems in UI email validation. This can be done with the following command:

update "user" set email = null;

Finally, we need to drop the non-null constraint for the email column:

alter table "user" alter column email drop not null;

bayanat v1.7.1

• Adjustments for back-end permissions
• Fix for issues which DA level users had accessing secondary components

bayanat v1.7

• Missing Persons: a new extension to the Actor component to facilitate the storage and analysis of missing persons data.
• Full refactoring of all back-end permissions.
• Full update to Arabic translation, which includes translation to the new Missing Persons extension as well as other strings that were added or change in previous versions and were missing.
• Major changes to deduplication matches processing:
  • Import command will filter unwanted matches, and administrator can specify a max distance to import.
  • Complete refactor of processing, utilizing celery and redis for faster processing and more control over speed:
    • Unprocessed matches are sent to celery in batches.
    • Batch size and time interval can be specified through variables in .env file.
  • Added an alternative fast-process command that utilizes multiprocessing for fast processing of the matches from CLI.
  • UI improvements for the deduplication dashboard.
  • Stop button to suspend processing, progress bar and status indicator.
  • Allow users with 'Mod' access level read-only access to the dashboard.
• Adding UI validation for Actor full name
• Add videojs to display cards, with central config for speed controls
• Move more settings to .env file
• Refactor actor search back-end to enable search for residence and origin locations, add fields to Actor search tool to search for the two locations
• Add Event comment to display and edit cards in Actors, Bulletins and Incidents
• Hid review from view-only users
• Implemented exact date search in Actors and Bulletins
• Return to page 1 of results after successful bulk update, fixing a bug that happened when the current page no longer exist due to search parameters.
• Added a button to clear all select items in menus.
• Added review to Bulletin edit form to allow users to check it while revisiting
• Added assigned user and current status to Bulletin preview card
• Fixed a bug in front-end with Actor-to-Actor relationship type not returned as an array.
• Fixed a bug in Actors where the place of origin wasn't saved.

This is a major release which brings the Missing Persons extension to Actors and other important changes. It's also important to update to this version as soon as possible to deploy the important changes to back-end permissions. Ensure a backup of the database is taken before attempting to upgrade. Users should also clear their browsers' caches after any update.

The Missing Persons extension is disabled by default. To enable it please check the configuration page.

There are some changes to the structure of the deduplication table. It's best to drop this table and recreate it using flask create-db.

Lastly, NGINX configuration needs to be updated in order to enable progress tracking of the dedupliciation processing. This can be done by disabling proxy buffering using proxy_buffering off; at location /. The NGINX configuration template has been updated.

bayanat v1.6

• Simple integration with Benetech's deduplication tool:
  • Import matches generated by generate_matches.py.
  • Process matches into relationships between Bulletins.
  • Dashboard to view matches and results of processing.
• Existing or newly added relationships are now hidden when searching for related items (Actors, Bulletins, Incidents).
• Search for children labels.
• Search for children sources.
• Force PDF viewer in Mozilla Firefox, fixing an issue where the file was downloaded with a specific browser setting.
• Fix for a bug with date entries when a 5-digits year is entered.

You should take a backup before upgrading to a newer version. Users should also clear their browsers' caches after any update.

To create the needed database structure for the deduplication dashboard, you should run in the directory of the Bayanat installation:

virtualenv env -p python3
source env/bin/activate
export FLASK_APP=run.py
flask create-db

bayanat v1.5

• Various changes to improve security
• Move sensitive settings from enferno/settings.py to a .env file
• Fix for an exception with text left in parent field when creating or editing a Label
• Fix for an exception with text left in parent field when creating or editing a Location
• Change to the way the location's full string is generated
• Fix for an exception with text left in parent field when creating or editing a Source
• Fix for a bug in removing a parent from a Source
• Fix for a bug in Labels form which make verified labels hidden in parent field
• Change 'Title' field in Bulletins to 'Original Title'
• Change 'SJAC Title' in Bulletins to 'Title'
• UI Improvements in all data tables
• UI improvements for actions feedback
• Add switch in settings file to disable media import tool

You should backup your settings file (enferno/settings.py) before updating to this new version. You should also take a backup of the database.

To update to this version, EnvironmentFile settings needs to be added to bayanat and celery systemd services files to point to the .env file. This is required as all sensitive settings were now moved from enferno/settings.py file. You can add the following line to both service files:

EnvironmentFile=/path/to/bayanat/.env

and remove other environment settings. You can take a look at the installation manual at docs.bayanat.org for templates for both service files.

Furthermore, you need to move all settings to a new .env file places in the root of the bayanat folder. A sample file is available in .env-sample. You can take a look at the configuration guidelines on docs.bayanat.org.

This method is more secure and also convenient for pulling updates without overwriting local settings.

There's an optional database change to disallow uploading duplicate files. You can run the following command in the PostgreSQL terminal:

alter table media add constraint media_etag_key Unique(etag);

However, you should check if you already have any duplicate entires in the table before making this change.

bayanat v1.4

• UI Improvements in main menus, search window, related items search and media import tool
• Improvements to media import tool
  • Video processing feature to convert video files to a browser-friendly extension
  • Select all and clear selection buttons
• Show controls in PDF viewer
• Add UNIX paths to source link validation

You should take a backup before updating to a new version.

Ffmpeg is required to use the video processing feature. You can install it using:

sudo apt install ffmpeg

It's is advisable to reduce the number of celery workers to a number suitable to the server specifications to avoid overloading the server with multiple video processing tasks.

bayanat v1.3

• Improvements to data import tool:
  • Server-side import with sub-folder scanning
  • Adding verified labels
  • Improved logging
  • Added import to activity monitor
  • Added batch ID
  • File hash check for duplicates
  • Fix for data persistence issue in UI
  • Support for more extensions
• Added missing place of origin to Actor edit form
• Added location type to location edit form
• PDF viewer now opens PDFs in a popup (same page)
• Indexed source link for search
• Added permission for bulk update to moderators

You should take a backup before updating.

To update from a previous version, you'll need to run the following in PostgreSQL terminal:

alter table bulletin add column search text generated always as ( id::text || ' ' ||
COALESCE(title, '') || ' ' || COALESCE(title_ar,'') || ' ' || COALESCE(description,'')  || ' ' || COALESCE(originid,'') || 
' ' || COALESCE(sjac_title,'')    || ' ' || COALESCE(sjac_title_ar,'')  || ' ' || COALESCE(description,'')   || ' ' || COALESCE(comments,'')  
) stored;
create index ix_bulletin_search on bulletin  using GIN (search gin_trgm_ops);
alter table actor add column search text generated always as ( id::text || ' ' ||
COALESCE(name, '') || ' ' || COALESCE(name_ar,'') || ' ' || COALESCE(originid,'') || ' ' || COALESCE(description,'')   || ' ' || COALESCE(comments,'')  
) stored;
create index ix_actor_search on actor  using GIN (search gin_trgm_ops);
alter table incident add column search text generated always as ( id::text || ' ' ||
COALESCE(title, '') || ' ' || COALESCE(title_ar,'') || ' ' || COALESCE(description,'')  || ' ' || COALESCE(comments,'')  
) stored;
create index ix_incident_search on incident  using GIN (search gin_trgm_ops);
drop index ix_bulletin_description;
drop index ix_bulletin_title;
drop index ix_bulletin_tsv;
drop index actor_fulltext;
drop index ix_actor_tsv;
drop index ix_incident_tsv;
create index ix_bulletin_history_bulletin_id on bulletin_history(bulletin_id);
create index ix_actor_history_actor_id on actor_history(actor_id);
create index ix_incident_history_incident_id on incident_history(incident_id);
update bulletin set tsv = null;
update actor set tsv = null;
update incident set tsv = null;

This is to migrate full text search to use postgres computed columns, cleaning up previous indexes and tsvector columns (no longer used).

Celery systemd service also needs to be updated for the use of data import tool. You can check the installation guide.

bayanat v1.2

• Data import tool to import video and image files in bulk from local folders
• Change Actor to Bulletin relationship type to multi
• PDF viewer
• Fixed a bug with mixed content in Locations
• Fixed a bug with password change for users
• Fixed a bug with pages number in Labels dashboard
• Fixed a bug with removing a related Actor from an Incident

To upgrade from v1.1, you'll need to run the following in PostgreSQL terminal:

alter table atob alter COLUMN related_as type integer[] using ARRAY[related_as];
update atob set related_as = '{}' where related_as='{NULL}';

This is to accommodate Actor to Bulletin relationship changes and migrate any existing relationships.

A new system package needs to be installed:

sudo apt install libimage-exiftool-perl

And requirements need to be updated:

cd bayanat
virtualenv env -p python3
source env/bin/activate
pip3 install -r requirements.txt

bayanat v1.1

• Major change to the search functionality, improving performance
• Added button to show item history
• Added children location to search results
• Added filters for creation and last change dates
• Added filter for peer review status
• Added ref field to peer review form
• Removed email verification from user management
• Changed captcha to optional
• Changed user order in menus to alphabetically
• Fixed multiple bugs in related items in Incidents
• Fixed multiple bugs in peer review form and process
• Fixed a bug in labels dashboard where verified labels were missing from the item menu
• Fixed a bug in description fields when switching between edit and review forms
• Fixed a bug in dates fields
• Fixed a bug in nested queries
• Fixed a bug in user menu pager
• Fixed a bug in image viewer in review form
• Fixed an issue in ref field in bulk form where it was removing all existing refs
• Fixed multiple issues in data persistence in edit and review forms
• Fixed a bug that prevented removing only event in a Bulletin
• Fixed a bug in related Bulletin search where some results where missing
• Fixed a bug in password change form that prevent users from changing their own password