Important
These docs are under construction and approximately 80% of what they will be.
Contents
- Introduction
- Base knowledge
- Terminology
- DEH in detail
- EspoCRM DEH Data model
- Lifecycle Diagrams
- Work environments
- DEH Configuration by admin
- Best practises
- Add/edit/remove fields
- Add/edit/remove layouts
- Translations in Flex (ask 510)
- Translations in Espo
- User management (draft)
- Roles
- Teams
- Edit opening/closing times in chatbot
- Edit introduction message in chatbot
- Adding/removing geographic permissions for messages/calls from foreign phone numbers
- TaskRouter (for future)
- Kobo form submissions to EspoCRM connection
- Reports/dashboards
- Configure notifications and stream
- Changing WhatsApp profile information in Console
- Enable email notifications
- Operational admin jobs
- Add/configure/remove users in Flex
- Add/configure/remove users in EspoCRM
- Add/remove WhatsApp templates
- Troubleshooting guidelines
- Steps to save browser logs
- Steps to get to technical error screenshot
- Known issues
- Monitoring logs, insights and costs
- Other important aspects
- Data Responsibility and Data Protection Impact Assessment (Emie)
- Lifecycle Diagrams
- Forbidden areas
As an admin you possess the highest level of access rights in DEH. Deviation from the instructions outlined in this manual may result in unintended system disruptions. Depending on the extent of the changes made and the modified settings, recovery could require significant effort from a team of developers, potentially taking several days. Therefore, we urge you to exercise caution and carefully follow this manual. For every production environment, there is also a staging environment to test things on. If you are unsure about changing certain things, you could oftentimes first try this in the staging environment.
This manual is only useful for already configured instances of DEH and it assumes that the reader has already read the agent/supervisor manual of DEH. This manual is not meant for setting up DEH from scratch since the GitHub documentation provides this already.
Why this section? The following gives the necessary background information on DEH. This could be beneficial when encountering an issue or when you want to get an overview of all the product components.
DEH is an omni-channel contact center with CRM integration for use across all Red Cross National Societies. This is a product for deploying digital services worldwide and leverages Twilio's communication capabilities mainly via Twilio Flex with EspoCRM as an underlying agent interface and CRM. The backend of this product is partly made possible via shared resources (a.o. data storage) in Azure.
The product comprises two main tangible components:
- A Twilio Flex instance providing agents with the functionality of receiving calls, Facebook Messenger, and WhatsApp via their device.
- An EspoCRM instance (embedded in Twilio Flex) allowing agents to retrieve and store relevant information relating to a person affected (PA) such as personal details, case information, interactions, etc.
| Terminology | Definition |
|---|---|
| EspoCRM | An open source Customer Relationship Management (CRM) solution |
| Azure | Microsoft's cloud computing platform serving as backend |
| Twilio Flex | Cloud-based contact center solution provided by Twilio |
| Twilio Console | UI of Twilio to manage accounts, buy and configure Twilio products such as Flex |
| Agent | Role of a user in Twilio Flex who interacts with PAs |
| Supervisor | Role in Twilio Flex who manages agents and can see internal dashboards |
| Admin | Role in Twilio Flex with privileged access. Admins can login via SSO and via the Console |
| Person Affected (PA) | Person in need of assistance from the Red Cross, or more broadly speaking, these are the persons who engage with the Red Cross |
| Case | A request, pertaining to a PA, which relates to a particular service that the Red Cross can assist with |
| Interaction | Details contact between a PA & a Red Cross agent. This may be a call, SMS, WhatsApp, etc. |
| Contact | The entity name for the a PA in EspoCRM |
Deploying DEH consists of deploying an Azure Infrastructure (one-time-only) and at least one DEH instance (one instance per country). The DEH instance is here defined as the combination of Twilio Flex and an instance of EspoCRM. The Azure Infrastructure uses shared resources which are used by multiple DEH instances to send data between Twilio Flex and EspoCRM. Usually there exists only two instances of Azure Infrastructure: one for staging and one for production. A DEH instance is deployed to only one Azure Infrastructure, while an Azure Infrastructure can have more than one DEH instances deployed. The relationship between Twilio Flex and an EspoCRM instance is one-to-one.
See the below data model to see how the three relate to each other:
EspoCRM is used as a CRM solution for storing information related to PA's, open cases, and interactions between agents & PA's. In addition to this the underlying database is used for storing extracted Twilio data utilised for additional context and reporting.
The default entity structure has been modified to suit the Product X use case. If starting from a fresh EspoCRM instance some modifications need to applied to match this configuration. The below image details the key entities and their relationships, where the 1-0 relation represents a one-to-many relationship:
To effectively work with DEH as an admin, various environments are essential:
- Twilio Flex
- Twilio Console
- Azure tenant for SSO
- EspoCRM
- Virtual Machine
Twilio Flex is the contact center product from Twilio where DEH is built on top of. See it as the engine of DEH, where DEH is the custom car built around this engine. Each DEH user should be assigned to one of the three default Twilio Flex roles: agents, supervisors, and admins.
The Twilio Console is the cockpit of all things related to Twilio, and change configurations of Twilio Flex. From the Console you can access DEH as an admin by opening Twilio Flex via the Console. In the Console you will also find products related to DEH which are not Flex, for example: Twilio Studio for chatbots and IVR flows (dialmenu for calls), WhatsApp templates, Twilio TaskRouter for configuring the way tasks (conversation or calls) are presented to users (like routing a specific chat or call to a specific agent) and there is the possibility to see logs and monitor errors via the Console.
Azure tenant for SSO is currently a separate Azure tenant (next to the existing NLRC Azure environment) where users can be added, configured, and removed from DEH.
EspoCRM is a open source CRM software that allows us to create, store and adjust contacts, cases and other information of the people we are in contact with.
The Virtual Machine (VM) is a computer resource that uses software instead of a physical computer to run programs and deploy apps. EspoCRM is hosted on a VM that was specifically set-up for DEH. The VM is one of the resources on Azure.
Note
This manual tries to refer as much as possible to other documentation. This is because EspoCRM has good general documentation which is updated regularly and should be the one source of truth. If there is generic EspoCRM knowledge missing in the EspoCRM docs, there is another place to capture relevant knowledge for the humanitarian sector specifically: EspoCRM Knowledge Base by 510. If something is not mentioned in the General EspoCRM docs AND not generic enough to place in the Knowledge Base, it will be described on this page.
When changing EspoCRM, there are certain best practises to follow. It is highly recommended to read these at least once. These can be found in the EspoCRM Knowledge Base Best Practises Page
To add, edit or remove a field, please refer to the field section in the EspoCRM General Docs, the best practises around fields in the EspoCRM Knowledge Base and the DPIA page
Note: there are certain fields which are impossible to remove. These are part of the DEH extension and DEH will not work properly without those fields. There is always the possibility to not use those fields in the layout.
To add, edit or remove layouts, please refer to the EspoCRM docs or the best practises around layouts in the EspoCRM Knowledge Base
Note
The below translation information is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
Translations of everything in Twilio Flex (so not the EspoCRM frame) can currently only be configured by 510. The text of the translations can be of Twilio Flex can be found in the following JSON.
The current process to change translations text in Twilio Flex would be the following:
- Get the JSON for the language and translations you want changed. 510 can assist here.
- Change the words/sentences that you want to change.
- 510 will upload the edited version of the JSON. Please note: this will affect the translation of the changed language globally for every instance.
Future process: there will probably be a translation JSON per langauge hosted in a blob storage. Admins have access to edit this JSON online which means changes are immediately deployed.
The EspoCRM Knowledge base translations section describes how to change labels in different languages, both manually and automatic.
The generic information on user management is very well described in the EspoCRM Docs.
Note
From this point ontwards the user management information is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
DEH specific users are 'admin' and 'regular' (for supervisors and agents). DEH specific API users are 'automatic-system-deh' (Twilio API integration) and can be extended with an Kobo API integration.
Everything about generic roles is described in the EspoCRM Docs.
Note
From this point onwards the user management inofrmation is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
DEH specific roles can be found here
Teams are a way to group users together. Everything generic about teams in EspoCRM is described in the EspoCRM Docs.
Teams are also required if the 'Sensitive case' functionality is added to DEH. Ask 510 to assist since this is an add on to the standard solution.
Note
From this point onwards, the below teams information is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
Teams in Twilio Flex are grouped using Skills and these are used for routing certain messages/calls to a certain team first before routing to everyone (for example). To add team in Twilio Flex, ask 510 to assist since this requires some configuration steps. To add a user to a team in Twilio Flex, you need the admin or supervisor role in Twilio Flex and then you can add/remove a skill for a user in Flex.
Note
The below chatbot information is useful for Twilio Flex. If you only use EspoCRM, this is not relevant. The opening hours/days determine if conversations/calls can end up with agents in Twilio Flex. If a message/call is received outside of opening times, it will not be shown to in Twilio Flex but it will be logged in EspoCRM as new conversation.
Steps:
- Go to Twilio Console
- Studio
- Flow
- Select flow named '(Default) Opening Hours Subflow'
- Click widget named ‘set_top_variables’
- Edit ‘openTime’ or 'closeTime' in hh:mm format
- Edit 'openDay' or 'closeDay'. 1 means Monday, 2 Tuesday, etc..
- Edit 'timeZone' in Continent/City format
- Save
- ‘Publish’ the flow on the top right of the screen
Note
The below chatbot information is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
The introduction message serves as the initial communication the receiver will receive upon engaging with the chatbot.
Steps:
- Go to Twilio Console
- Studio
- Flow
- Select flow of interest (for example, ‘(Default) Inbound Conversation Flow’)
- Click widget named ‘set_top_variables’
- Edit text in ‘welcomeMessage’
- Save
- ‘Publish’ the flow on the top right of the screen
Note: this is not an instruction to edit the introduction message for calls. That process has not been documented yet.
Note
The below phone number information is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
When using SMS and Call functionalities, Twilio only allows incoming numbers with a similar country code. If you require the capability to make and receive calls or SMS to and from other countries, you can enable access via a setting called geo-permissions in the Twilio Console.
Please keep in mind that:
- Certain countries carry a higher risk of toll fraud for calls, mostly related to toll-free numbers. More information about this multi-billion dollar fraud business can be found here
- Enabling SMS and/or Call functionality to foreign numbers involves higher costs. An estimation of costs can be found in the geopermissions settings, but it is wise to monitoring the costs regardless of whether foreign numbers are enabled or not.
Steps for geo-permissions regarding SMS:
- Go to Twilio Console
- Messaging
- Settings
- Geo permissions
- Filter by country (or Ctrl+F) to find the country of interest
- Save geo permissions
Steps for geo-permissions regarding calls:
- Go to Twilio Console
- Voice
- Settings
- Geo permissions
- Filter by country (or Ctrl+F) to find the country of interest
- Save geo permissions
Note
The taskrouter information is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
There is a publicly available and free product from 510 that allows you to create your own Kobo to EspoCRM connection. Following the steps in the knowledge base, you can send Kobo form submissions to an entity and field of choice in EspoCRM. It is recommended to use KoboConnect for this.
Reports are only available when the Advanced Pack has been purchased. This allows you to create dashboards and reports in EspoCRM. A generic explanation can be found in the EspoCRM docs There are also ready-made DEH specific reports which can be found here (Emie)
Install notifications and stream
Note
The below info on WhatsApp profiles is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
This is the information showcased on the WhatsApp number's profile. It includes a profile picture and a concise business description. Additionally, it can be expanded to include hyperlinks to the website, email, and address.
Steps:
- Go to Twilio Console (make sure you are in the correct account)
- Messaging
- WhatsApp senders
- Click on WhatsApp number of interest
- In the section ‘Business profile information’, you can edit information
- Update WhatsApp sender
There are two places where agents and supervisors need to be added: to Twilio Flex and to EspoCRM.
Steps for adding and configuring a user in Twilio Flex, using Azure Active Directory and/or Microsoft Entra Admin Center
- Make sure the user is already in Azure Active Directory or Microsoft Entra Admin Center, if not add this user first
- for assigning roles (agent/supervisor/admin) to already added user:
- Make sure you are on page ‘Azure Active Directory’ or 'Azure Entra ID'
- Enterprise applications (in the left bar)
- All applications (in the left bar)
- Twilio Flex (or 'DEH Twilio Flex' or similar)
- Users and groups (in the left bar)
- ‘+ Add user/group’ on the top bar
- Select user(s)
- Select role
- Assign
When the new user has first logged in to Twilio Flex, there is one step for adding 'Twilio Flex Skills' that needs to happen in Twilio Flex to fully onboard the user. A supervisor and/or admin can do this:
- Go to Twilio Flex (as admin or supervisor)
- Go to ‘Your Teams’ (left bar)
- Select user of interest
- Add skill(s) commmessenger if the agent should receive incoming Facebook Messenger, commsms if the agent should receive SMS, commvoice for calls and commwhatsapp for WhatsApp. Click the blue button ('+') to make sure to add the skill.
- Add skill(s) related to language (for example languagenl-nl, if user is able to receive conversations in Dutch, other examples include ‘languageen-us’ for English and more depending on the specific configuration)
- Add skill project*everyone
- Don’t forget to save with the blue button in the bottom right
Note: in the future this will be done differently.
Note: by default the user will have a limit of 1 open conversation/call in Twilio Flex. Admins can increase this in the Twilio Console in TaskRouter by the following steps:
- Go to Twilio Console
- Go to “TaskRouter - Overview
- Go to “Workspaces” to “Flex Task Assignment”
- Go to “Workers” (left menu)
- Select worker of interest
- Under the section ‘Task Channels’
- Adjust ‘chat’ capacity for the maximum number of WhatsApp conversations in personal queue
- Adjust 'voice’ capacity for the maximum number of calls in personal queue
Steps to add and configure a user in EspoCRM for when SSO is configured: (recommended approach)
- Make sure the user is already in Azure Active Directory or Microsoft Entra Admin Center, if not add/invite this user first
- Go to Enterprise Applications in Azure
- Search for DEH EspoCRM and click to enter
- Go to 'Users and groups'
- '+ Add user/group'
- Select the user that should have access to EspoCRM
- Select the base role of that user in EspoCRM (Agent or Supervisor)
- When this new users logs in to the EspoCRM part of DEH for the first time, a user will be automatically created in EspoCRM with Agent or Supervisor teams permissions. If you want to give other permissions to this user (e.g. add the Team 'First Aid' to this user), you should go EspoCRM -> Administration -> Users -> Teams: add relevant Teams
Note
The 'role' field in the user of EspoCRM should not be configured. The user inherits roles from the Teams where the user is assigned to. So you only have to assign Teams to a user and not roles. The Teams will have the correct roles attached.
- You can add other information as well (e.g. Name, Title, phone nr, gender, avatar), but these are usually not mandatory
- Click the 'Save' button.
Twilio Console is only meant for admins and the occasional supervisor. Please don't give access to people who don't need this. This is not the same as Twilio Flex but you can access Twilio Flex with an Admin roel
Steps for adding Twilio Console users (other admins and/or possible supervisors, not agents) to Twilio Console:
- Go to Twilio Console
- Account (top right)
- Manage users (and probably complete 2FA popup)
- Invite user
- Add email address and select role (Administrator for Admins, Developer for supervisors, Billing Manager for Budget Holders)
- Send invitation (please note that invitation expires after a period)
Steps to add and configure an admin and user in EspoCRM for when SSO is not configured: (not recommended approach for agents and supervisors)
- Go to EspoCRM and log in (you should have the admin role)
- Administration -> Users -> '+ Create User'
- 'User Name': add the email address of the user
- Add First Name and Last Name
- 'Email': add the email address of the user
- Type: Regular (this is the default) or Admin
- Teams: add relevant Teams (for example, Agent Team and/or Cash Team)
- Roles: don't assign any roles here since the roles are already inherited from the earlier selected Teams
- 'Is Active': make sure this is checked
- Leave the following fields empty: 'Default Team', 'Working Time Calendar, 'Layout Set', Password
- You can add other information as well (e.g. Title, phone nr, gender, avatar), but these are not mandatory
- Click the 'Save' button. This will send an email to the user with a generated password for the first login.
Steps to add and configure an admin and user in EspoCRM for when SSO is configured: (recommended approach but has a workaround in steps 9 and 10)
- Make sure the user is already in Azure Active Directory or Microsoft Entra Admin Center, if not add/invite this user first
- Go to Enterprise Applications in Azure
- Search for DEH EspoCRM and click to enter
- Go to 'Users and groups'
- '+ Add user/group'
- Select the user that should have access to EspoCRM
- Select the Admin role of that user in EspoCRM
- When this new users logs in to the EspoCRM part of DEH for the first time, a user will be automatically created, but without Admin permissions
- Log in as with another account as Admin in EspoCRM
- Change the type of earlier added user from Regular to Admin
- From now on, the user can log in as Admin with SSO
Note
The below info on WhatsApp templates is useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
Whatsapp templates serve two primary purposes: for agents sending messages in conversations that have been inactive for 24 hours (WhatsApp/Meta policy does not allow an agent to send self-written messages after this period), or when a standardized message is to be send out.
Steps:
- Go to Twilio Console (make sure you are in the correct account)
- Messaging
- Content Editor
- Create new
- Template Name should be something logic including language (for example: start_conversation_dutch)
- Select Template Language
- Select Content Type should be ‘Text’
- Click Create
- In Body, type your template (please keep in mind WhatsApp template rules)
- Click ‘Save and submit for WhatsApp approval’
- Wait until approved (usually under a minute and you will probably get a confirmation email)
- Add WhatsApp template as Canned Response in EspoCRM so that every user can find and use this
- Refresh Twilio Flex and then the templates are visible in the Canned Responses
The following websites are great resources to troubleshoot different areas:
- Twilio Products Current Incidents Status (If Twilio Products have an outage, they will communicate all updates via this website)
- User Network Test (If user experiences issues, this is an easy tool to run to see if this relates to network problems)
- A tutorial from Twilio to debug Twilio Flex can be found here
- Debugging Tools for Twilio Console errors
- Debugging your Twilio Application (very technical)
- Error Handling and Diagnostics (very technical)
Providing your browser logs can be helpful for the developer (Zing) in understanding and resolving the issue. You or your agent can save the logs by following the steps below. Be sure to include the logs, after removing any sensitive data, in your conversation with the developer for further assistance.
Steps:
- Right-click in the browser window or tab and select Inspect.
- Click the Network tab in the panel that appears.
- Recreate the steps that caused the issue to show, once the error has happened again.
- Click the download button ("Export HAR" appears when you hold the pointer over it).
- Name the file.
- Click Save and send it to help@zing.dev (Note: the HAR file will have the full contents of the request, so if there is any sensitive data in that request which should not be shown to external parties, you should open the file in a text editor and scrub/remove any sensitive values before sending)
If you or your agent encounter an issue that is not already known, or cannot be resolved with the information provided in this manual, nor through your expertise, take the following steps: Capture a screenshot of the technical error and forward it to our developer, Zing. They will assist you in addressing the issue.
Steps:
- Right click on the internet page and search for 'Inspect'
- Click inspect.
- In the top row you will see the word 'Console' next to 'Elements' and 'Sources", click on this 'Console'
- On the left bar you will see 'error', click on this.
- Make a screenshot
- Save the screenshot and send it to help@zing.dev
This is a list of currently known issues and how to mitigate those:
- ....
[see redline admin manual]
The below monitoring logs, insights and costs are useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
Steps to find error logs in Twilio Flex:
- Go to the Twilio Console and make sure you are in the right account
- Monitor on the top left bar (Ctrl + F is your friend)
- Logs
- Errors
- Error logs
Steps to find messaging logs:
- Go to the Twilio Console and make sure you are in the right account
- Monitor on the top left bar (Ctrl + F is your friend)
- Logs
- Messaging (note: during configuration there could be a setting configured which removes message logs after a period of time because of GDPR requirements)
Steps to find messaging insights (aggregated messaging logs):
- Go to the Twilio Console and make sure you are in the right account
- Monitor on the top left bar (Ctrl + F is your friend)
- Insights
- ‘Messages’ AND ‘Message Deliverability’
Steps to find call logs:
- Go to the Twilio Console and make sure you are in the right account
- Monitor on the top left bar (Ctrl + F is your friend)
- Logs
- Calls (note: during configuration there could be a setting configured which removes call logs after a period of time because of GDPR requirements)
Steps to find call insights (aggregated call logs):
- Go to the Twilio Console and make sure you are in the right account
- Monitor on the top left bar (Ctrl + F is your friend)
- Insights
- Calls
Steps to find currently active conversations (collection of messages) logs:
- Go to the Twilio Console and make sure you are in the right account
- Monitor on the top left bar (Ctrl + F is your friend)
- Logs
- Conversation (note: during configuration there could be a setting configured which removes conversation logs after a period of time because of GDPR requirements)
Steps to find overview of costs:
- Go to the Twilio Console and make sure you are in the right account
- Monitor on the top left bar (Ctrl + F is your friend)
- Insights
- Billing Usage
- DPIA (Emie)
[NOTE!] The below lifecycle diagrams are useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
These flowcharts represent all the tiny steps of DEH that take place from start: the moment a call or message was send in, to end. An admin can use the flowcharts to further understand DEH. The steps are different for incoming messasges (conversation), calls and outbound messages (outbound conversation). We refer to the images below.
Caution
There are certain areas which can have devastating consequences when changed. Only an admin has access to these areas, but should not touch this.
This is a list of potential destructive actions in EspoCRM to be aware of:
- Changing the global UI (Administration>User Interface). This affects UI for all users of EspoCRM.
- Remove EspoCRM Single Sign On (SSO) settings (Administration>Authentication). This will disable every user from using SSO to login in EspoCRM.
- Uninstall the EspoCRM Flex DEH Extension (Adminstration>Extensions). This will remove all DEH relevant extensions and DEH related data will not be visible anymore in EspoCRM.
- Uninstall the Advanced Pack Extension (Administration>Extensions). Only relevant if this Advanced Package is installed. Uninstalling it will remove all the reports (dashboards) and flowcharts/workflows.
- Upgrade EspoCRM (Administration>Upgrade). DEH is currently made for EspoCRM version 8.1.0. A newer version will not ensure compatibility with DEH.
- Certain role settings for users (Administration>Roles). The role permissions 'Export Permission' and 'Mass Update Permission' are very powerfull permissions which should be considered carefully before assigning to a role. This would allow a user to quickly export large volumes of personal data or change large volumes of records in EspoCRM.
- Disabling or removing an entity (Administration>Entity Manager>[Entity]>Edit).
- Removing relationships between entities (Administration>Entity Manager>[Entity]>Relationships).
- Removing Email configuration settings (Administration>Outbound Emails). Removing this will stop the email notifications to users and password reset emails.
- Removing all other users (and admins) so that no one has access anymore.
- Remove the API role or API user called automatic-system-deh (Administration>Roles and Administration>API Users). This will break the connection between Twilio Flex and EspoCRM and no data will be transferred anymore.
- It is not allowed to make fields required in the DehContact, DehInteraction, DehMessage and DehCall entity. Why? Because records in these entities will be created automatically via the system and it does values of required fields and the record creation/popup will therefore fail. You can make fields required within the entity DehCase.
Note
From this point onwards, the forbidden areas are useful for Twilio Flex. If you only use EspoCRM, this is not relevant.
Caution
This is a list of potential destructive actions in Twilio to be aware of:
- Remove Active Phone Numbers
- Remove SSO setting for Twilio Flex (Flex>Manage>Single sign-on). This will disable every user from logging in to Twilio Flex.
- Disconnect Studio Flows (chatbot/IVR) with Twilio Flex (Flex>Manage>Messaging>Filter by Address type)
- Removing TaskRouter settings
- Removing Studio flows
- Removing WhatsApp sender (Messaging>Senders>WhatsApp senders)
- Removing Facebook Messenger (Channels>Facebook Messenger)
- Removing Event Streams
- Removing API keys & tokens