Cerebral Cortex is the big data cloud companion of mCerebrum designed to support population-scale data analysis, visualization, model development, and intervention design for mobile sensor data.
CerebralCortex provides an interface to retrieve/store mobile sensor raw data and metadata.
This repository is allows you to install and evaluate the Cerebral Cortex platform.
We have renamed following repositories.
- CerebralCortex-Platform -> CerebralCortex
- CerebralCortex - > CerebralCortex-Kernel
- 2018.03.18 Cerebral Cortex Cloud Platform
- Grafana visualization support
- Jupyter Notebook analysis platform
- Cerebral Cortex 3.0.0
- File system based storage architecture
- Ability to collect data from mCerebrum app (Note: mCerebrum is not compatiable for CerebralCortex 3.0.0 yet.)
This software is intended for informational and demonstration purposes only and is not designed to diagnose, treat, cure, prevent, or track disease or health states. No content provided in this software is intended to serve as a substitute for any kind of professional (e.g., medical) advice.
The Cerebral Cortex platform can be installed and tested on any of the three major platforms: Linux, Mac OS X, and Windows. The following instructions will walk you through installing the dependencies necessary to run Cerebral Cortex. The Docker images take approximately 8 minutes to download, build, and install on a 1000 Mbit/second internet connection. We are continually working on improving the speed with which everything downloads and installs.
These steps are performed from the command line and do not need a graphical interface.
sudo apt-get update
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
sudo apt-get install docker-cesudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-composePlease consult the Docker site if you face any installation errors for step 1 and 2.
- Download or clone this CerebralCortex repository.
git clone https://github.com/MD2KOrg/CerebralCortex
cd CerebralCortex
docker-compose up -dIn case you encounter an error relating to permissions, add your user to the docker group.
sudo usermod -aG docker $USER
sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-composePlease consult the Docker site if you face any installation errors for step 1 and 2.
- Download or clone this CerebralCortex repository.
git clone https://github.com/MD2KOrg/CerebralCortex
cd CerebralCortex- Update following configs in
docker-compose.yml. CommentHOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'", uncommentKAFKA_ADVERTISED_HOST_NAME:, and put your public ip address there.
environment:
# use either HOSTNAME_COMMAND or KAFKA_ADVERTISED_HOST_NAME
HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"
# KAFKA_ADVERTISED_HOST_NAME: YOUR-LOCAL-HOST-IP (not localhost or 127.0.0.1)
- Run docker-compose
docker-compose up -d-
Download the CerebralCortex repository.
Download location: https://github.com/MD2Korg/CerebralCortex/archive/master.zip -
Extract the
CerebralCortex-master.zipfile and open it inExplorer -
Open
PowerShelland run the following commands
PS C:\Users\hnat> cd Downloads\CerebralCortex-master
PS C:\Users\hnat\Downloads\CerebralCortex-master>
PS C:\Users\hnat\Downloads\CerebralCortex-master> docker-compose upNote: Approve the Docker's requests for accessing a shared drive
Open the following link to view the Cerebral Cortex launch page: http://localhost/
This version Cerebral Cortex is not configured with SSL/TLS support out of the box and should NOT be utilized to collect human subject data for research. It is currently designed for development, testing, and training purposes. We are working on a secure-by-default configuration for Cerebral Cortex to avoid this problem in the future. If you would like to secure this system as is, you will need to configure NGINX's SSL/TLS support manually.
Use the commands below to confirm that everything is installed and working correctly. Docker-Compose commands can be used to interact with Cerebral Cortex's containers. The following commands lists the the status of all the services used by CerebralCortex. Most containers for Cerebral Cortex will report a (healthy) state as long as they are running.
docker-compose psThe above commands display the status of all the services as such as the example shown below.
Name Command State Ports
---------------------------------------------------------------------------------------------------------------------------------------
cerebralcortex-apiserver /entrypoint.sh /start.sh Up (health: starting) 443/tcp, 80/tcp
cerebralcortex-grafana /run.sh Up 0.0.0.0:3000->3000/tcp
cerebralcortex-influxdb /entrypoint.sh influxd Up 0.0.0.0:8086->8086/tcp
cerebralcortex-jupyterhub sh -c chown -R md2k /cc_da ... Up (health: starting) 0.0.0.0:32777->8000/tcp
cerebralcortex-kafka start-kafka.sh Up 0.0.0.0:9092->9092/tcp
cerebralcortex-mysql docker-entrypoint.sh mysqld Up 0.0.0.0:3306->3306/tcp, 33060/tcp
cerebralcortex-nginx nginx -g daemon off; Up (health: starting) 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
cerebralcortex-zookeeper /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
Data is automatically imported into the system when mCerebrum is connected to the cloud platform. You can also initiate a replay of the data in the following way.
mCerebrum is our mobile application for Android that collects, processes, and transmits data to Cerebral Cortex. It is available here: https://md2k.org/mc2015 for download and installation. This app will let you authenticate with the default user and password md2k:md2k and your DNS entry for the Cerebral Cortex server. From this point, it will automatically download the configuration file one the system and start the setup process. The phone will upload data every 15 minutes data sources that are not raw sensor data and every hour for high-rate raw sensors. These will be reflected in the data processing and visualization interfaces
Current limitations:
- The provide mCerebrum configuration
mC_Demo.zipin theminio/configurationfolder need to be modified to support the proper data upload URL. Once you extract this zip file, edit themCerebrum/org.md2k.datakit/config.jsonfile and change theupload:urlto your DNS entry for Cerebral Cortex. Recompress the mCerebrum folder and overwrite the current zip file. - If mCerebrum does not properly connect to your provided DNS host running Cerebral Cortex, you MUST force stop the app on the phone before relaunching and logging into Cerebral Cortex.
Both of these bugs are logged in the tracking system and we will fix them when we can but this should not prevent you from testing out the platform.
Cerebral Cortex provides two mechanisms to visualize and analyze your data. First, a user-centric interface is provided by the Grafana project which can be utilized to plot and explore Cerebral Cortex data streams. Second, a code-centric interface is provided by the Jupyter project and allows a user to write Python 3 code to interact with the Cerebral Cortex kernel.
Open this link in your web browser http://localhost/grafana/login to visualize your data.
- The default login and password are both
admin.
- Once you authenticate, you will see the following screen.
- Select the
Homedropdown at the top-left of the screen and choose the MD2K dashboard.
This is a pre-built visualization that provides some examples of the various types of displays that are possible.
For example:
- Smartphone Accelerometer/Gyrometers/Magentometer
- Battery levels of connected devices
- Step count as determined from the smartphone
You may create additional dashboards to visualize all of the raw and processed data.
Open this link in your web browser http://localhost/jupyterhub/hub/login to interact and analyze your data.
- A login screen will be shown as follows.
The username and password are both md2k. The warning shown is because this site is running locally on your machine and is not secured by a security certificate. There is no data leaving the machine and going across the internet.
- A file browser will appear after successful authentication and you should choose the
cc_demofolder.
- Click on the
CerebralCortex_Basic_Usage.ipynband it will open in a new tab. This provides an overview of how to utilize Cerebral Cortex and visualize some data.
This example notebook demonstrates the following:
- Import CerebralCortex libraries and loading configurations
- Get all users of a study
- Get all streams of a user
- Get days when a stream has data available
- Get a stream's raw data and metadata
- Plot stream raw data
- Authenticate with user credentials
- Click on Files tab
- Click on new and select
pySpark (Spark 2.3.2) (Python 3)to create a new Python script.
The CerebralCortex-DataAnalysis repository is available as a Docker container (cerebralcortex-dataanalysis). This repository contains the code to compute features on the data. These features are located in the core/feature directory.
The following features have been validated by our team and are considered stable with the remaining features still under development. Please have a look at the documentation for each of the above features to get more insight into their functionality. Sensors/features in parentheses should be considered dependencies to compute the specified feature.
- phone_features (Smartphone)
- phone_screen_touch_features (Smartphone)
- puffMarker (MotionSenseHRV)
- activity and posture classification (MotionSenseHRV)
- typing features (MotionSenseHRV)
- rr_interval (MotionSenseHRV)
The container can be run against any temporal subset of data within the system after you have collected and/or ingested data. From the CerebralCortex-Platform folder run the following command. The console will output a log of the individual modules debug messages. Additionally, this container will launch at the initial docker-compose startup and process any data already in the system.
Format: docker-compose run dataanalysis sh compute_features.sh StartDate(YYYYMMDD) EndDate(YYYYMMDD)
docker-compose run dataanalysis sh compute_features.sh 20181101 20190101The Jupyter notebook also has the DataAnalysis code in the DataAnalysis folder.
In this folder there is a sample notebook Simple_driver.ipynb that can be used
to execute features described above. In this sample, the phone based features are
computed. Change the input parameters in the notebook to compute the desired features.
Stop Cerebral Cortex
docker-compose downStart Cerebral Cortex
docker-compose upDelete all data and container data for Cerebral Cortex
docker-compose stop
docker-compose rm
make clean- I'm stuck, where do I get help?
Please look for more information or ask for help here: https://mhealth.md2k.org/discuss/
-
How do I find out about new releases and software announcements
Please sign up for our software announcement mailing list: md2k-software@googlegroups.com
-
System requirements
These are the minimum recommended system requirements for running Cerebral Cortex.
- 2-4 core CPU
- 16+ GB RAM
- 10GB disk + enough to support the total data collection from mCerebrum
Please read our Contributing Guidelines for details on the process for submitting pull requests to us.
We use the Python PEP 8 Style Guide.
Our Code of Conduct is the Contributor Covenant.
Bug reports can be submitted through JIRA.
Our discussion forum can be found here.
We use Semantic Versioning for versioning the software which is based on the following guidelines.
MAJOR.MINOR.PATCH (example: 3.0.12)
- MAJOR version when incompatible API changes are made,
- MINOR version when functionality is added in a backwards-compatible manner, and
- PATCH version when backwards-compatible bug fixes are introduced.
For the versions available, see this repository's tags.
Link to the list of contributors who participated in this project.
This project is licensed under the BSD 2-Clause - see the license file for details.
- National Institutes of Health - Big Data to Knowledge Initiative
- Grants: R01MD010362, 1UG1DA04030901, 1U54EB020404, 1R01CA190329, 1R01DE02524, R00MD010468, 3UH2DA041713, 10555SC
- National Science Foundation
- Grants: 1640813, 1722646
- Intelligence Advanced Research Projects Activity
- Contract: 2017-17042800006