Merge pull request #18 from bulk-reviewer/prerelease

Bulk Reviewer 0.2.0

.gitignore

@@ -15,5 +15,6 @@ src/main/backend/stoplists/
 src/main/backend_dist/
 src/main/br_processor.spec
 src/main/build/
+br_processor.spec
 env/
 docs/_build/

.travis.yml

@@ -1,46 +1,48 @@
-osx_image: xcode8.3
 sudo: required
-dist: trusty
 language: c
+compiler: gcc
 matrix:
   include:
-  - os: osx
-  - os: linux
-    env: CC=clang CXX=clang++ npm_config_clang=1
-    compiler: clang
-cache:
-  directories:
-  - node_modules
-  - "$HOME/.electron"
-  - "$HOME/.cache"
-addons:
-  apt:
-    packages:
-    - libgnome-keyring-dev
-    - icnsutils
-  artifacts:
-    paths:
-    - $(ls BulkReviewer-*{dmg,AppImage})
-before_install:
-- mkdir -p /tmp/git-lfs && curl -L https://github.com/github/git-lfs/releases/download/v1.2.1/git-lfs-$([
-  "$TRAVIS_OS_NAME" == "linux" ] && echo "linux" || echo "darwin")-amd64-1.2.1.tar.gz
-  | tar -xz -C /tmp/git-lfs --strip-components 1 && /tmp/git-lfs/git-lfs pull
-- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then sudo apt-get install --no-install-recommends -y icnsutils graphicsmagick xz-utils; fi
-- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" && brew upgrade python; fi
+    - os: linux
+      dist: bionic
+      addons:
+        apt:
+          sources:
+            - ubuntu-toolchain-r-test
+          packages:
+            - g++-7
+            - libgnome-keyring-dev
+            - icnsutils
+            - graphicsmagick
+            - xz-utils
+            - libewf-dev
+            - libssl-dev
+            - libsqlite3-dev
+            - libboost-dev
+            - libicu-dev
+            - libtool
+            - sleuthkit
+    - os: osx
+      osx_image: xcode11
+      addons:
+        homebrew:
+          packages:
+            - libewf
+            - afflib
+            - sleuthkit
+            - bulk_extractor
 install:
-- nvm install 8
-- curl -o- -L https://yarnpkg.com/install.sh | bash
-- source ~/.bashrc
-- npm install -g xvfb-maybe
-- sudo pip install --upgrade virtualenv && virtualenv -p python3 env
-- source env/bin/activate && cd src/main && pip install -r backend/requirements.txt && pyinstaller backend/br_processor.py --distpath backend_dist
-- yarn install
-- yarn run build
+  - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then git clone --recursive https://github.com/timothyryanwalsh/bulk_extractor && cd bulk_extractor && chmod 755 bootstrap.sh && ./bootstrap.sh && ./configure && make && sudo make install && cd ..; fi
+  - nvm install 10
+  - curl -o- -L https://yarnpkg.com/install.sh | bash
+  - source ~/.bashrc
+  - npm install -g xvfb-maybe
+  - sudo pip install --upgrade virtualenv && virtualenv -p python3 env
+  - source env/bin/activate && cd src/main && pip install -r backend/requirements.txt && pyinstaller backend/br_processor.py --distpath backend_dist
+  - yarn install
 script:
-- |
-  if [[ $TRAVIS_BRANCH == 'master' || $TRAVIS_BRANCH == 'prerelease' ]]; then
-    yarn run release
-  fi
+  - python backend/test.py && yarn run build
 branches:
   only:
   - master
+  - prerelease

README.md

@@ -6,7 +6,14 @@
 
 For detailed installation and use instructions, see the [documentation](https://bulk-reviewer.readthedocs.io/en/latest/index.html).
 
-Bulk Reviewer is an Electron desktop application that aids in identification, review, and removal of sensitive files in directories and disk images. Bulk Reviewer scans directories and disk images for personally identifiable information (PII) and other sensitive information using [bulk_extractor](https://github.com/simsong/bulk_extractor), a best-in-class digital forensics tool. The desktop application enables users to configure, start, and review scans; generate CSV reports of features found; and export sets of files (either those free of sensitive information, or those with PII that should be restricted or run though redaction software).
+Bulk Reviewer is an Electron desktop application that aids in identification, review, and removal of sensitive files in directories and disk images. Bulk Reviewer scans directories and disk images for personally identifiable information (PII) and other sensitive information using [bulk_extractor](https://github.com/simsong/bulk_extractor), a best-in-class digital forensics tool. The desktop application enables users to:
+
+* Scan disk images and directories for private information with bulk_extractor (configurable with user-supplied regular expressions and stoplist files)
+* Review features found by type and by file in a user-friendly dashboard that supports annotation and dismissing features as false positives
+* Generate CSV reports of features found
+* Export sets of files
+	* Cleared: Files free of PII
+	* Private: Files with PII that should be restricted or run through redaction software
 
 Currently, Bulk Reviewer can scan directories and disk images for:
 
@@ -15,6 +22,7 @@ Currently, Bulk Reviewer can scan directories and disk images for:
 * Credit card numbers
 * Email addresses
 * Phone numbers
+* vCards (Virtual Contact Files)
 * URLs, web domains, RFC822 headers, and HTTP logs
 * GPS data
 * EXIF metadata
@@ -54,53 +62,53 @@ Scripts for installing system dependencies for macOS and Ubuntu 18.04 are includ
 
 1. Download the `BulkReviewer-x.x.x-x86_64.AppImage` [AppImage](https://appimage.org/) from the [latest Bulk Reviewer release](https://github.com/bulk-reviewer/bulk-reviewer/releases).
 
-2. Move the AppImage to the "Forensics and Reporting" folder on the BitCurator desktop.
+2. Save the AppImage file to location of your choice (e.g. the "Forensics and Reporting" folder on the BitCurator desktop).
 
-3. Make the Bulk Reviewer AppImage executable (with `chmod a+x FILE` in terminal or by right-clicking the AppImage, selecting Properties, and then selecting "Allow executing file as program" under the Permissions tab).
+3. Make the Bulk Reviewer AppImage executable (with `chmod +x FILE` in terminal or by right-clicking the AppImage, selecting Properties, and then selecting "Allow executing file as program" under the Permissions tab).
 
 4. Double-click the AppImage. A prompt will ask if you want to integrate Bulk Reviewer with your system. Choose "Yes" to install Bulk Reviewer.
 
-From this point forward, Bulk Reviewer can be launched by selecting it from the Applications menu or double-clicking on the AppImage file in the "Forensics and Reporting" folder.
+From this point forward, Bulk Reviewer can be launched by selecting it from the Applications menu or double-clicking on the AppImage file.
 
 ### Ubuntu 18.04
 
-1. Install dependencies in a terminal:
+1. Install dependencies in a terminal (you can skip this step if bulk_extractor and Sleuth Kit are already installed):
 
 ``` bash
 wget "https://github.com/bulk-reviewer/bulk-reviewer/blob/master/install_ubuntu18.sh"
-chmod a+x install_ubuntu18.sh
+chmod +x install_ubuntu18.sh
 sudo ./install_ubuntu18.sh
 ```
 
-2. Download the Bulk Reviewer [AppImage](https://appimage.org/) `BulkReviewer-x.x.x-x86_64.AppImage` from the [latest Bulk Reviewer release](https://github.com/bulk-reviewer/bulk-reviewer/releases).
+2. Download the Bulk Reviewer [AppImage](https://appimage.org/) `BulkReviewer-x.x.x.AppImage` from the [latest Bulk Reviewer release](https://github.com/bulk-reviewer/bulk-reviewer/releases).
 
-3. Move the AppImage to your home directory or desktop and make the file executable.
+3. Save the AppImage to location of your choice (e.g. your home folder or desktop).
 
-4. Double-click the AppImage. A prompt will ask if you want to integrate Bulk Reviewer with your system. Choose "Yes" to install Bulk Reviewer.
+4. Make the Bulk Reviewer AppImage executable (with `chmod +x FILE` in terminal or by right-clicking the AppImage, selecting Properties, and then selecting "Allow executing file as program" under the Permissions tab).
+
+5. Double-click the AppImage. A prompt will ask if you want to integrate Bulk Reviewer with your system. Choose "Yes" to install Bulk Reviewer.
 
 From this point forward, Bulk Reviewer can be launched by selecting it from the Applications menu or double-clicking on the AppImage file.
 
 ### macOS
 
 1. Make sure you have [Homebrew](https://brew.sh/) and [XCode](https://developer.apple.com/xcode/) installed.
 
-2. Download the `install_mac.sh` script from this repository.
-
-3. In a terminal, change directory to where you saved the `install_mac.sh` script and install dependencies:
+2. Install dependencies in a terminal (you can skip this step if bulk_extractor and Sleuth Kit are already installed):
 
 ``` bash
-brew install libewf afflib sleuthkit
-chmod a+x install_mac.sh
+wget "https://github.com/bulk-reviewer/bulk-reviewer/blob/master/install_mac.sh"
+chmod +x install_mac.sh
 ./install_mac.sh
 ```
 
-4. Download the `BulkReviewer-x.x.x.dmg` from the [latest Bulk Reviewer release](https://github.com/bulk-reviewer/bulk-reviewer/releases).
+3. Download the `BulkReviewer-x.x.x.dmg` from the [latest Bulk Reviewer release](https://github.com/bulk-reviewer/bulk-reviewer/releases).
 
-5. Double-click the dmg to open the Bulk Reviewer installer. Drag the Bulk Reviewer icon to the Applications folder to install.
+4. Double-click the dmg to open the Bulk Reviewer installer. Drag the Bulk Reviewer icon to the Applications folder to install.
 
 ## Development
 
-Bulk Reviewer is an Electron desktop application with a Python backend. Local development requires Python 3, Node, and npm/yarn (instructions here use yarn).
+Bulk Reviewer is an Electron desktop application with a Python backend. Local development requires Python 3, Node 10, and npm or yarn (instructions here use yarn).
 
 1. Clone this repository
 
@@ -111,6 +119,8 @@ git clone https://github.com/bulk-reviewer/bulk-reviewer
 2. Prepare Python virtual environment
 
 ``` bash
+cd bulk-reviewer/
+
 # First time
 virtualenv -p python3 env
 source env/bin/activate
@@ -139,24 +149,29 @@ yarn run dev
 2. Package Python script as executable
 
 ``` bash
-cd src/main
-pyinstaller backend/br_processor.py --distpath backend_dist
-rm -rf br_processor.spec
-rm -rf build
+./build-backend.sh
 ```
 
+(If build-backend script is not executable, make it executable with `chmod +x build-backend.sh`)
+
 3. Build Electron application for production
 
 ``` bash
-# Return to main bulk-reviewer directory
-cd ../..
-
-# Run build command
 yarn run build
 ```
 
 The resulting built application can be found in the `build` directory.
 
+## Test
+
+1. Follow Development steps 1-2 above
+
+2. Run backend tests
+
+``` bash
+python src/main/backend/test.py
+```
+
 ## Logo design
 [Bailey McGinn](https://baileymcginn.com/)
 

build-backend.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+
+# Delete backend_dist if exists
+if [ -d src/main/backend_dist ]; then
+  rm -rf src/main/backend_dist
+fi
+
+# Run pyinstaller
+pyinstaller src/main/backend/br_processor.py --distpath src/main/backend_dist
+
+# Clean up
+rm -rf src/main/br_processor.spec
+rm -rf src/main/build
+rm -rf br_processor.spec

docs/conf.py

@@ -24,9 +24,9 @@
 author = 'Tim Walsh'
 
 # The short X.Y version
-version = '0.1'
+version = '0.2'
 # The full version, including alpha/beta/rc tags
-release = '0.1.0'
+release = '0.2.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/index.rst

@@ -16,6 +16,6 @@ Bulk Reviewer documentation
     reviewdashboard
 
 
-**Current version: Bulk Reviewer 0.1.0**
+**Current version: Bulk Reviewer 0.2.0**
 
 Bulk Reviewer is an Electron desktop application that aids in identification, review, and removal of sensitive files in directories and disk images. Bulk Reviewer scans directories and disk images for personally identifiable information (PII) and other sensitive information using `bulk_extractor <https://github.com/simsong/bulk_extractor>`_, a best-in-class digital forensics tool. The desktop application enables users to configure, start, and review scans; generate CSV reports of features found; and export sets of files (either those free of sensitive information, or those with PII that should be restricted or run though redaction software).

docs/installation.rst

@@ -22,9 +22,9 @@ For production systems, a recent CPU with 4+ cores and at least 8 GB of RAM is r
 
 Installation in BitCurator
 --------------------------
-1. Download the ``BulkReviewer-x.x.x-x86_64.AppImage`` `AppImage <https://appimage.org/>`_ from the `latest Bulk Reviewer release <https://github.com/bulk-reviewer/bulk-reviewer/releases/>`_.
+1. Download the ``BulkReviewer-x.x.x.AppImage`` `AppImage <https://appimage.org/>`_ from the `latest Bulk Reviewer release <https://github.com/bulk-reviewer/bulk-reviewer/releases/>`_.
 
-2. Move the AppImage to the "Forensics and Reporting" folder on the BitCurator desktop.
+2. Move the AppImage to the "Forensics and Reporting" folder or another appropriate location on the BitCurator desktop.
 
 3. Make the Bulk Reviewer AppImage executable (right-click on the AppImage file, select Properties, and then select "Allow executing file as program" under the Permissions tab; or change the file permissions with ``chmod u+x`` in a terminal).
 
@@ -42,7 +42,7 @@ Installation in Ubuntu 18.04
     chmod a+x install_ubuntu18.sh
     sudo ./install_ubuntu18.sh
 
-2. Download the ``BulkReviewer-x.x.x-x86_64.AppImage`` `AppImage <https://appimage.org/>`_ from the `latest Bulk Reviewer release <https://github.com/bulk-reviewer/bulk-reviewer/releases/>`_.
+2. Download the ``BulkReviewer-x.x.x.AppImage`` `AppImage <https://appimage.org/>`_ from the `latest Bulk Reviewer release <https://github.com/bulk-reviewer/bulk-reviewer/releases/>`_.
 
 3. Move the AppImage to your home directory or desktop and make the file executable (right-click on the AppImage file, select Properties, and then select "Allow executing file as program" under the Permissions tab; or change the file permissions with ``chmod u+x`` in a terminal).
 

docs/newscan.rst

@@ -11,7 +11,8 @@ Processing options
 ------------------
 
 * **Name (required)**: Enter a name to identify your session. This could be a unique identifier such as an accession number or a brief description. The value entered into this field is used as the basename for the resulting JSON file and reports directory in the home user's ``bulk-reviewer`` directory (this is created by Bulk Reviewer during processing if it does not already exist).
-* **Source type/source (required)**: Select a source type of "Directory" for folders of digital files or "Disk image" for disk images. After you have selected the correct source type, use the "Choose directory" or "Choose file" button (depending on source type) to select the source you would like to scan for sensitive information.
+* **Source directory/disk image (required)**: Select a source type of "Directory" for folders of digital files or "Disk image" for disk images. After you have selected the correct source type, use the "Choose directory" or "Choose file" button (depending on source type) to select the source you would like to scan for sensitive information.
+* **Use existing bulk_extractor reports**: Select an existing bulk_extractor reports directory. If selected, Bulk Reviewer will not run bulk_extractor but instead read from the feature files in the specified directory. This allows users to run bulk_extractor separately, with any combination of scanners and arguments desired. When selected, the regular expressions, spotlist, and SSN identification mode options below will be ignored. Please note that currently for this option to work the source directory or disk image must be at the same filepath location as they were when the bulk_extractor reports were created.
 * **Regular expressions file (optional)**: If desired, use the "Choose file" button to select a plain text file containing `regular expressions <https://www.regular-expressions.info/>`_ to search for in the source directory or disk image. This allows users to search for any number of custom patterns. If a regular expressions file is specified, matches are included along with other features found by bulk_extractor in Bulk Reviewer's JSON output and in the Review Dashboard. To create a regular expressions file to use with Bulk Reviewer, create a plain text file in any text editor and add each regular expression you would like to scan for on its own line.
 * **Stoplist directory (optional)**: If desired, use the "Choose directory" button to select a directory containing stoplists to use with bulk_extractor. Stoplists in bulk_extractor are used to exclude known-acceptable features. For more detail, see section "4.4 Suppressing False Positives" of the `bulk_extractor Users Manual <http://downloads.digitalcorpora.org/downloads/bulk_extractor/BEUsersManual.pdf>`_. Stopped features will not be included in Bulk Reviewer's JSON output or the Review Dashboard, but are available for review in the ``_stopped`` feature files present in the reports directory created with each scan. The `domain.txt <https://github.com/bulk-reviewer/bulk-reviewer/blob/master/stoplists/domain.txt>`_ stoplist included in the Bulk Reviewer repository on Github is especially useful for ignoring common Adobe, Microsoft, PURL, and W3 namespaces. Other bulk_extractor stoplists are available for download at http://downloads.digitalcorpora.org/downloads/bulk_extractor/. Note that use of stoplists can dramatically increase processing time.
 * **Social Security Number identification mode**: Select from the dropdown the bulk_extractor SSN mode you would like Bulk Reviewer to use. Options are: