Master Doc Update From Dev

docs/api/api_project.json

@@ -1,14 +1 @@
-{
-  "name": "",
-  "version": "0.0.0",
-  "description": "",
-  "sampleUrl": false,
-  "defaultVersion": "0.0.0",
-  "apidoc": "0.3.0",
-  "generator": {
-    "name": "apidoc",
-    "time": "2018-05-11T19:12:21.742Z",
-    "url": "http://apidocjs.com",
-    "version": "0.17.6"
-  }
-}
+{  "name": "",  "version": "0.0.0",  "description": "",  "sampleUrl": false,  "defaultVersion": "0.0.0",  "apidoc": "0.3.0",  "generator": {    "name": "apidoc",    "time": "2018-06-05T16:17:32.568Z",    "url": "http://apidocjs.com",    "version": "0.17.6"  }}

docs/dev-guide/dev-guide-overview.md

@@ -0,0 +1,69 @@
+# Augur Development Guide: Making Contributions
+
+## Getting Started
+See our [Contributing to Augur](/CONTRIBUTING.md) guide for specifics on how we review pull requests. tl;dr: Fork our repo. Make a change. Use the mailing list for [CHAOSS](https://lists.linuxfoundation.org/mailman/listinfo/oss-health-metrics), Submit a pull request. 
+
+## Building Changes
+After making your changes, run 
+```bash
+make build
+```  
+to update the docs and frontend before adding them to your staging area.
+```
+```
+
+## Augur Development Overview
+
+### tl;dr
+  1. [Back End Development Guide](/docs/dev-guide-pt1.md) 
+  2. [Front End Development Guide](/docs/dev-guide-pt2.md)
+---------------------
+
+### Augur's Design Value System
+Augur's architecture is designed with an eye toward fulfilling its primary missions of rapid open source software metric prototyping and using data visualization to facilitate discussions among folks who manage open source communities. 
+
+**Core aims: **
+  1. Rapid metrics prototyping
+  2. Using data visualization to support discussion
+
+**Our visualization design follows two principles:**
+  1. Allow comparisons across projects
+  2. Where logical, show trends over time on a metric
+
+Project comparison helps people understand what a metric tells them.  If I show you total commits in a month or a year, what does that tell you about the health of an open source project?  If  you are able to compare a project you are managing with a project or two in the same space that you are familiar with, is that helpful? In most cases the answers are yes. We aim to produce not only metrics, but enough information for consumers of Augur to construct meaning. Which then helps the CHAOSS community build better, more useful metrics. 
+
+Time is, in effect, a project focused type of comparison. If you can see the changes in different metrics on your project over time, its easier to maintain awareness of how metrics compare with results. 
+
+--------------------------------
+
+## Augur's Architecture
+In our aspiration to implement our value system for metrics in software, we seperate concerns pragmatically.  Any metrics dashboard system has to do 4 things: 1) Ingest data, 2) store data, 3) reshape data for analysis and 4) present data. Of course, these "dashboard requirements" can be interpreted and circumvented.  For example, if robust API's are available, like the [GitHub Version 4 API](https://developer.github.com/v4/), persistence can be considered optional. 
+
+Right now, Augur satisfies the enumerated dashboard system requirements in concrete ways we describe in the following four sections. 
+
+### Augur Data Ingestion
+We use the GHTorrent database, or its MSR14 little brother to help you get up and running quickly. You may find this dataset insufficient for a particular metric you want to build. API's available from a number of places can be accessed from Augur. 
+
+Inside your Augur system root directory there is another directory named Augur.  This is where the python files that you can modify live.  
+
+1. **downloads.py** : gathers download statistics for github repositories. Currently configured for npm and ruby gems download data. 
+2. **ghtorrent.py** : reads the ghtorrent database you installed.  There are two functions at the top of this file that allow you to do counts on the GHTorrent Schema tables quickly using Python. 
+    -     def __single_table_count_by_date(self, table, repo_col='project_id', user_col='author_id', group_by="week"): Generates query string to count occurances of rows per date for a given table.
+    -     def __sub_table_count_by_date(self, parent_table, sub_table, parent_id, sub_id, project_id): Generates query string to count occurances of rows per date for a given query sub-table. A query sub-table is a table that describes in more detail a specfic asset of another query table-for example, the table "pull_request_comments" is a sub table of "pull_request", where the query is pull requests.
+3. **ghtorrentplus.py** : Accesses the aggregate tables Augur creates for GHTorrent. 
+4. **githubapi.py** : Pulls data from the GitHub API
+5. **librariesio.py** : Pulls data from the libraries.io API (Package manager download data)
+6. **localcsv.py** : Pulls data from a .csv file you persist. 
+7. **publicwww.py** : Pulls download data from the https://publicwww.com website.
+
+If you want to ingest substantial amounts of new data, you may want to contribute to the [Augur-OSSifragae](https://github.com/OSSHealth/augur-ossifragae) project, which focuses on systematic, structured ingestion of open source respository data from heterogeneous sources. (Note: an [Ossifragae](https://en.wikipedia.org/wiki/Bearded_Vulture) is a bearded vulture that is one of a handful of birds said to yield valid signs for ancient Roman Augurs (visionaries) to follow. We think the existance of "OSS" at the beginning of the birds name is a sign. We took it.)
+
+### Augur Data Storage
+The database system that you built with GHTorrent or MSR14 is our principle data storage environment right now. We have added a few small details to the GHTorrent database in a seperate schema called GHTorrent_Plus.  This schema is built on deployment, and supports aggregations of the GHTorrent Schema information so that metrics can be generated more quickly in a few cases. 
+
+### Augur Data Reshaping for Analysis
+Now that you understand the basic structure of Augur, our [Back End Development Guide](/docs/dev-guide-pt1.md) will be the place to start for reshaping data and building out analysis endpoints. The end result of new back end code is a **REST API Endpoint.**
+
+### Augur Data Presentation 
+Once you have a **REST API Endpoint**, you can stop and say, "I have built an endpoint. Here is my pull request".  OR, you can build front end visualizations for those endpoints, following our [Front End Development Guide](/docs/dev-guide-pt2.md).
+

docs/dev-guide/dev-guide-pt1.md

@@ -0,0 +1,101 @@
+# Developer Guide Part 1 - The Backend
+
+## Structure of the Backend
+
+Augur uses the Flask framework for its backend, which is stored in the directory `augur`. `augur/__init__.py`, `augur/server.py`, `augur/deploy.py`, and `augur/util.py` contain the components. The other `augur/*.py`files contain python funtions that return dataframes to be serialzed into json by the functions in `augur/server.py`. The titles of those files are the data sources the metrics use.
+
+## Setting up your environment
+
+Before you begin, make sure to activate the augur Anaconda environment by running `conda activate augur`. If this environment doesn't exist, try running `make install-dev` again and watch out for any errors.
+
+## Writing a Function for Augur
+
+### Should I create a new .py file?
+
+If your python function uses a new data source, create a new Python file. If you use an already implemented data source, create your new functions under that file. For instance, if you were to create a metric using data from the GitHub API, you would write a function in [`augur/githubapi.py`](https://github.com/OSSHealth/augur/blob/master/augur/githubapi.py)
+
+#### Adding a new .py file
+
+In the file, create a class to put your functions into, then in `augur/__init__.py` add a line with the following format
+
+```python
+from .file_name import Class
+```
+For example if I added a file named `augur/chaoss.py` that contains the class `Chaoss` the addition to `augur/__init__.py` would be
+
+```python
+from .choass import Chaoss
+```
+
+### Writing a function
+
+In Augur there are metrics and timeseries metrics. For all metrics, the function should return a Dataframe that can be serialized into json. For timeseries metrics, the Dataframe needs to have a column named `date` that holds timestamps.
+
+#### Adding dependencies
+
+If you need to add a dependency to Augur for your function, simply add the import statment to the file as usual, then in `setup.py` add the dependency to the `install_requires` list. For example, if my new function uses a package called `mizzou`, I would find the `install_requires` list:
+
+```python
+install_requires=['beautifulsoup4', 'flask', 'flask-cors', 'PyMySQL', 'requests', 'python-dateutil', 'sqlalchemy', 'pandas', 'pytest', 'PyGithub', 'pyevent', 'gunicorn'],
+```
+
+and add `mizzou` as such:
+
+```python
+install_requires=['beautifulsoup4', 'flask', 'flask-cors', 'PyMySQL', 'requests', 'python-dateutil', 'sqlalchemy', 'pandas', 'pytest', 'PyGithub', 'pyevent', 'gunicorn', 'mizzou'],
+```
+
+#### Adding tests
+
+Augur uses pytest for tests. Tests are in the `test` directory. If you created a new file for your data source, you will also need to create a new file to test it. You can use pytest fixtures and environment variables to pass data to tests.
+
+```python
+@pytest.fixture
+def chaoss():
+    import augur
+    chaossServer = os.getenv("CHAOSS_TEST_URL")
+    assert chaossServer is not None and len(chaossServer) > 8
+    return augur.Chaoss(chaossServer)
+```
+
+Now any test that tests functions in the Chaoss class will be able to access an instance of the class
+
+```python
+def test_data_source(chaoss):
+    assert chaoss.data_source('argument').isin(['expected_value']).any
+```
+
+Make sure every function you write has a test.
+
+## Creating an endpoint for a function
+
+If you created a new data source, make sure you create an instance of your class, loading any configuration you need with the `read_config` function.
+
+To create an endpoint for a function, in [`augur/server.py`](https://github.com/OSSHealth/augur/blob/master/augur/server.py), call  `addMetric()` or `addTimeseries()`
+
+```python
+addTimeseries(app, file_name.function_name, 'function_name')
+```
+for a function `foo()` in `augur/bar.py`
+
+```python
+addTimeseries(app, bar.foo, 'foo')
+```
+If the metric is not a timeseries metric, replace `AddTimeseries()` with `AddMetric()`
+
+```bash
+
+## Using the Python Debugger
+
+The server in Augur has a built-in IPython debugger to make testing your functions easier during development.
+
+After you have added an instance of your class to server.py, you can test it by running:
+
+```bash
+export AUGUR_INTERACTIVE=1
+augur
+````
+
+Augur will load configuration and create instances of all the classes, but will drop down to IPDB shell instead of running the Flask server.
+
+To disable the IPDB shell, simply export ```bash export AUGUR_INTERACTIVE=0``` and run `augur`.

docs/dev-guide/dev-guide-pt2.md

@@ -0,0 +1,131 @@
+# Developer Guide Part 2 - The Frontend
+
+## Structure
+Augur uses Vue.js, MetricsGraphics, Kube for its frontend. The frontend is stored in the `frontend` directory, but the parts that are relevant to adding new metrics are in the `frontend/app/` directory, which contains the following parts:
+* `frontend/app/AugurAPI.js` which interfaces with the backend
+* `frontend/app/AugurStats.js` which performs statistical operations on the data
+* `frontend/app/Augur.js` which renders the page
+* `frontend/app/assets/` which contains the assets, such as images, for the frontend
+* `frontend/app/include/` which contains the Kube and MetricsGraphics resources
+* `frontend/styles/` which contains the stylesheet for Augur
+* `frontend/components/` which contains the Vue.js components
+  * `frontend/components/charts/` contains the different chart templetes
+  * `frontend/components/BaseRepoActivityCard.vue` contains the repo activity time series charts
+
+## How to add a new timeseries metric
+
+### Adding an endpoint
+
+In `frontend/app/AugurAPI.js`, add an attribute to the repo class that holds a times eries object at the end of the file like this 
+```javascript
+repo.<endpointName> = Timeseries('endpoint_name')
+```
+So if your endpoint name is `foo_bar` then the attribute would be
+```javascript
+repo.<fooBar> = Timeseries('foo_bar')
+```
+
+### Adding a chart
+In `frontend/app/components/BaseRepoActivityCard.vue`, in the template in the section tag, add a div like this
+```html
+    <div class="row">
+      <div class="col col-'width'>
+        <line-chart source="attributeName" 
+                    title="Chart title" 
+                    cite-url="Optional link to explanation"
+                    cite-text="Optional link title">
+        </line-chart>
+      </div>
+```
+The recommended width is 6 for half width and 12 for full width. So if I wanted to add the `foo_bar`chart, it would look like this
+```html
+    <div class="row">
+      <div class="col col-12>
+        <line-chart source="fooBar" 
+                    title="Foo Bar" 
+                    cite-url="https://foobar.com"
+                    cite-text="Link to foo bar explination">
+        </line-chart>
+      </div>
+```
+### Adding Comparison Functionality
+In `frontend/app/components/ComparedRepoActivityCard.vue` in the template in the section tag, add a div like this
+```html
+    <div class="row">
+      <div class="col col-'width'>
+        <line-chart source="attributeName" 
+                    title="Chart title" 
+                    cite-url="Optional link to explanation"
+                    cite-text="Optional link title"
+                    v-bind:compared-to="comparedTo">
+        </line-chart>
+      </div>
+```
+The recommended width is 6 for half width and 12 for full width. So if I wanted to add the `foo_bar`chart, it would look like this
+```html
+    <div class="row">
+      <div class="col col-12>
+        <line-chart source="fooBar" 
+                    title="Foo Bar" 
+                    cite-url="https://foobar.com"
+                    cite-text="Link to foo bar explination"
+                    v-bind:compared-to="comparedTo">
+        </line-chart>
+      </div>
+```
+## How to add a new nontimeseries metric
+
+### Adding an endpoint
+
+In `frontend/app/AugurAPI.js`, add an attribute to the repo class that holds an endpoint object at the end of the file like this
+```javascript
+repo.<endpointName> = Endpoint('endpoint_name')
+```
+So if your endpoint name is `foo_bar` then the attribute would be
+```javascript
+repo.<fooBar> = Endpoint('foo_bar')
+```
+
+### Adding a chart
+In the `frontend/app/components/charts` directory define a chart for the metric
+
+In `frontend/app/components/BaseRepoActivityCard.vue` or `frontend/app/components/BaseRepoEcosystemCard.vue` import the chart at the bottom and export it
+
+In the same file, in the template in the section tag, add a div like this
+```html
+    <div class="row">
+      <div class="col col-'width'">
+        <chart-type source="attributeName"
+                    title="Chart title"
+                    cite-url="Optional link to explanation"
+                    cite-text="Optional link title">
+        </chart-type>
+      </div>
+...
+import ChartType from './charts/ChartType'
+
+module.exports = {
+  components: {
+    ChartType
+  }
+};
+```
+The recommended width is 6 for half width and 12 for full width. So if I wanted to add the `foo_bar` chart, it would look like this
+```html
+    <div class="row">
+      <div class="col col-12">
+        <foo-bar source="fooBar"
+                    title="Foo Bar"
+                    cite-url="https://foobar.com"
+                    cite-text="Link to foo bar explanation">
+        </foo-bar>
+      </div>
+...
+import FooBar from './charts/FooBar'
+
+module.exports = {
+  components: {
+    FooBar
+  }
+};
+```

docs/dev-guide/ghtorrent-restore.md

@@ -0,0 +1,5 @@
+-- Create Database and Grant Access
+create database ghtorrent_blue;
+grant all on ghtorrent_blue.* to 'ghtorrent'@'%';
+grant all on ghtorrent_blue.* to 'ghtorrent'@'localhost';
+grant file on *.* to 'ghtorrent'@'localhost'