Omnibus PR for our consideration 2019.10.04 #746

rdhyee · 2019-10-04T18:15:17Z

There is a lot of work about lots of different things represented here -- more than what ideally should be in one nice feature PR. Hence, this PR is meant as a vehicle for assessing the current status of my work and how to break things up for merging.

Some of the things covered in this PR (not an exhaustive list):

use of Vagrant and ansible for creating machines
upgrade to Python 3.7 and Django 2.2.x
use of pipenv for dependency management
implementation of Let's Encrypt - Free SSL/TLS Certificates for staging servers

At this point, we're ready to try a staging server out in earnest to make sure there aren't any major problems -- and to understand whether the differences between the current production server and the ones created by vagrant/ansible are acceptable.

There are specific issues remaining to work out-- which will be detailed in the comments on this PR or separate issues.

update 3.6 to 3.7 in invocation to install pipenv

…file

… in build.yml

…sing problems

…oundError: No module named '_lzma' in pandas 0.25

…w installed

…p staging.opencontext.org

…irst place -- new Django version?

…lf-signed wildcard ssl cert for opencontext as placeholder for certbot

…0:00)

rdhyee · 2019-10-04T19:37:01Z

.gitattributes

@@ -20,3 +20,7 @@
 *.PDF	 diff=astextplain
 *.rtf	 diff=astextplain
 *.RTF	 diff=astextplain
+
+sysadmin/files/referral-spam.conf diff=ansible-vault merge=binary


I think it makes sense to encrypt open-context-py/referral-spam.conf at ry20191004 · rdhyee/open-context-py -- hence this line to help understand diffs in that file

rdhyee · 2019-10-04T19:38:42Z

sysadmin/templates/nginx_conf.j2

@@ -0,0 +1,374 @@
+$ANSIBLE_VAULT;1.2;AES256;oc


TO DO: extract out the sensitive stuff (like how we block certain requests) from the parts that would be very helpful to be plain text

rdhyee · 2019-10-04T19:41:24Z

Pipfile

@@ -0,0 +1,58 @@
+[[source]]


General approach for setting up Pipfile is to set minimum versions but not fix any versions unless necessary.

TO DO: Document use of pipenv and whether we should use an alternative · Issue #749 · ekansa/open-context-py

rdhyee · 2019-10-04T19:53:50Z

conftest.py

@@ -0,0 +1,22 @@
+import pytest


This file enables the use of production database for tests. One thing that needs to confirmed: is this the right place for this file?

OK. I can look into this also. We should mainly just configure database access for tests that go in the tests/regression path. The "unit" tests should be completely database free, and the "integration" tests can use a short term database that gets temporarily set up and then torn down after the tests complete.

rdhyee · 2019-10-04T20:38:21Z

sysadmin/build.yml

+
+    - name: Include vars of extra.yml
+      include_vars:
+        file: extra.yml


I will document how to use this in practice. e.g.,

oc_install_dir: /var/oc-venv allowed_host: happy.opencontext.org deployed_host: https://happy.opencontext.org deployed_site_name: "happy OCserver" server_name: happy.opencontext.org git_user_name: "Raymond Yee" git_user_email: "raymond.yee@gmail.com"

One thing that would be nice to work out: how to override variables that are in a dict like SECRET_KEYS

rdhyee · 2019-10-04T20:39:16Z

sysadmin/build.yml

+        name: www-data
+        state: present
+
+    - name: add {{user}} to www-data group


We will need a way to add more people than just {{user}} to the www-data group

rdhyee · 2019-10-09T18:53:39Z

sysadmin/Vagrantfile

+  #       exec "vagrant " + ARGV.join(' ')
+  # end
+
+  config.vm.define "opencontext" do |node|


So far, I've put in a lot of work on getting opencontext_predb working (the configuration that relies on a pre-existing database and solr instance). But I'd like to make sure opencontext configuration, which builds up a database and solr database from scratch also works.

That sounds great!!

rdhyee · 2019-10-09T19:00:01Z

opencontext_py/tests/test_basic.py

+        m_json_ld.request_full_path = '/projects-search/'
+        m_json_ld.spatial_context = spatial_context
+        json_ld = m_json_ld.convert_solr_json(response.raw_content)
+        assert json_ld['totalResults'] == 2


Change this to not test for a specific number of projects -- but to perhaps an inequality (e.g., >1).

rdhyee · 2019-10-09T19:11:30Z

sysadmin/Vagrantfile

+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+  # required_plugins = %w( vagrant-vbguest vagrant-disksize )


this commented out section is relevant to running Vagrant in conjunction with virtualbox, which has its own complications. I'll leave it commented out until one day there's a demand to get the virtualbox config working again.

rdhyee · 2019-10-09T19:18:59Z

sysadmin/crawl_open_context_static.ipynb

@@ -0,0 +1,145 @@
+{


To do: document how this Jupyter notebook can be used to download static files.

rdhyee · 2019-10-09T20:27:54Z

sysadmin/templates/secrets.json.j2

+    "SOLR_COLLECTION": "{{SECRET_KEYS['SOLR_COLLECTION']}}",
+    "STATIC_ROOT": "{{oc_install_dir}}/static",
+    "GEOIP_PATH": "{{geoip_path}}",
+    "FILE_CACHE_PATH": "{{oc_install_dir}}/cache/file-cache",


Possible enhancement: because there is more than one place where a file cache path is derived as "{{oc_install_dir}}/cache/file-cache", it might be helpful to compute that variable in one place (such as in the ansible playboo) and set to an ansible variable like file_cache_path that then gets inserted in secrets.json.j2

rdhyee · 2019-10-09T22:25:57Z

opencontext_py/apps/archive/tests.py

+        client = Client()
+        response = client.get(self.proj_context_uri, follow=True)
+        assert response.status_code in [200,301]
+        self.context_str = response.content
        self.data_str = self.load_json_file_str(self.data_file)


at this point, the setup still fails because self.data_str is still None.

do we need to check in dt-bone.json into the STATIC_IMPORTS_ROOT dir? Is there a complication to this?

…encontext

…into staging_prod_ssl

ekansa

OK! I read through your comments and commented. I think this looks good, and we should start trying to work from this new framework. I'm going to need your help / support however transitioning and learning new workflows and "ops" stuff, but lets get this done and move on to search issues. Thanks for all your hard work Raymond! We're definitely on a much better dev-ops foundation now!

…ose from the opencontext_predb

rdhyee · 2019-10-18T15:37:30Z

sysadmin/build.yml

+    # - name: Getting PYTHONPATH
+    #   local_action: shell python -c 'import sys; print(":".join(sys.path))'
+    #   register: pythonpath
+


for opencontext, there are still steps left to build an instance with test data, which then get fed into a solr index. I might want to ansibilize those steps:

vagrant up opencontext --provider=google

After the machine comes up:

scp $(vagrant ssh-config opencontext | tail -n +2 | awk '{if ($1) print " -o "$1"="$2;}') ~/Downloads/feb-2019-oc-test.backup localhost:/tmp

Alternatively -- login to machine and

wget -P /tmp https://dl.dropboxusercontent.com/s/0gnydnhw4dpco6c/feb-2019-oc-test.backup

login to machine

vagrant ssh opencontext cd /var/oc-venv pipenv shell

Then:

python manage.py flush --noinput

Load the data:

PGPASSWORD=opencontextpw pg_restore -c -v -d opencontextdb -U opencontextuser -h localhost -Fc -j 8 /tmp/feb-2019-oc-test.backup

then to set up solr:

sudo -u solr bash -c "/opt/solr/bin/solr delete -c open-context" sudo -u solr bash -c "/opt/solr/bin/solr create_core -c open-context" sudo cp /var/oc-venv/solr-config/Solr-7/schema.xml /var/solr/data/open-context/conf/schema.xml #sudo -u solr bash -c "cp /var/oc-venv/solr-config/Solr-7/solrconfig_201808.xml /var/solr/data/open-context/conf/solrconfig.xml" sudo cp /var/oc-venv/solr-config/Solr-7/solrconfig.xml /var/solr/data/open-context/conf/solrconfig.xml sudo cp /var/oc-venv/solr-config/Solr-7/currency.xml /var/solr/data/open-context/conf/currency.xml sudo cp /var/oc-venv/solr-config/Solr-7/elevate.xml /var/solr/data/open-context/conf/elevate.xml sudo cp /var/oc-venv/solr-config/Solr-7/email_url_types.txt /var/solr/data/open-context/conf/email_url_types.txt sudo chown solr:solr /var/solr/data/open-context/conf/* curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=open-context&&wt=json" sudo -u solr bash -c "/opt/solr/bin/solr restart"

and then fire up the django shell ( python manage.py shell)

from opencontext_py.apps.ocitems.manifest.models import Manifest from opencontext_py.apps.indexer.reindex import SolrReIndex uuids = [m.uuid for m in Manifest.objects.all()] print('Items to index:{} '.format(len(uuids))) sri = SolrReIndex() sri.reindex_uuids(uuids)

rdhyee · 2019-10-18T15:39:14Z

sysadmin/build.yml

+    #   local_action: shell python -c 'import sys; print(":".join(sys.path))'
+    #   register: pythonpath
+
+    # - debug:


Another thing that I've not built into build.yml is the series of database fixes (fixtures?) that might need to be run on old data. I've documented them at Some data fixes to apply to Open Context data

rdhyee added 30 commits September 17, 2019 11:05

Changes need to update to Django 2.2.x and Python 3.7

ab842c7

add new requirements to python_apt_package_deps for Python 3.7

7f7ee7e

update 3.6 to 3.7 in invocation to install pipenv

updated tests -- made to run on top of django2 updates

72cd357

add pandas requirement

cbc2fe8

first pass at allowing for customization of parameters in the Vagrant…

f29bb12

…file

getting pip to install correctly...but other problems in build left

56fedca

experiment with extra.yml to allow for overriding values of variables…

899801e

… in build.yml

loosen the versions in Pipfile

5e9c908

don't let pandas update pass 0.24.x at this point since 0.25.0 is cau…

776da7a

…sing problems

update Python to 3.7.4 and add liblzma-dev package because ModuleNotF…

782c932

…oundError: No module named '_lzma' in pandas 0.25

pandas = ">=0.24.0" since pandas 0.25 works because liblzma-dev is no…

5f08c5e

…w installed

working configuration for opencontext_predb Vagrant machine to spin u…

0fcb003

…p staging.opencontext.org

start tagging opencontext_predb instance

8b48bd5

some prerequisites for letsencrypt

30f5318

fixes to test_basic.py -- I'm not sure why these tests broke in the f…

cbe3f35

…irst place -- new Django version?

added server_name to nginx_conf.j2

2094d89

for debugging ansible module gcp_dns_resource_record_set

ede8656

latest attempt to run gcp_dns_resource_record_set

fe5d454

stepping stone to adding more sensitive stuff to nginx_conf.j2

69b68d2

adding more options to nginx_conf.j2

f64cfc4

adding configuration for SSL part of nginx configuration and add a se…

065213e

…lf-signed wildcard ssl cert for opencontext as placeholder for certbot

fixed syntax problems in build.yml

70da5bd

adding *.opencontext.org ssl cert (Expiry Date: 2019-12-23 14:22:14+0…

dc87af3

…0:00)

add network and subnetwork to Vagrantfile

793fded

typo in Vagrantfile for subnetwork

0f35377

pass 1 at incorporating rest of variables in secrets.json

db9b170

change oc_install_dir to /var/oc-venv

9634fbe

consolidate static_dir

afe5a85

adding utility scripts

2b14c1d

change file ownership to www-data

85e241c

rdhyee mentioned this pull request Oct 4, 2019

Vagrant ansible work in progress #740

Closed

rdhyee commented Oct 4, 2019

View reviewed changes

rdhyee mentioned this pull request Oct 4, 2019

Document use of pipenv and whether we should use an alternative #749

Open

rdhyee commented Oct 4, 2019

View reviewed changes

rdhyee added 4 commits October 5, 2019 11:12

try to setgid on {{oc_install_dir}}

d035927

set chmod g=y for {{oc_install_dir}} at the end

fed8503

debugging: permissions

8db5bb2

create static cache explicitly

21b4e7e

rdhyee commented Oct 9, 2019

View reviewed changes

move to next stages of fixing tests

969c8f2

rdhyee commented Oct 9, 2019

View reviewed changes

rdhyee added 4 commits October 10, 2019 17:59

refinements that I think are now working for opencontext_predb and op…

1dd9435

…encontext

Merge branch 'staging_prod_ssl' of github.com:rdhyee/open-context-py …

858c917

…into staging_prod_ssl

new certs should cover both *.opencontext.org and opencontext.org

cc1f14b

changing secret_key to match the curren production key

1fa5444

ekansa approved these changes Oct 12, 2019

View reviewed changes

rdhyee added 2 commits October 15, 2019 11:15

added cronjob to restart uwsgi on reboot

e586c26

fill out some parameters that I had left empty in opencontext with th…

12a9564

…ose from the opencontext_predb

rdhyee commented Oct 18, 2019

View reviewed changes

ekansa merged commit d892078 into ekansa:master Oct 18, 2019

rdhyee deleted the staging_prod_ssl branch November 26, 2019 15:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Omnibus PR for our consideration 2019.10.04 #746

Omnibus PR for our consideration 2019.10.04 #746

rdhyee commented Oct 4, 2019

rdhyee Oct 4, 2019

rdhyee Oct 4, 2019

ekansa Oct 12, 2019

rdhyee Oct 4, 2019

rdhyee Oct 4, 2019

ekansa Oct 12, 2019

rdhyee Oct 4, 2019

ekansa Oct 12, 2019

rdhyee Oct 4, 2019

rdhyee Oct 11, 2019

ekansa Oct 12, 2019

rdhyee Oct 4, 2019

rdhyee Oct 9, 2019

ekansa Oct 12, 2019

rdhyee Oct 9, 2019

rdhyee Oct 9, 2019

rdhyee Oct 9, 2019

rdhyee Oct 9, 2019

rdhyee Oct 9, 2019

ekansa Oct 12, 2019

ekansa left a comment

rdhyee Oct 18, 2019

rdhyee Oct 18, 2019


		Vagrant.configure(VAGRANTFILE_API_VERSION) do \|config\|

		# required_plugins = %w( vagrant-vbguest vagrant-disksize )

Omnibus PR for our consideration 2019.10.04 #746

Omnibus PR for our consideration 2019.10.04 #746

Conversation

rdhyee commented Oct 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekansa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment