General documentation improvements

[kksgandhi]

Hello! Firstly, thank you so much, this project is an incredible idea.

I am trying to get a self-hosted server running, but am running into issues. I'm slowly fixing them, but since these fixes aren't covered by the documentation, I'm unsure if they are intended solutions or merely hacks that will cause problems down the line.

I'm hoping for a more fleshed out documentation with perhaps even a tutorial on self hosting the server. If folks can give me unofficial advice in this thread, I'd be more than happy to write up the advice into formal documentation and open a pull request.

---

These are some of the issues I've run into so far.

1. I didn't realize I had to create a venv and install requirements.txt using pip install -r requirements.txt (I understand that this is simple, but even still it is something that could be added to the documentation to ease startup costs)
2. I created local settings and copied the recommendations from the readme, but the first import line failed since settings.py didn't exist (and I wasn't sure what the README meant when it said by "importing settings"). I simply commented that line out.
3. Connection was refused when trying to curl localhost:8080, so I changed the command to run to python manage.py runserver --settings=fairwork_server.local_settings lalhost:8080
4. curl localhost:8080 would connect, but caused an internal error

I'm now reading Django documentation to see what other steps I can take, but since this includes configuration that isn't covered in basic Django tutorials, I'm unsure how applicable it will be.

Furthermore, even when I have the server running, it's unclear how I will connect it to my MTurk HITs, since the fairwork website only has information on connecting using the hosted fairwork server.

Again, I sincerely appreciate the work that has gone into this project. I'm so incredibly happy that there are people out there who agree that, as researchers, we should be the ones taking the high road.

[tnarg999]

Thanks for trying out Fair Work and sorry for the late reply! We did not give a lot of direction on how to self-host Fair Work and we are grateful that you took the time to document the issues you ran into. I’ll try to provide some answers below:

1. A virtual environment isn’t exactly necessary to build Fair Work, but it certainly makes things easier. We can update our documentation to say that we recommend using venv and to run pip install -r requirements.txt to get all the necessary requirements for Fair Work.
2. We use settings.py to hold some sensitive variables and thus for security reasons the file is not included. You could leave it in plain text in your local_settings.py file since you are self-hosting Fair Work or for extra security measures, you could create your own settings.py file to hold variables such as your email password. We can update the documentation to say this.
3. python manage.py runserver --settings=fairwork_server.local_settings starts the Django server. By default, it runs on localhost:8000, but you can specify the port as per the documentation here https://docs.djangoproject.com/en/3.2/ref/django-admin/.
4. What internal error are you seeing when you start the Django server? It might be helpful to set the Django DEBUG setting to True to get more information (see https://docs.djangoproject.com/en/3.2/ref/settings/ for more info)

Once the server is running, if you go to localhost:8080 (or wherever the Django server is running), you should see the same website as if you went to https://fairwork.stanford.edu/. From there, you can register your API keys and get the html script to add to your MTurk HITs. However, after doing that you will still need to set up a method of delivering emails and you will also need a script to run the three python commands on a scheduled basis (see the “Auditing HIT payment” section of the documentation”).

For this reason, we actually recommend trying to set up your own Heroku server to self-host Fair Work (we currently use Fair Work). By using the SendGrid and Scheduler plugins, it is easy to set up an email server and run the python commands on a scheduled basis (and using the starter versions of these plugins is free). Hosting on Heroku also provides some more stability as opposed to running a localhost server on a computer at home. The latter is also certainly possible though.

Hope this helps! Please feel free to ask any other questions if you run into anymore issues trying to self-host Fair Work.

[kksgandhi]

Thank you! The internal error is copied below, the most important line is that 'Settings' object has no attribute 'ROOT_URLCONF'

System check identified no issues (0 silenced).
June 24, 2021 - 13:02:50
Django version 2.0.5, using settings 'fairwork_server.local_settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Traceback (most recent call last):
  File "/usr/lib/python3.5/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 146, in __call__
    response = self.get_response(request)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/handlers/base.py", line 79, in get_response
    set_urlconf(settings.ROOT_URLCONF)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/conf/__init__.py", line 57, in __getattr__
    val = getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'ROOT_URLCONF'
[24/Jun/2021 13:02:56] "GET / HTTP/1.1" 500 59

We are trying to avoid Heroku, since (due to privacy concerns) we are trying to avoid all third parties.

Doing some googling, it appears that there are a fair number of discussion threads on how to fix this issue. In an attempt to fix this, I added ROOT_URLCONF = "fairwork_server.urls" to local_settings.py, but this just raised another error:

Performing system checks...

Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x7ff118500d90>
Traceback (most recent call last):
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/utils/autoreload.py", line 225, in wrapper
    fn(*args, **kwargs)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/management/commands/runserver.py", line 120, in inner_run
    self.check(display_num_errors=True)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/management/base.py", line 364, in check
    include_deployment_checks=include_deployment_checks,
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/management/base.py", line 351, in _run_checks
    return checks.run_checks(**kwargs)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/checks/registry.py", line 73, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/urls/resolvers.py", line 397, in check
    for pattern in self.url_patterns:
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/utils/functional.py", line 36, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/urls/resolvers.py", line 536, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/utils/functional.py", line 36, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/urls/resolvers.py", line 529, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/ubuntu/fairwork/fairwork_server/urls.py", line 20, in <module>
    path('admin/', admin.site.urls),
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/contrib/admin/sites.py", line 285, in urls
    return self.get_urls(), 'admin', self.name
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/contrib/admin/sites.py", line 237, in get_urls
    from django.contrib.contenttypes import views as contenttype_views
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/contrib/contenttypes/views.py", line 2, in <module>
    from django.contrib.contenttypes.models import ContentType
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/contrib/contenttypes/models.py", line 134, in <module>
    class ContentType(models.Model):
  File "/home/ubuntu/fairwork/venv/lib/python3.5/site-packages/django/db/models/base.py", line 108, in __new__
    "INSTALLED_APPS." % (module, name)
RuntimeError: Model class django.contrib.contenttypes.models.ContentType doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

Unfortunately I have to focus on other research and can't continue debugging. If there's an easy fix then I'll certainly implement it, but otherwise no worries!