Augur Install does not display helpful information on failure

[Nebrethar]

Description:
I just tried to perform a dev install of augur. I ran the dev install script make install-dev and I get a nondescript error. I don't have any error details to work with.

How to reproduce:

1. Download augur
2. Run make install-dev
3. See error

Expected behavior:
A clear and concise description of what you expected to happen.

Screenshots

Log files
There are no log files that I can find here.

Software versions:

• Augur: Latest Master
• OS: WSL2 on Windows 1909

[Nebrethar]

I tried this on ubuntu 20.04 and I am getting an error at a different section of the installation. This error was immediate.. Here, I am also not finding any error information.

[Nebrethar]

I raised another concern, but moved it to #909

[ccarterlandis]

The are log files under logs/install, one for each step. This definitely needs to be documented explicitly in the install script output so people know where to look, and really, we should not be suppressing the errors at all - they should still be going to stderr. You've likely inadvertently written over those files already, but if you run them again they'll be there. Leaving this open as a reminder to output the log file location on error and to print those errors to the console.

[abhinavbajpai2012]

This kind of error can occur even when the installation files don't have read/write access. In some OS, when you clone a repo from github, by-default all the access isn't granted. Try granting 777 to all the installation files.

[ccarterlandis]

Huh I had no idea that was a thing, thanks for the call-out @abhinavbajpai2012. That could entirely be what's happening here...what OS are you using @Nebrethar?

[Nebrethar]

I've tried this with WSL2 and two different ubuntu VMs. One of the VMs had augur previously and one was a fresh install of ubuntu.

I tried to chmod (777) all of my augur files, just to make it work, but I'm running into the problem I had with the virtual #909 environment and cannot get past it. I do have a different error this time - I will post it there; we may have to fix that first

@abhinavbajpai2012 @ccarterlandis

[Nebrethar]

I'm really trying to make something clear:

• If I wanted to contribute to the Augur Project, and I was brand new to the project, I would be so intimidated by the lack of documentation and tricky install that I would never come back.

• When I was new to the augur project, I found it very difficult to learn how I can contribute. It took a lot of working with @ccarterlandis and @sgoggins and other people in the project to even learn how it works.

• The installation has become harder and harder over my two years' involvement with the project. The documentation has fluctuated in its difficulty. These should become more clear and easy to understand over that long of a period of time.

• Right now, from a contribution perspective, the project doesn't feel open source to me. It very much feels like an internal project with secret tricks.

These are the four main things I'm trying to communicate. I'm glad to help move this in a direction where it's easier to understand and contribute to, but I don't want to be the only one.

[sgoggins]

@Nebrethar : Ubuntu 20 was released a few days more than a month ago. I do not have a machine on Ubuntu 20, but I am not experiencing these errors on the last LTS, Ubuntu 18. I cannot say with certainty, but I think there may be a permissions issue as suggested.

Also, it looks like you may be running on Ubuntu for Windows, and that is not the same thing as Ubuntu and is likely the core issue here. On Native Ubuntu 18, for sure, Augur installs consistently.

In our platforms supported documentation we are pretty clear that we do not try to support Windows installation because its a lot of labor with little demand in the open source community.

[sgoggins]

@Nebrethar : I am going to say this in public because its important for everyone to understand: The "Ubuntu" you get in a Microsoft Windows distribution simply does NOT function for development anything like regular Ubuntu. This use of the Ubuntu brand under Windows is confusing, and the source of a great deal of PAIN for students in computer science that I teach, who expect they are getting a OS subsystem that will function EXACTLY LIKE Ubuntu. It simply does not, and frequently creates exactly the kinds of issues you describe for a dozen of the 70 students on Windows in my software engineering class.

Seriously: Ubuntu on windows is the most #2020 "OS" out there. :)

[ccarterlandis]

I'll add something to the docs to explicitly mention that WSL is not supported

[sgoggins]

@ccarterlandis : I would be as explicit about that as I am in my comment above. The Ubuntu on Windows "Subsystem" is NOT Ubuntu, and does not work like Ubuntu. It confuses a lot of people, and frankly I think its a bit dishonest and misleading.

[sgoggins]

(On Microsoft's part)

[ccarterlandis]

If I wanted to contribute to the Augur Project, and I was brand new to the project, I would be so intimidated by the lack of documentation and tricky install that I would never come back.

...

The installation has become harder and harder over my two years' involvement with the project. The documentation has fluctuated in its difficulty. These should become more clear and easy to understand over that long of a period of time.

Right now, from a contribution perspective, the project doesn't feel open source to me. It very much feels like an internal project with secret tricks.

I'm sorry you've been having a poor developer experience @Nebrethar, it's definitely not what we're going for. If I'm understanding you correctly, I think the main issue here is our extreme lack of attention to the documentation, which has lead to a confusing, messy and tricky install/user experience. All of your points are 100% valid, and may I say there is no greater wake-up call than being told your open source project feels like an internal one...

For context, I'm the only person who works on our documentation and installation with any sort of regularity, and my time I have to do that is often very limited. On top of school and my personal life, It's extremely hard to keep up when everything always falls to me. I can barely keep up with keeping the already written documentation up to date, so when things get tricky or out of date, usually the only way it'll get fixed is if someone notices it's broken and opens an issue. I mean even right now, the docs haven't been updated in months, nothing except for changing some trivial sample output code. There are a lot of docs I would like to write and update, everything from a robust developer guide and a DB schema analysis to common install FAQs and architecture overviews, all of which I think would go a long way in making Augur better. Without additional help however, I can't say when or even if those docs will ever get written.

Wrt to the install, I have also worked really hard to make the install more streamlined since last year. I thought it was going well but I am very close to the code so understandably it doesn't seem that bad to me. That being said, I do feel pretty strongly that the install requires too much input from the user and it has been slowly creeping in scope since it started, which doesn't help its complexity. Would love to hear your thoughts on how we can improve this.

I think there is also a larger issue of maintainability here. There are a lot of things in Augur that we have "patched up" in not the best way, and most of it was cobbled together to begin with. Much of the application code needs refactoring for robustness, security, stability, clarity, and design. Any code standards we do "follow" (ones we set for ourselves or otherwise) are almost certainly not actually enforced all the time, and we do an abysmal job of commenting too, which does not help.

@Nebrethar I appreciate you being vocal about your frustrations and for starting the discussion. It's an important one to have and I am happy to take part in it

[ccarterlandis]

^ Being said, let's keep this thread on track. @Nebrethar do you see ANY log files in the logs/ or logs/install/ directory after running make install-dev?

[Nebrethar]

Just for notation purposes:

Picture 1 is the installation failure on my 20.04.1 VM
Picture 2 is the installation failure on my 18.04.5 LTS VM

Both look like dead ends to myself or any new contributor.

[ccarterlandis]

Those permission errors in the 2nd picture are almost certainly what's causing the issue. To be honest, I'm not sure why it would be failing...maybe it's got something to do with the script itself not being allowed to create folders/files. Can you create that logs and logs/install from the command line? Just asking to see if that will fail, not as a solution/workaround

[Nebrethar]

@ccarterlandis regarding Ubuntu 20, I found a "permission denied" error in the log where the pip install of augur could not access the virtual environment folder that I created.

Using chmod 777 -R got it working, but that should also not be a requirement to install augur.

On 18.04 augur does not have any capability of installing without changing install scripts, and it does not create logs.

This comes down to some issues that led to this fiasco:

• The install logs are not known to the user.
• The use of sudo for debugging will break the program as it will not find the virtual environment.
• There is not enough supporting information elsewhere to help the user debug.

[Nebrethar]

I created the required directories for logging on 18.04, and they couldn't create a log file.
I used chmod 777 -R on the log directory and it failed, as seen below.

[ccarterlandis]

The use of sudo for debugging will break the program as it will not find the virtual environment

I would argue that sudo should never be used for debugging, and in fact I'd be willing to bet that's what's actually causing some of these permissions errors (I will concede that the docs make no mention of avoiding sudo, but this is because it assumes the user is familiar enough with sudo to already know that. This was a misguided assumption on my part). Using sudo with anything and then not running everything afterwards with sudo is almost certainly going to turn out to be a bad time. Even if you do always use sudo, there are commands inside these scripts that don't use sudo since they don't need to.

This installation of Augur is probably too messed up at this point from all the sudo. If you haven't already, I would try to completely rebuild the VM from scratch and then try to install without using sudo at all. All of our deployment servers run Ubuntu and Augur usually installs without a hitch so I'm a bit skeptical that it's actually a new Ubuntu problem. Also, what branch are you using?

[Nebrethar]

@ccarterlandis @sgoggins I am reinstalling 18.04, will pick this up tomorrow 👍

Tomorrow because I also need some time off of this 😅

[abhinavbajpai2012]

Recursively giving access to the root Augur directory should have worked. I had a similar error when I tried installing Augur on Deepin 20. As far as I remember, the error code was also the same (40). I just gave full access to the directory and then it started working.

Also, WSL doesn't work for Augur. Even if you solve the permission issue, the installation will give some error midway. I installed Ubuntu from the Microsoft Store and tried installing Augur on it and got some error which was something we can't solve. WSL's installation of Python is in some-way connected to the Windows installation on Python. I think there was some problem with OS commands (dir vs cd kind of) but don't remember completely.

[Nebrethar]

@abhinavbajpai2012 Thanks for your insight. I will respond to your latest comments one-by-one. First please see my comment from earlier.

Recursively giving access to the root Augur directory...

This is not a good practice when installing software. ...chmod 777... especially on something that is intended to run as a server, is a security vulnerability at best. This got me further in the install process with both ubuntu 18.04 and 20.04. I really hope it won't be this way in the future. If we require chmod 777, let's put it in the documentation so the community can see that problem.

Also, WSL doesn't work for Augur.

I've reiterated my solution to this a few times. I was only using WSL for the initial issue comment. After that, I have not been trying or mentioning the install using WSL. I've been working with Ubuntu 18.04 and Ubuntu 20.04. I do appreciate your personal experience with it, though, and the fact that you tried it too.

[Nebrethar]

I would argue that sudo should never be used for debugging, and in fact...

I ended up agreeing with this. I've only really worked on local environments most of the time so my use of sudo is becoming a big problem as I install things from source and find permissions errors.

I will try an install on an environment with minimal sudo usage. Let's see how that goes 😅

[Nebrethar]

This is on 18.04 LTS

I got to the part where I install frontend dependencies, and I have an Error 1

Looked in logs, and found that I don't have NPM installed yet on this machine. I need to have some way of knowing that before it gives me an error.

Continuing...

[Nebrethar]

This is on Ubuntu 18.04 LTS

I got another of the same error screen. I would send the screenshot but it is identical.

Here is the log:

I'm not sure of the nature of this error. Can someone please help with this?

[ccarterlandis]

@Nebrethar I am out of my depth with npm, but according to this issue, it might be an internal problem (hence that This is an error with npm itself line near the bottom), maybe due to this version of npm no longer being supported on this OS? Either way, the thread suggests updating node and seeing if that resolves the error. To upgrade everywhere on your machine I think the command is npm upgrade -g npm, but I could be wrong. I'm also not really sure if it's supposed to be updating node or npm, like I said I don't know anything about node/npm 💀 Let me know if that fixes the error

[sgoggins]

Fixed with latest Augur release.