Azure Web App hosting

[aoberoi]

Description

Azure-based hosting in App Service's Web Apps (formerly known as Websites) per the official Hubot deployment instructions is currently broken.

After some investigation in #283, we don't yet know if this is specific to hubot-slack, or if this is an issue with commonly used deployment instructions including the official instructions in hubot's documentation.

Workaround info:

1. The easiest workaround at the moment is to use the Linux platform when creating the environment for your Azure Web App. This eliminates iisnode, which we've determined was the root of the issue.

2. If you'd like to still run on the Windows platform, another workaround can be accomplished by setting a few specific configuration options. Thanks to @dude0001 for finding and sharing this!

   • Override the NODE_PATH environment variable according to this answer
   • Make sure that the nodeProcessCountPerApplication value is set to 1. The default value is 1, so if its not already set, you should be fine.
   • Enable the "Always On" option for your application.

This issue will help organize the tasks related to further investigate and resolve this issue, and improve the experience of deploying Hubot to Azure. We will close this issue when the instructions above work out of the box, or have an even easier mechanism for users of this adapter specifically.

Subtasks:

• Investigate whether issue is resolved by using the Azure CLI 2.0 (instead of the older Azure CLI). This will include using the Azure Resource Management APIs (instead of the Azure Service Management APIs) and building against App Services' Azure Web Apps (instead of Azure Websites).

  ➡️ No, its not specific to the CLI version. The 1.0 CLI has a helper to generate a deployscript locally, but the 2.0 CLI will generate that same deployscript upon push.

• Investigate if there's a way around the creation of server.js in the instructions, and if that might resolve the issue loading the hubot dependency.

  ➡️ The server.js file isn't the problem, the problem is that iisnode imposes an interceptor.js entry point, and server.js (or any application entry point we use) is require()d into the node program instead of being run directly. This forces require.main.require() to look in the wrong places to load the module: in the interceptor.js directory.

• Research different options for loading the hubot dependency (see Use require.main.require 'hubot' instead of require 'hubot' #137, Avoid our devDependencies when requiring 'hubot' #138) and see if they are viable fixes for the issue.

  ➡️ The choice to use require.main.require() still seems to be the "right" one, given the circumstances. iisnode is doing the weird, unexpected thing, but there are issues bigger than loading the hubot dependency that need to be wrangled. iisnode can potentially spawn many node processes, each scoped to an incoming HTTP request (because that's how Azure Web Apps are designed to work). This model is a poor fit for a long-running process like hubot with hubot-slack because we don't want many websocket client connections in distinct processes.

• Investigate if the issue is resolved by choosing the Linux platform instead of the Windows platform. This could be a short-term fix, but we should still strive for Windows compatibility. Are the same multi-process issues present?

  ➡️ If you're willing to use the Linux platform, the currently known issues are resilved.

• Explore using Azure Web Apps with WebJobs for deployment. Write a script that will copy the source to the appropriate WebJob directory before installing dependencies. Make hubot run in a continuous WebJob. Use a simple dummy entrypoint script that proxies requests to that WebJob, or simply get interceptor.js to do that.

  ➡️ Thanks to the workaround found by @dude0001, it seems that this exploration won't actually be necessary. That workaround, summarized above, should be roughly equivalent to running in a WebJob. If you find otherwise, please let us know and we'll update this information.

• Create an Azure Resource Management template for Hubot and distribute it in the Microsoft marketplace (see: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-deploy-portal#deploy-resources-from-marketplace).

• Contribute updated deployment instructions to hubotio/hubot docs.

Requirements (place an x in each of the [ ])

• I've read and understood the Contributing guidelines and have done my best effort to follow them.
• I've read and agree to the Code of Conduct.
• I've searched for any related issues and avoided creating a duplicate issue.

[aoberoi]

Application used to reproduce the issue (the git history is useful too): https://github.com/aoberoi/azure-hubot

The error is visible in the "Log Stream" view of the app from within the Azure Dashboard. In order to trigger the error, you visit the /test endpoint, which causes Azure to run the node program, which then crashes with the following message:

[Wed Jan 24 2018 01:28:38 GMT+0000 (Coordinated Universal Time)] DEBUG Loading adapter slack

[Wed Jan 24 2018 01:28:38 GMT+0000 (Coordinated Universal Time)] ERROR Cannot load adapter slack - Error: Cannot find module 'hubot'

I've tried to use environment variables HUBOT_LOG_LEVEL=debug and NODE_DEBUG=module, but I don't get any more information than that.

[aoberoi]

Actually, setting those environment variables from the Azure dashboard doesn't seem effective. Adding them from the CLI yielded lots more logs to look into. Will take some time to find the right module load: https://gist.github.com/aoberoi/f22ce55759aa8f93ea88e39b22c51571

[aoberoi]

Open question: why doesn't the deploy script use bin/hubot.cmd in the hubot app template?

It seems like Azure's node-specific deployment scripts are targeting projects that don't have a Windows-friendly launcher (e.g. make sure npm install is run first, then find an entry point), but the app template already provides that type of launch script specific for Windows.

[aoberoi]

The most interesting part of the log is in these three lines:

MODULE 11880: looking for "D:\\home\\site\\wwwroot\\node_modules\\hubot-slack\\src\\bot" in ["D:\\home\\site\\wwwroot\\node_modules\\hubot-slack"]

MODULE 11880: Module._load REQUEST hubot parent: .

MODULE 11880: looking for "hubot" in ["D:\\Program Files (x86)\\iisnode\\node_modules","D:\\Program Files (x86)\\node_modules","D:\\node_modules","D:\\local\\UserProfile\\.node_modules","D:\\local\\UserProfile\\.node_libraries","D:\\Program Files (x86)\\nodejs\\8.9.3\\lib\\node"]

Notice how instead of looking for the 'hubot' module in the "D:\home\site\wwwroot\node_modules", where it would have been found, the require.main.require leads node to search inside "D:\Program Files (x86)\iisnode\node_modules", where it doesn't exist. So there's some unexpected behavior where Azure is copying most of the program into this "D:\home\site\wwwroot" directory, but running server.js from inside the "D:\Program Files (x86)\iisnode" directory.

[aoberoi]

All signs point to a deficiency in https://github.com/projectkudu/KuduSync. Given that the project hasn't been touched in over 2 years, and that its forums haven't had any real activity in about that time, I think its safe to assume the project is dead. There's not much to be gained by engaging its maintainers and asking for changes.

At this point, it seems wise to explore what the modern deployment workflows from Azure and see if they no longer rely on KuduSync and we can rid ourselves of the issue that way.

Update: it turns out the deployment (which is basically the copying of files from the git repo to a wwwroot directory) is not the issue at all. KudoSync is one part of the deployment. It seems to be an issue with iisnode and how node is executed in that environment.

[aoberoi]

I think I've gained some understanding after reading the following file: https://github.com/tjanczuk/iisnode/blob/master/src/samples/configuration/web.config

In iisnode there's the concept of an interceptor, which is an application that runs instead of the target application. It sounds like this file would be treated as the main module in the node process, and therefore require.main would be a reference to its module value. That would explain the lookup path and not being able to load 'hubot'.

The question this creates is: Can we override this interceptor to be a part of the Azure Web App site?

[aoberoi]

Even if there's a successful workaround with setting our own interceptor, there's some serious limitations to iisnode.

It looks like long running processes are just not considered a use case fit for iisnode: tjanczuk/iisnode#569. This would result in many hubot processes being spawned and killed, which means the websocket connection would frequently be opened and closed, or potentially more than one concurrent connection being open at a time. I can see lots of undeterministic issues in this scenario.

Relevant: https://docs.microsoft.com/en-us/azure/architecture/best-practices/background-jobs. It looks like WebJobs is actually a better fit. It seems relatively easy to get going (especially considering the hubot.cmd script already being available as noted above). One limitation I do see is that robot.router functionality would pretty much become useless. Perhaps there's a way to have a Web App that just forwards requests to the express port in order to gain that functionality back.

Update: OOooOO It looks like forwarding requests is very much possible: https://github.com/projectkudu/kudu/wiki/Azure-Web-App-sandbox#network-endpoint-listening.

[dude0001]

Was this ever resolved? I am able to work around the issue with the answer here, and tell my app to additionally look at wwwroot\node_modules module dependencies. Curious if there is a better way as I'm trying to learn.

[aoberoi]

@dude0001 i haven't tried the workaround you mentioned, and i'm glad its working for at least a couple people. however, this only solves the loading dependencies problem, and not the bigger issue described above:

iisnode can potentially spawn many node processes, each scoped to an incoming HTTP request (because that's how Azure Web Apps are designed to work). This model is a poor fit for a long-running process like hubot with hubot-slack because we don't want many websocket client connections in distinct processes.

i think a better workaround would be to choose Linux as the platform for this app, now available on Azure, since it gets you away from iisnode.

we still haven't tried to implement this with Azure WebJobs, but that would be the way to fix this issue and still run on Windows.

[dude0001]

Do you think there are still issues with this configuration? If so, I'd be interested in contributing to using web jobs. Do you have any direction or tasks that need to be done?

I have the iisnode setting nodeprocesscountperapplication set to one to limit the web app to one instance. I then have the Always On option enabled in the web app itself.

.

[aoberoi]

@dude0001 thanks for point out these settings, i wasn't aware!

the combination of these two settings would indeed satisfy all my current concerns. of course, i can't speak from experience of running a hubot instance with these, so if you're willing to do this and report back any issues, we would be very grateful.

specifically, the nodeProcessCountPerApplication sounds like it would keep the hubot application limited to one process when sent to 1, which is actually the default, so that's great! also, the Always On option would also be necessary if we implemented the application as a WebJob. so it no longer seems to be necessary to work on a alternative of deploying hubot as a WebJob, since that route and the current implementation seem effectively the same.

if you'd like to contribute even further, here are a couple ways you can:

1. Help us create an Azure Resource Management template for Hubot and distribute it in the Microsoft marketplace (see: docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-deploy-portal#deploy-resources-from-marketplace).

2. Verify the deployment instructions to hubotio/hubot docs, and help us make suggestions to improve them. We would likely want to mention the Always On option (since its not default).

[aoberoi]

I've updated the summary of this issue to reflect what we've learned so far.