Is this project dead now?

[bryantbiggs]

☝🏼 - honest question. trying to decide if I should fork or whats going on

[AndrewFarley]

@bryantbiggs Sorry, I think @bsamuel-ui is afk at the moment, and I'm quite swamped as well. But I'm around and just saw this via Slack, and will turn back on my notifications for this repo. Can I ask for assistance perhaps in deciding which pending MR's need approving so I can cut a release? There seems to be a lot of great ones in there. Anyone offer to help sift through them with me?

[AndrewFarley]

Oh, man it looks like I lost or no longer have write-access to accept merges. :( So... we're at the whim of others. @dschep / @bsamuel-ui around to help grant me or someone else maintainer privileges?

[bryantbiggs]

to start, getting some basic dependencies updated to patch vulnerabilities would be good (shameless: #586). outside of that, how can I help?

[dschep]

Fair question @bryantbiggs! I initially wrote this when at UnitedIncome, and continued to do some maintenance/mgmt because I was Serverless Inc's Python SME, but I haven't done anything with this in quite a while now (I do still use it on a side project here & there, more on that in a bit).

So.. that is to say, I'm not sure. I dunno if UnitedIncome is using it internally still(cc @dwolfand), but if they are, it's probably like how it was used when I was there & wrote this plugin(and as I alluded to, is how I still occasionally use this plugin on side-projects). That is to say: we don't use most of the features.

I initially wrote this because I wanted a solution simpler(and not tied to wsgi) than serverless-wsgi. We made it open source, then we accepted PRs from the community to try and keep the sls python plugin ecosystem from being fragmented. But now, that means there's now plenty of code in this plugin we're not super familiar with, and don't use.

@AndrewFarley, you & @miketheman have Triage permissions now it seems. Not sure when that got downgraded. I seem to still be an admin so I could change that, but I don't want to step on @dwolfand's toes since this repo is in the UnitedIncome org.

[bsamuel-ui]

UI is still using it internally, and I think I've got similar problems as @dschep, there's stuff we just don't use, especially the Windows support. Let's chat about making this be not dead, especially getting a release out the door sounds promising. @bryantbiggs I've sent you an invite to the serverless-contributors stack, can you check the #python-requirements channel>

[bryantbiggs]

thanks all! I use this in nearly every project (unless a client makes me use node), so I am very appreciative of what you all have provided here. And yes, I can understand that as the maintainers there are some hard decisions to make. If I had to fork the project, I'll be honest - Windows stuff would go right out the window immediately. No offense to Windows users, and I wish NPM was able to show stats on breakdown of which OS users are using, but I feel like that packages like this where its probably 5-10% of the user base is using something that is disrupting the flow/progress for +80%, that maybe its a hard decision that needs to be made where Windows support is dropped and its up to that userbase to take a fork and run with it if they so choose. Anywho, just a thought - I'll checkout the Slack invite and thanks for all the efforts!

[AndrewFarley]

The idea of gutting the Windows support from this module is quite appealing in some ways. It is some of the least covered code as far as testing goes, and that's because it's core developers and maintainers (or, whats left of us/them) don't use Windows at all. So it's no surprise to me that in the past randomly after releases were cut some issues popped up on Windows. I found it painful to navigate around the Windows support in this plugin while developing as well. I recall my larger PR took me months of extra work and rework to navigate the code to work properly on Windows, and only on windows, it worked fantastically on Linux/OS-X (as that's where I live/work/play).

Lets do this as a thought experiment @bsamuel-ui / @bryantbiggs what it would/might look like...

• To start things off we probably want to get some usage metrics from folks about what OS and what features they use of this plugin. We can either...
  • Publish a new version of this plugin that gathers anonymous information about which features are being used and on what OS, etc. Send it off to something free/easy/cheap (eg: google forms/sheets, or even AWS APIG/Lambda/DynamoDB which would still cost like nothing)
  • Or... add some voting into the issue-tracking to gather information and try to ask people (maybe add some printed stuff into a newly published version to ask people to go vote and tell us)

Both of these methods have flaws, some people will gripe about security/privacy if we do the former, and if we do the latter we'll likely have very biased and missing information. If we do gathering anonymous info, we could upon install and use we can print out and warn people right up front of what we're doing in order to help inform us of how best to support and maintain the plugin. People can opt-out simply with an ENV var, or a config option which we state very clearly also in the README and in the docs, and in the npm repo registry.

Moving forward from the data-gathering, depending on the information gathered we can do things such as...

• If Windows proves to be below a threshold (eg: 5 or 10%) then we can consider forking this repo to another named serverless-python-requirements-windows and do a call-out for maintainers of a windows fork. In that repo, they/we can strip Linux support from it completely and add CI testing focused on the various cases and conditions of windows and the various kinds of docker on windows. This would greatly simplify and streamline and shorten the development and testing cycles having a simpler codebase and less tests to run through and validate code from.
• This repository could then go through a major release removing Windows support entirely, removing the tests and the code for it and focusing on it working on unix-ey machines. This would similar as above, simplify the code, reduce the time to run tests, and allow us to focus on and deliver smaller/quicker iterations from focused functional tests.
• Based on the data-gathering after a few months months we could go through features below a threshold and consider/discuss removal.
• Finally, there's a interesting implementation nuances to this to discuss though. If we did fork/split, there's not an super easy way to conditionally install plugins in Serverless. I'm sure the general public would appreciate a way to ensure that a codebase could support Linux and Windows based development. There is some trickery in that serverless supports embedding Javascript. I've done in the past which would look something similar to this.

serverless.yaml:
custom:
  pythonRequirements: ${file(./build_helpers.js):get_python_reqs_config}
...

---

build_helpers.js:
var os = require("os")

// This gets the Python requirements configuration dependant on OS
module.exports.get_python_reqs_config = function() {
  if (os.platform() != 'linux')
    return Promise.resolve({invalidateCaches: false, useStaticCache: true, useDownloadCache: true, dockerizePip: true, dockerImage: "andrewfarley/serverless-personal-building-with-mysql"});
  return Promise.resolve({invalidateCaches: false});

I don't know how ideal, or comfortable that is. And I haven't tested if this embedding javascript works for plugins, would require some validation. Likely in the package.json you would have to install both, but only one gets used depending on what is in plugins:. If this method works, we could just have an working example demonstrating how it's still possible to allow devs any environment they want for development with a single codebase and single serverless.yaml config file.

How does any/all of that sound? Is it feasible to some of you/us?

I'd personally +1 the idea to remove Windows support and focus on Linux native and Docker-based support. Even without doing data-gathering, but that might be valuable as well to ensure we don't alienate a huge percentage of our userbase without being informed at least at first.

[software-2]

Windows user chiming in here to anecdotally show there's indeed much interest in continuing to use this project with Windows development environments. My teams have a mix of Linux/Mac/Windows users, and it would be a big downside if we had to worry about project fragmentation and hoping a fork is in sync with the main project, especially when the main project has threads questioning the life of the project.

[dschep]

Tagging @heri16 and @kichik as they're the contributors listed in the README for Windows support.

[bryantbiggs]

@software-2 I think that brings up the crux of all of this - the project is sort of in a weird cycle (if I understand correctly) at the moment. The core group (OG and new) that looks after this project, they do not work on Windows machines and therefore do not dabble in the Windows portion of the codebase. However, significant effort is required to make advancements/refactors/changes due to needing to support Windows. Its a tough spot, nobody is trying to screw anyone over, but at the same time, development on the project is stalling (presumably) because of this support for Windows. Personally it doesn't make sense to me - lambda environments are *nix based (or *nix-likeish), the containers used in this project to bundle/package the artifacts only work using *nix constructs (which is where WSL comes in), etc. Again, its a tough decision but it seems either a major release is cut where Windows support is dropped and the project is able to go ahead and flourish, or it stays on its current course which does not look promising unfortunately. If there is indeed a large Windows user base, perhaps they will come forward and we can figure out a game plan

[bsamuel-ui]

Publish a new version of this plugin that gathers anonymous information about which features are being used and on what OS, etc.

Now that the project is sponsored by Capital One, we have strict privacy requirements around gathering information. It's a good deal for our customers, but it makes that hard.

I checked some of the various npm download stats sites, none of them break out downloads per OS, so I'm guessing npm doesn't track that.

Windows is definitely contributing to making the tests hard to work with, but I'd like to see if there are easier ways to unblock this.

Is there any reason to maintain Python 2 support? If not, a next step is to spin off a new major version dropping support for that entirely.

[dschep]

Is there any reason to maintain Python 2 support? If not, a next step is to spin off a new major version dropping support for that entirely.

Agreed. Nuke Python 2. It's officially EOL, and AWS Lambda EOL is exactly 2 months from now according to this:
https://aws.amazon.com/blogs/compute/continued-support-for-python-2-7-on-aws-lambda/

[miketheman]

Progress here! #603

[bsamuel-ui]

Version 5.1.1 is now out, so this beautiful Norweigan blue has a bit of life left in him.

[xavivars]

I'm one of the ones using this plug-in on Windows. My team is split between macOS & Windows dev environments. That said, if windows support needs to be dropped in order to move forward to make this plug-in 2.x compatible, IMHO it's worth the change (as long as there are workarounds like WSL).

Keeping the plug-in alive and well maintained for a subset of the OSes is better than have it "almost-dead" for all of them.

[CalMlynarczyk]

Hi all, thank you for your concern for this project. If you have not seen already, the team at United Income is seeking interested parties to take ownership of this project. We do not want anyone to think this project is dead, but we have not been able to provide the resources to support it that we once were able to. If anyone is interested in taking ownership, please reach out to the email addresses listed in that post.

[CalMlynarczyk]

We wanted to let everyone know that we have found a new owner to take over this project and are working with our internal partners at Capital One to approve and execute the project transfer.

[bryantbiggs]

hey all - I decided to take the step of forking and starting to update separately in order to get some of the features/fixes in that we are trying to utilize (python3.9 support being one of them). you can follow along here if interested https://github.com/clowdhaus/serverless-python-requirements

some of the notable changes that I think we've discussed here and in other threads:

• dropped python 2.7 and 3.6 support due to EOL
• added supported for python 3.9
• switched container images used to use AWS SAM build official containers (lambci seems to be falling behind and they don't support 3.9 yet)
• fixed CI to test using docker on MacOs
• removed Windows support from the codebase (might be more in the codebase, but will continue digging through)

Hopefully at some point we can switch back to this mainline but for now, this will due. also, let me know if I can help out in anyway

[pgrzesik]

Hello @bryantbiggs - is anything preventing you from continuing work on this original repo? We (me & @medikoo) are currently in the process of transfering this project to serverless organization as we want to help out with development and ensure that this plugin is in good place.

[bryantbiggs]

hey @pgrzesik - thats a tricky question but I'll try my best

Getting changes in has been slow, most likely due to this transition (totally understandable). But more importantly, I am not a Windows user and trying to get these changes through and ensure it works on Windows and all the edge cases that may come along with it would be a daunting task. So I think it really came down to the current speed of changes with this project and getting to the current outcome we need today and weighing that against some of the larger changes that have been previously discussed but have lacked any movement:

• Remove Python 2.7 support #603
• Is this project dead now? #599 (comment)

But having said all that, as I said previously - my desire is to use this community module and not have to maintain a diverged copy. So depending on how we can move on some of these larger, more controversial changes - maybe we can get things rolling back here again. Again - I don't want to have a forked copy but it seemed like the fastest/smoothest/lowest effort route to get our teams to the things we are looking for today

[pgrzesik]

Hey @bryantbiggs, thanks a lot for sharing your perspective 🙇

Getting changes in has been slow, most likely due to this transition

Definitely that has been a big pain point - we only started the transition a few weeks back and at this point we still don't have full access, but we're making further steps to take care of this plugin. We hope that we will be able to greatly improve pace of development/accepting changes for this plugin.

But more importantly, I am not a Windows user and trying to get these changes through and ensure it works on Windows and all the edge cases that may come along with it would be a daunting task

That's totally understandable. From our perspective, dropping Windows support is currently not an option as we want to continue supporting people on all platforms (even though I know how challenging Python dependency management on Windows can be 😬 ). I'm hoping that we will be able to get inspiration from your changes that are Linux/MacOS specific to improve experience for all OSes.

As for the changes in your fork - do you have any open PRs/bigger changes that are being addressed there? I'd love to take a look if possible. 🙇

[miketheman]

Looks like the repo has now been transferred to Serverless Inc, so that resolves some parts of this discussion! 🎉