include ca file via certifi module.

[topic2k]

On startup (max once a day) check for new certifi release and let the user decide to autoupdate (restart required).
The updated module will be placed in %ProgramData%/EventGhost/lib/site-packages. That path is added as first path in search order for imports.
An environment variable REQUESTS_CA_BUNDLE will be set to the cacert.pem file.
The requests package uses the certifi package to find the certs file.
To use it with pycurl, do it like this:

import pycurl
import certifi

curl = pycurl.Curl()
curl.setopt(pycurl.CAINFO, certifi.where())
# ...

• add certifi to lib27/site-packages
• update pip to 10.0.1 (9.x includes a file with invalid name for import [pip/_vendor/re-vendor.py])
• disable pip version check
• remove requests from site-packages and instead install with pip
• remove (py)curl from lib27/site-packages an get it from pip
• add pip, pycurl and requests to includeModules in Build.py
• add %ProgramData%/EventGhost/lib/site-packages as first search path for modules

[codeclimate]

The PR diff size of 21460 lines exceeds the maximum allowed for the inline comments feature.

[codeclimate]

Code Climate has analyzed commit c48a95b and detected 0 issues on this pull request.

View more on Code Climate.

[AppVeyorBot]

WIP-2019.02.08-11.17.30 completed (commit 894b9b793f)
Artifacts:
Build.log
CHANGELOG.md
CHANGELOG_THIS_RELEASE.bb
CHANGELOG_THIS_RELEASE.md
EventGhost_WIP-2019.02.08-11.17.30_Setup.exe

[kdschlosser]

I already did up a whole code segment for this. i just haven't done the PR yet.. for a specific reason.

and that reason has to do with pycurl actually.

here is what I did up.
https://github.com/kdschlosser/EventGhost/tree/updated_build

the reason i did not specifically do a PR yet is because of pycurl. If pycurl is compiled and the environment variable CURL_CA_BUNDLE is pointing to the cert file then the cert file gets built into pycurl and there is no need to do

curl = pycurl.Curl()
curl.setopt(pycurl.CAINFO, certifi.where())

what I was going to do is setup a build system on appveyor to install and build pycurl with the most current cert file. and that can be then placed on the EG server.. when the person clicks the button to download the new cert file. it will download the cert file and also download the pycurl that has the cert file built into it.

I also removed that antique version of requests and all other modules from site-packages. I did this for reasons like this one.

a very major bug in the requests module that was not fixed until recently and that bug was that requests.Session would not use the environment variable for setting the cert path.

[kdschlosser]

I did also want to ask a question.. what is the purpose for adding pip to the included modules?

I am asking this because there is a whole nested mess of things that have to be changed to get pip to run when python has been all packaged up into an executable.

[topic2k]

the reason i did not specifically do a PR yet is because of pycurl. If pycurl is compiled and the environment variable CURL_CA_BUNDLE is pointing to the cert file then the cert file gets built into pycurl and there is no need to do

What Ddo you mean with 'compiled' and 'built into'? Isn't it just package to install?

and that can be then placed on the EG server..

but then you have to keep that file updated. I'm in favour of using pypi.org.

a very major bug in the requests module that was not fixed until recently and that bug was that requests.Session would not use the environment variable for setting the cert path.

Maybe you use an old requests version? I did a test and it uses either os.environ['REQUESTS_CA_BUNDLE'] or os.environ['CURL_CA_BUNDLE'] and if both not set, as default uses certifi.where(). (requests docu: CA Certificates)

I did also want to ask a question.. what is the purpose for adding pip to the included modules?

I'm using pip for downloading and installing the certifi package. Sure, it could also be done with requests or pycurl, but i had in mind that in the future it could also be used for automated package downloading in case a plugin need a package we don't ship.

I am asking this because there is a whole nested mess of things that have to be changed to get pip to run when python has been all packaged up into an executable.

Hmm, at least for the current use case it's working. Don't know which 'mess of things' you mean.

[kdschlosser]

I know there were all kinds of problems getting pip to function properly in eventghost.
pip does us setuptools did you also add that into EG?

theere are still issues with the requests library working correctly with the REQUESTS_CA_BUNDLE variable set. go look on the requests GitHub repo you will find quite a few issues still open. But you have moved it to a requirement so we should be good to go there. did you set up a dependency in CheckDependncies for it?

and pyCurl is a python wrapper for libcurl. libcurl gets compiled for the OS it is running on. They may have prebuilt wheels they are distributing I have not checked.

[kdschlosser]

try using pip for more then just certifi running it from say a a python script.

the other thing is if we allow package to be installed via code without some kind of program to manage what is going on one of the things that can happens is this. if a plugin install a package that is versioned 2.0 and another plugin then installs/upgrades the same package to 3.0 this has to potential of breaking the first plugin. another is what happens when the user uninstalls the plugin? where do the libraries get stored? you are setting it for this one use case. but unless it is handled by us who knows where the library get stashed.

[kdschlosser]

I am all for adding pip to EventGhost.

but it does need to be managed. and it has to keep track of plugins that are in use and what versions of the libraries are being used by the different plugins. we are also going to have to provide some kind of a mechanism to properly handle extensions getting built on Windows, this is what setuptools is used for in pip. and the setuptools has a really handicapped MSVC location system.

This will no doubt open the flood gates for people reporting bugs for libraries not installing properly.