Add a Task wrapper for MicroSoft OpenPAI

[liudongqing]

Description

Microsoft OpenPAI Job wrapper for Luigi.
Add luigi/contrib/pai.py
Add test/contrib/pai_test.py

Motivation and Context

Microsoft OpenPAI Job wrapper for Luigi.

"OpenPAI is an open source platform that provides complete AI model training and resource management capabilities,
it is easy to extend and supports on-premise, cloud and hybrid environments in various scale."

For more information about OpenPAI : https://github.com/Microsoft/pai/

Have you tested this? If so, how?

Tested in our local open pai cluster.

[dlstadther]

Thanks for submitting @liudongqing

I have a number of comments to make, but before i go into detail i wanted to provide some high level thoughts and questions.

1. Could you explain the __slots__ usage and __slot_to_dict()?
2. When a class property is required to be overriden, decorate it with @abc.abstractproperty.
3. You have a mix of camelCase and snake_case instance variables. Can we stick to just one? Preferably, snake_case.
4. On multiple occasions, you reference an element of a Request response before verifying that it returned with a valid HTTP code such to contain text or json
5. Similarly, verify that the contents include what you expect. (Looking at the token instantiation here - will that POST response always include a token key in the response?)

[liudongqing]

@dlstadther Thanks for your comments, see my response below:

1. The PaiJob/TaskRole/Port represent the OpenPAI Job structure, use slots to make sure no attributes other than predefined ones. __slot_to_dict() used to json.dump as default function serialize the PaiJob object.
2. Done.
3. The camelCase was used in he PaiJob/TaskRole/Port to make sure the attribute name are same as OpenPAI job. Other than that, I use snake_case to follow python convention.
4. Added some checkings.
5. It should always have the same attributes when the response status_code are 200.

[liudongqing]

@dlstadther any new comments?

[dlstadther]

Sorry @liudongqing I'll try to re-review this this week. Thanks for following up!

[liudongqing]

@dlstadther Thanks for your comments, sorry for delayed response due to a long vacation.

[liudongqing]

@dlstadther , could you please help to follow? Thanks!

[dlstadther]

Mostly just some visual cleanup.

But do still have hesitations about the unittest.

[liudongqing]

The unittests failed, looks like requests lib got an issued Urllib3 1.24. I checked the master branch, it failed as well.

[liudongqing]

@dlstadther passed UT by adding the requests>=2.20.0 per comment in the issue.

[dlstadther]

Thanks so much for your prompt fixes. I believe this is the last thing from me! :)

[dlstadther]

I think this LGTM now

[liudongqing]

@dlstadther Thanks! Any more actions I need to take to merge that or just waiting someone to do it at convenience time?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.

[dlstadther]

@liudongqing I'm so sorry this fell off my radar! I noted this was ready to merge and i meant to do so. Could you update the merge conflict so that i can merge?

[dlstadther]

Thanks so much @liudongqing !