Allow annotations in connections

[mmourafiq]

Use case

We would like to use Vault for managing some connection's secrets.

EDIT: Different use case where node scheduling can be also linked directly to a connection:

When I need to iterate multiple times over the input data I would like to copy the data from the network storage to the SSD of the experiment machine as first step in my polyaxon file and then use that local copy to iterate on.

In this case we might want to always schedule a connection with a specific node or nodes(s).

Feature description

1. Allow annotations in connections.
2. Allow node scheduling in connections.

Alternatives

I think these use-cases can be manged right now using RunProfile.

N.B. the second issue was raised by a CE user, where the RunProfile is not accessible.

Considerations

If we provide annotations/node-scheduling and other meta-data in connections we need to think about:

• The overlap between the RunProfile and Connection concepts. RunProfile concept still handles more logic, but we should not duplicate efforts
• If both concepts provide overlapping logic it can result in unexpected behaviour.