Support for maxSessions in multiCapabilities with shardTestFiles

[smhc]

The shardTestFiles option is a useful feature, however it can cause test failures when running specs that can't run in parallel against a common server. This may be bad practice for the tests, nonetheless the tests run fine when run sequentially.

We still have plenty of 'read only' specs that can run in parallel however. What we'd like to do is run a set of 'read only' tests in parallel against the server then a set of 'read write' tests sequentially, but not have the 'read only' and 'read write' tests running at the same time.

For example it might be implemented by adding a 'maxSessions' per capability, something like (commented out pseudo options):

    multiCapabilities: [
        {
            shardTestFiles: true,
            maxInstances: 4,
            //maxSessions: -1,
            //parallelCapability: true
            browserName: chrome,
            specs: ['readonly/*.spec.js']
        },
        {
            shardTestFiles: false,
            maxInstances: 1,
            //maxSessions: 1,
            //sequential: true
            //parallelCapability: false
            browserName: chrome,
            specs: ['readwrite/*.spec.js']
        }
    ],

It can be worked around by having two protractor.conf.js files and running them separately and sequentially, but it's not ideal.

[hvdb]

looks like you run your tests against a server which actually performs te read/write actions.

Best case is to use mockhttpbackend to give the correct response.
That way you always know what the response would be and you don't have interference with other tests.

However if you want this, you can use the setup like above? maybe set a version for the chrome so that you can distinguish them.

[smhc]

The problem with the above setup is it will run the readonly and readwrite tests concurrently even though maxInstances=1 for the readwrite tests.

Hence the request for an enhancement to flag specific capability entries to not run in parallel with any other compatibility entry.

[hvdb]

ahhh now i understand your problem!

It is not the problem of sharding the test files.
it will not shard the test files but the first and second capability is executed directly.
Hum don't know if that is really what you want, the power of the multicapabilities is that they run paralel, resulting in faster tests.
I think best solution is to have a second conf and then execute them after each other.
But the boolean seems like an approach to support this scenario.

However it is a fix to a problem that shouldn't be there, tests should be able to run independent. (in my opinion) and you shouldn't rely on a real response.

[juliemr]

I think that this is a case of 'split it into two conf files', as you said. I think that splitting tests is complex enough as is, and adding something like maxSessions per capability would just make it even more difficult to decide on the correct way to set up your config.

[smhc]

I agree a maxSessions per capability wouldn't be the right approach. But what about simply toggling running capabilities concurrently or not?

Concurrency can be enabled/disabled per capability (rather than globally) via sharding. It therefore seems reasonable and useful to be able to turn off concurrent capabilities, especially if sharding is false for some capabilities. (i.e someone using sharding=false doesn't want concurrency for some reason)

See below for an example:

    multiCapabilities: [
        {
            shardTestFiles: true,
            maxInstances: 4,
            browserName: chrome,
            specs: ['readonly/*.spec.js']
        },
        {
            shardTestFiles: false,
            maxInstances: 1,
            browserName: chrome,
            specs: ['readwrite/*.spec.js']
        }
    ],
    //proposed:
    //concurrentMultiCapabilities: false