Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hosted macOS workflows will experience longer wait times the week of December 14th. #2247

Closed
4 of 9 tasks
thejoebourneidentity opened this issue Dec 11, 2020 · 26 comments
Closed
4 of 9 tasks

Comments

@thejoebourneidentity
Copy link
Contributor

thejoebourneidentity commented Dec 11, 2020

Breaking changes
GitHub's macOS datacenter is performing scheduled maintenance December 14-18. During this time customers may experience longer than normal wait times for their macOS workflows to begin.

Target date
December 14-18

Possible impact
Jobs using any of the macOS virtual environments may experience longer than normal queue times. In some cases it's possible workflows will time out, be cancelled, and need to be restarted.

Virtual environments affected

  • Ubuntu 16.04
  • Ubuntu 18.04
  • Ubuntu 20.04
  • macOS 10.13
  • macOS 10.14
  • macOS 10.15
  • macOS 11.0
  • Windows Server 2016 R2
  • Windows Server 2019
@Rashidium
Copy link

Good luck on your work. I wonder, if there will be any discounts on macOS hosted github-actions in this time interval?

@styfle
Copy link

styfle commented Dec 12, 2020

Does this maintenance include fixing the network issues with macOS?

@dscho
Copy link
Contributor

dscho commented Dec 14, 2020

I wonder, if there will be any discounts on macOS hosted github-actions in this time interval?

The problem is that there is no way to re-run individual jobs in GitHub Actions so far.

EDIT: This means that discounts only on macOS-hosted GitHub Actions would not remediate the problem because the only way to re-run those jobs is to re-run the entire workflow, incurring additional cost for jobs that run on non-macOS platforms (because they will have to be re-run, even if they had already run, and succeeded).

@dleff78
Copy link

dleff78 commented Dec 15, 2020

Volleyball geil

@ghost
Copy link

ghost commented Dec 15, 2020

Thank you

@natmaxwell83
Copy link

Nice

@daniilshumkoidt
Copy link

Quite bad timing knowing that App Store Connect is going on holiday next week and everyone rushes to release their apps.

@FormerlyChucks
Copy link

Thanks for the warning!

@TPXP
Copy link

TPXP commented Dec 16, 2020

I'm also facing network issues when installing yarn packages on a MacOS runner. I get error messages similar to the ones shown by @styfle , has this issue been identified and will the maintenance try to fix it, or should I raise it to the Github support?

@Brooooooklyn
Copy link

Does this maintenance include fixing the network issues with macOS?

@styfle actually it's a bug from yarn. yarnpkg/yarn#8242

The reason for this bug is that the macOS host disk is slow, and extra large number of files from npm package such as rxjs would trigger this bug in yarn.

Maybe you have already observed that this issue always happened on rxjs @material/icons etc. It couldn't be so accident that network always failure on downloading these packages.

@styfle
Copy link

styfle commented Dec 17, 2020

@Brooooooklyn Its not just yarn that fails, all network requests fail eventually.

Even spawning a server on localhost will sometimes fail with ECONNREFUSED.

@nodeselector
Copy link

@styfle can you provide repro steps? I want to try and chase that down.

@styfle
Copy link

styfle commented Dec 17, 2020

@nodeselector Thanks! The yarn issue should be easy to reproduce because it fails early.

Create a repo with this package.json and yarn.lock in the root.

And then create a workflow that runs yarn install.

on:
  push:
    branches:
    - master
    tags:
    - '!*'
  pull_request:

jobs:
  test:
    timeout-minutes: 15
    strategy:
      fail-fast: false
      matrix:
        os: [macos-latest]
        node: [10, 12]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 100
      - uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node }}
      - run: yarn install

@jonboyandlayla
Copy link

Awesome indeed.

@thejoebourneidentity
Copy link
Contributor Author

The maintenance for this week is now complete! If there are continued issues, please let us know by creating new issues because they will be unrelated to this maintenance cycle. Thanks for your patience everyone!

@bohiendq
Copy link

HiengoKu

@ginahen2018
Copy link

ginahen2018 commented Dec 21, 2020 via email

@ginahen2018
Copy link

ginahen2018 commented Dec 21, 2020 via email

@richiksc
Copy link

richiksc commented Jan 3, 2021

@thejoebourneidentity It's Jan 2, and wait times on macOS 11.0 runners still seem to be very long. Is this because of a general scarcity of Big Sur runners or is there ongoing maintenance on those?

@tonyarnold
Copy link

I’m seeing over an hour wait in the queue for my workflows on a paid, private repository.

@richiksc
Copy link

richiksc commented Jan 3, 2021

I’m seeing over an hour wait in the queue for my workflows on a paid, private repository.

My workflow with two jobs on a PR now shows three checks, two passing and one failing. One of them is the original Ubuntu job, which started instantly. The failing one is the original macOS job, which sat in the queue for almost 2 hours until it was cancelled, and now reports as a failed check on the PR, making it unclear for maintainers. The other passing check is the automatically restarted macOS job, which completed two hours after the original Ubuntu job completed, which was soon after the PR push.

This is a public repository, however.

@nakkht
Copy link

nakkht commented Jan 3, 2021

@richiksc @tonyarnold
#2247 (comment)

@tonyarnold refer to created issue #2381
I also do experience long waiting queues for macOS runners - both private/public repositories.

@tonyarnold
Copy link

Could someone from GitHub please update this issue and let us know what's happening with the extremely long queue times for macOS 11 runners? I'm at a bit of a standstill with basic CI taking over an hour every time it's run.

@KeiroMidori
Copy link

Same issue here, queue times are really long on mac-os hosted machines. (around 1h to start the workflow when it doesn't "check fail")

Can we get an ETA on a resolution for this?

franckrasolo added a commit to franckrasolo/slice.rs that referenced this issue Oct 10, 2021
@RaminDadou
Copy link

Migraines

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

32 participants