Zulip Chat Archive

Stream: general

Topic: github outages


Anne Baanen (Jun 08 2021 at 09:54):

Are the mathlib docs are down for everyone or just me? Error 503 at https://leanprover-community.github.io/mathlib_docs/

Oliver Nash (Jun 08 2021 at 09:55):

Not just you.

Marc Huisinga (Jun 08 2021 at 09:56):

Works for me apparently

Gabriel Ebner (Jun 08 2021 at 09:57):

Apparently fastly has an outage: https://www.fastly.com/

Julian Berman (Jun 08 2021 at 09:59):

Doesn't look like it's on status.github.com yet, but the first reporting tweet from 3 minutes ago from someone just says: GitHub pagesしんでる.

Eric Rodriguez (Jun 08 2021 at 10:03):

yeah, a bunch of websites are down

Julian Berman (Jun 08 2021 at 10:04):

https://status.fastly.com/incidents/vpk0ssybt3bj

Eric Wieser (Jun 08 2021 at 10:07):

~~This has caused a linter PR to fail! https://github.com/leanprover-community/mathlib/pull/7838~~

Gabriel Ebner (Jun 08 2021 at 10:09):

The log shows an unrelated error though (unrelated to the outage I mean):

2021-06-08T06:28:07.8158691Z ##[error]/home/runner/work/mathlib/mathlib/src/data/mv_polynomial/equiv.lean:270:38:
(deterministic) timeout
2021-06-08T06:28:08.5861536Z ##[error]Process completed with exit code 1.

EDIT: it's also from four hours before the outage?

Scott Morrison (Jun 08 2021 at 10:10):

Half the internet is down.

Eric Rodriguez (Jun 08 2021 at 10:10):

thankfully olean storage isn't :b

Johan Commelin (Jun 08 2021 at 10:15):

Scott Morrison said:

Half the internet is down.

What a rotten place. :stuck_out_tongue_wink:

Gabriel Ebner (Jun 08 2021 at 10:35):

Eric Wieser said:

This has caused a linter PR to fail!

We now have several PRs whose failure is due to the outage. For example: https://github.com/leanprover-community/mathlib/runs/2772873418?check_suite_focus=true (the "fancy" log display is also affected by the outage, you have to view the raw logs)
The reason is that elan can no longer be installed.

Eric Rodriguez (Jun 08 2021 at 10:39):

poor fastly people are probably having a terrible day

Eric Wieser (Jun 08 2021 at 10:43):

The "This incident affects:" list on the fastly status page has grown substantially since I last looked.

Julian Berman (Jun 08 2021 at 10:45):

I wonder whether they're continuing to investigate the issue.

Gabriel Ebner (Jun 08 2021 at 10:45):

The github profile pictures are back online! (partially)

Julian Berman (Jun 08 2021 at 10:45):

Identified - The issue has been identified and a fix is being implemented.

Eric Rodriguez (Jun 08 2021 at 10:50):

restarted all the actions!

Gabriel Ebner (Jun 08 2021 at 10:53):

Fastly is still down.

Eric Rodriguez (Jun 08 2021 at 10:57):

they seem to have some selective websites up, like GH and SO

Anne Baanen (Jun 08 2021 at 11:35):

The issue has been identified and a fix has been applied. :octopus:

Floris van Doorn (Jun 08 2021 at 15:23):

bors seems to be still down. I cannot access the dashboard, and the CI for the staging branch succeeded an hour ago...

Damiano Testa (Jun 08 2021 at 15:24):

I do not know if this is related, but PR #7645 built on my computer, but timed out on a circumcenter file that should not have been affected by the changes in the PR.

Floris van Doorn (Jun 08 2021 at 15:26):

Yeah, that looks like it happened during the outage.

Bryan Gin-ge Chen (Jun 08 2021 at 15:54):

Looks like there's a separate issue affecting bors? https://github.com/bors-ng/bors-ng/issues/1263

Sebastien Gouezel (Jun 08 2021 at 15:59):

Damiano Testa said:

I do not know if this is related, but PR #7645 built on my computer, but timed out on a circumcenter file that should not have been affected by the changes in the PR.

Normally this should have been fixed by #7840. Can you try merging master to see if it fixes your problem?

Damiano Testa (Jun 08 2021 at 16:03):

I tried merging master after the timeout, now CI is building, but it normally takes several hours to finish, so I may only know the outcome in a while!

Floris van Doorn (Jun 08 2021 at 16:25):

Damiano Testa said:

I tried merging master after the timeout, now CI is building, but it normally takes several hours to finish, so I may only know the outcome in a while!

If you don't make any changes in low-level files, CI is much quicker, since it will download all the previously compiled files, even if the previous run failed.

Damiano Testa (Jun 08 2021 at 20:10):

An update: the PR that timed out on the circumcenter PR now successfully built! The other one that was queued to be merged failed, but I think that bors will try it again independently.

Bryan Gin-ge Chen (Aug 05 2021 at 14:30):

Looks like GitHub Actions is having trouble again: https://www.githubstatus.com/incidents/7p1nnvkgh96y

Probably the failure of #8527 was caused by this.

Yaël Dillies (Aug 06 2021 at 13:23):

They say it's resolved, but #8508's checks have been waiting in the queue for a day now. And I've rerun all jobs...

Eric Wieser (Aug 06 2021 at 13:26):

Did you rerun all jobs under all workflows? Or just under one workflow?

Eric Wieser (Aug 06 2021 at 13:27):

I just restarted your build job and it seems fine - I can only assume you restarted some jobs that were already working just fine

Yaël Dillies (Aug 06 2021 at 13:29):

Oh, maybe I've misunderstood the button.

Bryan Gin-ge Chen (Oct 05 2021 at 14:44):

I noticed that CI was not triggered on some commits (e.g. this one from #9528). It looks like GitHub actions is having issues again: https://www.githubstatus.com/incidents/bdbzpz7qxmbx

Eric Rodriguez (Mar 17 2022 at 14:43):

just to note there's another one right now ^^

Pierre-Alexandre Bazin (Mar 17 2022 at 15:27):

$ git push --set-upstream origin ufd_dvd_prime_power
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 1.50 KiB | 383.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
remote: Internal Server Error
To https://github.com/leanprover-community/mathlib.git
 ! [remote rejected]     ufd_dvd_prime_power -> ufd_dvd_prime_power (Internal Server Error)
error: failed to push some refs to 'https://github.com/leanprover-community/mathlib.git'

ah that's probably why I got this error when trying to push a new branch

Bryan Gin-ge Chen (Nov 27 2023 at 20:35):

Just a heads up: GitHub is having issues now so bors (among other things) might not work as expected.

Eric Rodriguez (Nov 27 2023 at 22:38):

It seems to have been resolved by now.


Last updated: Dec 20 2023 at 11:08 UTC