Zulip Chat Archive

Stream: CI

Topic: CI runner status

Scott Morrison (Aug 26 2021 at 01:58):

Just announcing there will be some noise on this stream while I test a script that is checking whether the github CI runners are online.

build time bot (Aug 26 2021 at 02:00):

build time bot (Aug 26 2021 at 02:03):

lane.vm.uni-freiburg.de appears to be offline

build time bot (Aug 26 2021 at 02:04):

lane.vm.uni-freiburg.de is online

build time bot (Aug 26 2021 at 02:11):

test from idaeus

build time bot (Aug 26 2021 at 02:21):

lane.vm.uni-freiburg.de appears to be offline

build time bot (Aug 26 2021 at 02:45):

lane.vm.uni-freiburg.de is online

Scott Morrison (Aug 26 2021 at 03:11):

Okay, testing over for now, this appears to work. @Johan Commelin, would you mind taking one offline later and waiting (<=5 minutes) to see if the bot notices?

Johan Commelin (Aug 26 2021 at 04:48):

@Scott Morrison I'll do that later today.

Minor nitpick: Is it possible to rename the bot easily? Because it's more like "CI runner bot" or "uptime bot" instead of "build time bot".

Johan Commelin (Aug 26 2021 at 09:19):

nael seemed to have a problem, so I rebooted it. (This is the VM that handles the bors queue, so we might have to rebors some PRs.)

Johan Commelin (Aug 26 2021 at 09:20):

I've also switched off lena. I'll wait 5 minutes before turning it on again.

CI runner status bot (Aug 26 2021 at 09:23):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

Johan Commelin (Aug 26 2021 at 09:27):

Great! That seems to work. I'm restarting lena.

CI runner status bot (Aug 26 2021 at 09:30):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

Scott Morrison (Aug 26 2021 at 09:56):

It seems the @ didn't actually produce notifications, however.

Scott Morrison (Aug 26 2021 at 09:57):

What if I do it? @github-runners-maintainers.

Gabriel Ebner (Aug 26 2021 at 09:57):

I've got all three notifications so far.

Johan Commelin (Aug 26 2021 at 09:58):

I also got them.

Johan Commelin (Aug 26 2021 at 09:58):

@Johan Commelin do I get notified if I do this?

Johan Commelin (Aug 26 2021 at 09:58):

Answer: yes

CI runner status bot (Aug 26 2021 at 23:24):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

Bryan Gin-ge Chen (Aug 26 2021 at 23:30):

I'm not able to ssh into lena. Here's what ssh -v outputs (with some paths modified):

OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data ~/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: /etc/ssh/ssh_config line 52: Applying options for *
debug1: Connecting to lena.vm.uni-freiburg.de port 22.
debug1: Connection established.
debug1: identity file ~/.ssh/id_rsa type -1
debug1: identity file ~/.ssh/id_rsa-cert type -1
debug1: identity file ~/.ssh/id_dsa type -1
debug1: identity file ~/.ssh/id_dsa-cert type -1
debug1: identity file ~/.ssh/id_ecdsa type -1
debug1: identity file ~/.ssh/id_ecdsa-cert type -1
debug1: identity file ~/.ssh/id_ed25519 type -1
debug1: identity file ~/.ssh/id_ed25519-cert type -1
debug1: identity file ~/.ssh/id_xmss type -1
debug1: identity file ~/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9

Then I'm just left waiting without getting a shell.

CI runner status bot (Aug 26 2021 at 23:39):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Aug 26 2021 at 23:39):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Aug 26 2021 at 23:40):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

Johan Commelin (Aug 27 2021 at 05:06):

Flakey lena...

CI runner status bot (Aug 30 2021 at 12:59):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Aug 30 2021 at 12:59):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

Johan Commelin (Aug 30 2021 at 13:00):

Wow, lena again... and she's fast!

Scott Morrison (Aug 30 2021 at 13:02):

Perhaps I should the number of times the scripts retries before it reports something is offline. Currently it tries twice, with a 60 second sleep in between.

Scott Morrison (Aug 30 2021 at 13:02):

"increase"

Johan Commelin (Aug 30 2021 at 13:03):

Hmm, intuitively, 60s sounds reasonable to me.

Johan Commelin (Aug 30 2021 at 13:03):

If you can't pong a ping within 60s, you're a lousy bot, right?

CI runner status bot (Sep 01 2021 at 00:23):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 01 2021 at 00:24):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

Johan Commelin (Sep 01 2021 at 05:04):

Seriously lena, what is wrong with you? Your siblings are never misbehaving, always quietly doing their jobs. You're always acting up like a rebellious teenager. It's time to grow up, don't you think?

Mario Carneiro (Sep 01 2021 at 05:11):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be sulking

Scott Morrison (Sep 01 2021 at 05:35):

Perhaps testing ssh connectivity is not the issue. I just asked it to reboot.

Bryan Gin-ge Chen (Sep 03 2021 at 18:43):

lane and lena were offline just now; I've just restarted them. @Scott Morrison did the status bot go down or did I just happen to notice before the bot did?

Scott Morrison (Sep 03 2021 at 19:28):

Hmm, looks like it was running. Let me improve the local logging later, so it records timestamps of successful connections.

CI runner status bot (Sep 05 2021 at 15:29):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 05 2021 at 15:33):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Sep 05 2021 at 22:16):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 05 2021 at 22:20):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Sep 07 2021 at 00:38):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 07 2021 at 00:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

Bryan Gin-ge Chen (Sep 07 2021 at 01:00):

Looks like this blip caused the latest staging build to fail: https://github.com/leanprover-community/mathlib/runs/3528554965

Anne Baanen (Sep 07 2021 at 12:18):

Are the runners OK? According to GitHub, only three commits are in progress, and 26 are queued, some for 2 hours.

Bryan Gin-ge Chen (Sep 07 2021 at 13:58):

The page I usually check is this one and it looks like all the workers are active there.

The "in progress" list can be a bit misleading since each commit has multiple jobs associated to it, so some commits show up as queued even though one of its jobs is actually in progress.

CI runner status bot (Sep 07 2021 at 19:13):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

Bryan Gin-ge Chen (Sep 07 2021 at 19:17):

I can't seem to ssh into nael at the moment. Note that this seems to have caused the latest build failure: https://github.com/leanprover-community/mathlib/runs/3536609973

CI runner status bot (Sep 07 2021 at 19:22):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

Bryan Gin-ge Chen (Sep 07 2021 at 20:31):

Looks like nael had another blip that didn't get picked up by the bot: https://github.com/leanprover-community/mathlib/runs/3537399971

Johan Commelin (Sep 07 2021 at 20:37):

It seems like we've been stress-testing the runners today.

CI runner status bot (Sep 11 2021 at 04:46):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 11 2021 at 05:04):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 12 2021 at 11:01):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 12 2021 at 11:17):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 12 2021 at 11:17):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 12 2021 at 14:48):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 12 2021 at 14:48):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 13 2021 at 12:26):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 13 2021 at 12:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Sep 13 2021 at 14:41):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 13 2021 at 14:41):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 13 2021 at 14:45):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Sep 13 2021 at 23:22):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 13 2021 at 23:29):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 14 2021 at 03:23):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 14 2021 at 03:25):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 15 2021 at 22:18):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 15 2021 at 22:34):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

Johan Commelin (Sep 16 2021 at 03:49):

Where is the yoyo emoji when you need it?

Kevin Buzzard (Sep 16 2021 at 10:30):

that also applies to the timings :-)

Kevin Buzzard (Sep 16 2021 at 10:31):

(or maybe a 5% discrepancy is normal for this sort of thing?)

Jannis Limperg (Sep 16 2021 at 21:44):

Kevin Buzzard said:

(or maybe a 5% discrepancy is normal for this sort of thing?)

The timings run on a VM in the cloud, which introduces a ton of noise since other VMs may use the same processor. When I set it up, the noise wasn't so bad, but it has increased a lot recently, to the point where the timings are really only good for a very rough sense of where the compilation times are going (up up and away).

CI runner status bot (Sep 18 2021 at 17:16):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 18 2021 at 17:47):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 18 2021 at 17:47):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 18 2021 at 21:03):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 18 2021 at 21:05):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 18 2021 at 22:59):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 18 2021 at 23:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 19 2021 at 14:46):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 19 2021 at 14:53):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 20 2021 at 03:29):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 20 2021 at 04:15):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 20 2021 at 06:52):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 20 2021 at 07:02):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 20 2021 at 07:02):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 22 2021 at 05:07):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 22 2021 at 05:25):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 22 2021 at 05:32):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 22 2021 at 05:54):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 22 2021 at 05:54):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 23 2021 at 10:58):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 23 2021 at 11:01):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

Bryan Gin-ge Chen (Sep 23 2021 at 14:35):

nale is currently offline according to this page. I'll wait a few minutes to see if this bot picks it up and then restart it manually.

Bryan Gin-ge Chen (Sep 23 2021 at 14:44):

I have just restarted the runner on nale.

@Scott Morrison It seems that the bot missed nale being offline for at least 7-8 minutes.

CI runner status bot (Sep 23 2021 at 14:47):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 23 2021 at 14:48):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 23 2021 at 17:45):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 23 2021 at 17:45):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 23 2021 at 23:29):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 23 2021 at 23:49):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 23 2021 at 23:56):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 23 2021 at 23:56):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Sep 24 2021 at 00:01):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 24 2021 at 00:05):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 24 2021 at 00:10):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Sep 24 2021 at 11:20):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 24 2021 at 11:58):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 24 2021 at 14:40):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 24 2021 at 15:03):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 03:18):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 26 2021 at 03:25):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 04:34):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 26 2021 at 04:38):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 04:38):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 04:56):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 05:02):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 26 2021 at 05:02):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 05:25):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 26 2021 at 05:25):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 16:03):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 26 2021 at 16:07):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Sep 26 2021 at 16:42):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 26 2021 at 16:42):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Sep 27 2021 at 18:51):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 27 2021 at 18:52):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Sep 28 2021 at 02:21):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 28 2021 at 02:46):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 28 2021 at 04:42):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 28 2021 at 04:45):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 29 2021 at 04:09):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Sep 29 2021 at 04:12):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 29 2021 at 18:43):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

Gabriel Ebner (Sep 29 2021 at 18:46):

What kind of "offline" is this actually? SSH timeout? https://github.com/leanprover-community/mathlib/settings/actions/runners shows that Lena is dutifully building her PRs. She's also responding to pings.

Johan Commelin (Sep 29 2021 at 18:48):

It might be that she's so busy she doesn't let the SSH server get any cycles?!

Bryan Gin-ge Chen (Sep 29 2021 at 18:50):

That seems to be the most common situation when the runners "go offline"; I can't seem to SSH in (see this message) but ping works.

CI runner status bot (Sep 29 2021 at 18:51):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 29 2021 at 18:51):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Sep 29 2021 at 18:51):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

Thomas Browning (Sep 29 2021 at 18:52):

lena is really online right now

Gabriel Ebner (Sep 29 2021 at 18:52):

slightly overloaded indeed:

$ ssh lena.vm.uni-freiburg.de uptime
 18:51:44 up 12 days, 11:39,  0 users,  load average: 30.09, 30.20, 24.88

Johan Commelin (Sep 29 2021 at 18:59):

Is this something we can improve by giving the scheduler some hints?

Johan Commelin (Sep 29 2021 at 19:00):

Something nice something something

Gabriel Ebner (Sep 29 2021 at 19:45):

I think the problem is likely too little RAM. The failure mode looks like the system is swapping so much it can't get any other work done.

Johan Commelin (Sep 29 2021 at 19:47):

Uggh

Johan Commelin (Sep 29 2021 at 19:47):

Guess we need to fix our CI machines

Gabriel Ebner (Sep 29 2021 at 19:55):

Feel free to chime in on the secret cabal channel!

Gabriel Ebner (Sep 29 2021 at 19:57):

But it would be great if could get some extra RAM on the Freiburg machines as well.

CI runner status bot (Oct 01 2021 at 14:10):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 01 2021 at 14:23):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 01 2021 at 14:23):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 02 2021 at 10:02):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 02 2021 at 10:11):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 02 2021 at 22:34):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

Bryan Gin-ge Chen (Oct 02 2021 at 22:41):

I just restarted lane and nela which looked like they were offline. I don't know how long they were down though. It looks like lena is offline but I can't SSH in...

CI runner status bot (Oct 02 2021 at 22:46):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 04 2021 at 01:24):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 04 2021 at 01:36):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 04 2021 at 15:56):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 04 2021 at 16:00):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Oct 04 2021 at 23:03):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 04 2021 at 23:15):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Oct 06 2021 at 22:28):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 06 2021 at 22:38):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 07 2021 at 21:07):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 07 2021 at 21:09):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Oct 10 2021 at 18:59):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 10 2021 at 19:01):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

Eric Wieser (Oct 11 2021 at 18:30):

Lena is out of disk space! https://github.com/leanprover-community/mathlib/actions/runs/1329779313

Gabriel Ebner (Oct 11 2021 at 18:37):

Should be fixed now.

Bryan Gin-ge Chen (Oct 12 2021 at 18:58):

I happened to notice nale was offline, so I just cleared the tmp-* files and restarted the runner.

Gabriel Ebner (Oct 12 2021 at 19:12):

Hmm, lena is also filling up /tmp again:

$ ssh lena.vm.uni-freiburg.de ls -lhS /tmp/ | head
total 1.5G
-rw------- 1 ghrunner ghrunner 385M Oct 12 17:59 tmp-242609uUbATNlg48KV
-rw------- 1 ghrunner ghrunner 377M Oct 12 08:21 tmp-184654aEgLVcTxJdgo
-rw------- 1 ghrunner ghrunner 375M Oct 11 18:48 tmp-1073567jSpOcX6hU3t
-rw------- 1 ghrunner ghrunner 374M Oct 12 15:05 tmp-219373x67k3vOOr2Kv
drwxr-xr-x 2 ghrunner ghrunner 4.0K Oct 12 16:18 hsperfdata_ghrunner
drwxr-xr-x 2 root     root     4.0K Sep 24 06:55 hsperfdata_root
drwx------ 3 root     root     4.0K Sep 17 07:12 snap.lxd
drwx------ 3 root     root     4.0K Sep 17 07:12 systemd-private-fdcdb34ddf6045c4834bfc5f561a6b64-systemd-logind.service-Kx3Kbj
drwx------ 3 root     root     4.0K Sep 17 07:11 systemd-private-fdcdb34ddf6045c4834bfc5f561a6b64-systemd-resolved.service-f1n5pf

Gabriel Ebner (Oct 12 2021 at 19:16):

It looks like all of our runners are accumulating files in /tmp...

Scott Morrison (Oct 12 2021 at 22:45):

The runners were maxed out, so I've restarted mine.

Gabriel Ebner (Oct 13 2021 at 08:02):

Bug filed: https://github.com/actions/upload-artifact/issues/256

Alex J. Best (Oct 14 2021 at 14:55):

Looks like the runners are out of space again https://github.com/leanprover-community/mathlib/actions/runs/1342114687 ?

Bryan Gin-ge Chen (Oct 14 2021 at 14:57):

Maybe I'm missing it, but I'm not seeing the "out of space" message in the logs you linked?

Floris van Doorn (Oct 14 2021 at 16:13):

image.png

Floris van Doorn (Oct 14 2021 at 16:14):

I also got one: https://github.com/leanprover-community/mathlib/actions/runs/1337561139

Bryan Gin-ge Chen (Oct 14 2021 at 16:20):

OK, I've just gone through and cleared all the temp files from the Freiberg runners (sudo rm /tmp/tmp-*). Hopefully that helps! We may want to automate this since who knows when GitHub will fix the issue, but I can't look into it right now.

Alex J. Best (Oct 18 2021 at 16:20):

I'm getting out of space errors again at https://github.com/leanprover-community/mathlib/actions/runs/1355343749

Bryan Gin-ge Chen (Oct 18 2021 at 16:44):

I see an actual build error there, but I'll clear the temp files again shortly:

configuring mathlib 0.1
Error: /home/ghrunner/actions-runner/_work/mathlib/mathlib/src/algebra/category/Group/colimits.lean:296:5:
invalid namespace name 'quotient_add_group'
Error: Process completed with exit code 1.

edit: temp files cleared

Alex J. Best (Oct 18 2021 at 17:10):

Right, I saw both a build error and an oom error, but it seems they are independent!

Bryan Gin-ge Chen (Oct 18 2021 at 17:12):

Hmm, I can't see the oom error in those logs but the "upload precompiled mathlib zip file" seems to be forever stuck with the yellow status. Does GitHub actually show something useful there for you? Or maybe I'm just looking in the wrong place?

Alex J. Best (Oct 18 2021 at 17:13):

On the link I sent to at the bottom there are 4 Annotations:

Build mathlib
System.IO.IOException: No space left on device at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source) at System.IO.FileStream.FlushWriteBuffer() at System.IO.FileStream.Flush(Boolean flushToDisk) at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message) at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut) at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args) System.IO.IOException: No space left on device at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source) at System.IO.FileStream.FlushWriteBuffer() at System.IO.FileStream.Flush(Boolean flushToDisk) at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message) at GitHub.Runner.Common.Tracing.Error(Exception exception) at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args) Unhandled exception. System.IO.IOException: No space left on device at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source) at System.IO.FileStream.FlushWriteBuffer() at System.IO.FileStream.Flush(Boolean flushToDisk) at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at System.Diagnostics.TraceSource.Flush() at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing) at GitHub.Runner.Common.TraceManager.Dispose() at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing) at GitHub.Runner.Common.HostContext.Dispose() at GitHub.Runner.Worker.Program.Main(String[] args)
Build mathlib
/home/ghrunner/actions-runner/_work/mathlib/mathlib/src/algebra/category/Group/colimits.lean:296:5: invalid namespace name 'quotient_add_group'
Build mathlib
Process completed with exit code 1.
Build mathlib
You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 63 MB

Bryan Gin-ge Chen (Oct 18 2021 at 17:20):

Aha, thanks. I never scroll down to the annotations since I just try to click through to the step with the red X.

Bryan Gin-ge Chen (Oct 20 2021 at 15:49):

lena was offline, so I just restarted it. @Scott Morrison is the bot down?

edit: I also went through the other runners and cleared the temp files just in case.

Eric Wieser (Oct 23 2021 at 13:55):

Looks like the batch of #9894 #9889 #9887 #9885 #9872 #9868 #9863 #9848 #9847 #9846 #9838 #9837 #9829 #9821 #9813 #9758 #9701 crashed

CI runner status bot (Oct 24 2021 at 22:52):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 24 2021 at 23:08):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:09):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:13):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 24 2021 at 23:17):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

Scott Morrison (Oct 24 2021 at 23:18):

Hmm... Should I turn this bot off? I can't tell if it is doing anything helpful. I can easily change its reporting criteria if anyone has a suggestion. Currently it saying "offline" if two consecutive attempts to login to an ssh shell (separated by a 60s pause) fail.

CI runner status bot (Oct 24 2021 at 23:22):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:26):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:32):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 24 2021 at 23:32):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:35):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

Scott Morrison (Oct 24 2021 at 23:36):

The runners are all active at the moment. https://github.com/leanprover-community/mathlib/settings/actions/runners

CI runner status bot (Oct 24 2021 at 23:38):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 24 2021 at 23:40):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 24 2021 at 23:40):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 24 2021 at 23:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:40):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:40):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Oct 24 2021 at 23:40):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

Johan Commelin (Oct 25 2021 at 03:49):

Would it make a difference if you switch to ping instead of ssh?

Scott Morrison (Oct 25 2021 at 08:23):

Okay, it's using ping -c 1 now. Let's see what happens.

Gabriel Ebner (Oct 25 2021 at 08:28):

Ideally I think we should be scraping the information at https://github.com/leanprover-community/mathlib/settings/actions/runners, and post here if the status changes to offline.

Sebastien Gouezel (Oct 25 2021 at 08:38):

Do I remember correctly that Azure1 and Azure2 are extremely expensive, and the plan was to disable them once we have enough Hoskinsons?

Gabriel Ebner (Oct 25 2021 at 09:12):

Yes.

Kevin Buzzard (Oct 25 2021 at 14:38):

Oh we're going to have Hoskinsons? :D

Johan Commelin (Oct 25 2021 at 14:41):

It should have been formatted as hoskinsons :wink:

CI runner status bot (Oct 26 2021 at 03:38):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 26 2021 at 03:40):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Oct 26 2021 at 20:01):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 26 2021 at 20:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Oct 27 2021 at 15:52):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 27 2021 at 16:01):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

Eric Wieser (Oct 29 2021 at 21:42):

lane has died in the middle of multiple PR batches today

Eric Wieser (Oct 29 2021 at 21:42):

the latest:
https://github.com/leanprover-community/mathlib/actions/runs/1400651968

Eric Wieser (Oct 29 2021 at 21:49):

Here's a failing PR build: https://github.com/leanprover-community/mathlib/actions/runs/1400699073

Eric Wieser (Oct 29 2021 at 21:49):

I guess it's probably out of memory / disk space?

Bryan Gin-ge Chen (Oct 29 2021 at 21:53):

I can't SSH in, so it's definitely still overloaded, one way or another.

Eric Wieser (Oct 30 2021 at 13:38):

How much work does it add to just nuke these runners from the github UI when this happens, until someone can fix them?

CI runner status bot (Oct 31 2021 at 13:31):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 31 2021 at 13:32):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Oct 31 2021 at 13:35):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Oct 31 2021 at 13:35):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

Eric Wieser (Oct 31 2021 at 16:26):

I removed the pr label from lane, it can't be trusted to do anything at all right now

CI runner status bot (Nov 05 2021 at 00:18):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 05 2021 at 00:20):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 05 2021 at 22:26):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 05 2021 at 22:30):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 07 2021 at 10:49):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 07 2021 at 10:50):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Nov 09 2021 at 02:16):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 09 2021 at 02:17):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 09 2021 at 02:20):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Nov 09 2021 at 02:20):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Nov 14 2021 at 09:01):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 14 2021 at 09:05):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Nov 16 2021 at 09:31):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 16 2021 at 09:35):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Nov 16 2021 at 12:56):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 16 2021 at 12:58):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 16 2021 at 13:00):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 16 2021 at 13:26):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Nov 16 2021 at 13:26):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Nov 16 2021 at 13:30):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Nov 16 2021 at 13:46):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 16 2021 at 14:41):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 16 2021 at 15:00):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 16 2021 at 15:00):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Nov 16 2021 at 17:01):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 16 2021 at 17:05):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

Scott Morrison (Nov 16 2021 at 19:53):

Hrmmm...

Gabriel Ebner (Nov 16 2021 at 21:26):

@Scott Morrison The VMs were upgraded to 20G RAM today.

Scott Morrison (Nov 17 2021 at 23:32):

@Jannis Limperg just noting that the benchmark bot hasn't been running since the 8th.

Jannis Limperg (Nov 18 2021 at 12:36):

Thanks for the ping! Restarted now.

CI runner status bot (Nov 18 2021 at 16:37):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 18 2021 at 16:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 20 2021 at 19:47):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 20 2021 at 19:50):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 20 2021 at 23:30):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 20 2021 at 23:30):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Nov 21 2021 at 22:32):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 21 2021 at 22:35):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Nov 22 2021 at 01:17):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 22 2021 at 01:20):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Nov 22 2021 at 23:21):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 22 2021 at 23:25):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 25 2021 at 07:21):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 25 2021 at 07:25):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Nov 25 2021 at 09:57):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 25 2021 at 10:00):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Nov 28 2021 at 19:13):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 28 2021 at 19:15):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Nov 30 2021 at 02:21):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Nov 30 2021 at 02:25):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

Riccardo Brasca (Nov 30 2021 at 10:39):

CI is failing for #10350 with "Error: No space left on device".

Gabriel Ebner (Nov 30 2021 at 13:08):

I wish the self-hosted github runner would clean up the build directories...

CI runner status bot (Dec 01 2021 at 04:41):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 01 2021 at 04:45):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Dec 01 2021 at 10:26):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 01 2021 at 10:30):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Dec 05 2021 at 02:06):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 05 2021 at 02:10):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Dec 09 2021 at 15:51):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 09 2021 at 15:55):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

Riccardo Brasca (Dec 10 2021 at 11:05):

#10350 fails linting mathlib, but if I click "details" too see the problem I only see the yellow circle rotating, without any message. Is this a problem of CI or there is something I can do to fix it?

Riccardo Brasca (Dec 10 2021 at 11:06):

I've tried to re-run all the jobs, but it didn't worked

Anne Baanen (Dec 10 2021 at 11:09):

For some reason, the error only appears on the status page: https://github.com/leanprover-community/mathlib/actions/runs/1562877810
The self-hosted runner: lena lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Anne Baanen (Dec 10 2021 at 11:10):

Since the runners should be back online, I've set them to re-rerun and see what happens...

CI runner status bot (Dec 14 2021 at 02:21):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 14 2021 at 02:22):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 14 2021 at 02:25):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Dec 14 2021 at 02:25):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Dec 16 2021 at 05:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Dec 16 2021 at 05:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Dec 18 2021 at 03:39):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Dec 18 2021 at 03:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Dec 18 2021 at 03:40):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Dec 18 2021 at 03:40):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Dec 18 2021 at 03:40):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Dec 19 2021 at 06:56):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 19 2021 at 07:00):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Dec 19 2021 at 07:12):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Dec 19 2021 at 07:17):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

Eric Wieser (Dec 23 2021 at 13:47):

lane seems to be offline

Gabriel Ebner (Dec 23 2021 at 14:08):

Disk is full.

CI runner status bot (Jan 13 2022 at 19:51):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jan 13 2022 at 19:55):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jan 17 2022 at 15:31):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jan 17 2022 at 15:34):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jan 17 2022 at 15:35):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jan 17 2022 at 15:35):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Jan 21 2022 at 11:21):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jan 21 2022 at 11:22):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jan 21 2022 at 11:25):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jan 21 2022 at 11:25):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jan 24 2022 at 16:46):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jan 24 2022 at 16:50):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jan 30 2022 at 19:45):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Feb 10 2022 at 05:00):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Feb 10 2022 at 05:00):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Feb 10 2022 at 05:00):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Feb 12 2022 at 21:31):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 12 2022 at 21:35):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Feb 13 2022 at 23:36):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 13 2022 at 23:37):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 13 2022 at 23:40):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Feb 13 2022 at 23:40):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Feb 15 2022 at 04:16):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Feb 15 2022 at 04:20):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Feb 21 2022 at 23:01):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 21 2022 at 23:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Feb 25 2022 at 05:15):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Feb 25 2022 at 05:15):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Feb 25 2022 at 05:56):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 25 2022 at 05:57):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 25 2022 at 06:01):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 25 2022 at 06:02):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 25 2022 at 06:06):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Feb 25 2022 at 06:36):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Feb 25 2022 at 06:41):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Feb 25 2022 at 06:45):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Mar 01 2022 at 21:41):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 01 2022 at 21:42):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 01 2022 at 21:45):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Mar 01 2022 at 21:45):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Mar 07 2022 at 13:55):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Mar 07 2022 at 13:55):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Mar 08 2022 at 20:56):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 08 2022 at 21:00):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Mar 15 2022 at 23:51):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 16 2022 at 00:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Mar 16 2022 at 00:31):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 16 2022 at 00:32):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 16 2022 at 00:36):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 16 2022 at 00:36):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Mar 16 2022 at 00:40):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Mar 16 2022 at 00:40):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Mar 22 2022 at 22:06):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 22 2022 at 22:10):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Mar 24 2022 at 06:01):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 24 2022 at 06:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Mar 25 2022 at 08:16):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 25 2022 at 08:17):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 25 2022 at 08:20):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Mar 25 2022 at 08:20):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Mar 25 2022 at 20:31):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 25 2022 at 20:35):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Mar 29 2022 at 19:16):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 29 2022 at 19:20):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Mar 30 2022 at 01:11):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 30 2022 at 01:15):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Mar 31 2022 at 20:51):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Mar 31 2022 at 20:55):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Apr 03 2022 at 00:20):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 03 2022 at 00:20):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Apr 03 2022 at 00:20):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Apr 03 2022 at 00:20):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Apr 07 2022 at 18:01):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 07 2022 at 18:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 08 2022 at 06:11):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 08 2022 at 06:15):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 09 2022 at 00:21):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 09 2022 at 00:25):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 09 2022 at 06:01):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 09 2022 at 06:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Apr 12 2022 at 21:51):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 12 2022 at 21:55):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Apr 13 2022 at 09:31):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 13 2022 at 09:35):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Apr 19 2022 at 13:51):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 19 2022 at 13:52):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 19 2022 at 13:56):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Apr 19 2022 at 14:00):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 20 2022 at 20:36):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 20 2022 at 20:37):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 20 2022 at 20:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 20 2022 at 20:40):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Apr 21 2022 at 06:56):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 21 2022 at 07:00):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 28 2022 at 17:26):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 28 2022 at 17:30):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Apr 29 2022 at 15:01):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 29 2022 at 15:02):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Apr 29 2022 at 15:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Apr 29 2022 at 15:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (May 05 2022 at 19:01):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (May 05 2022 at 19:05):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

Eric Wieser (May 06 2022 at 14:46):

I've had the lean-permutation runners betray me twice in https://github.com/leanprover-community/mathlib/actions/runs/2280987158 now

CI runner status bot (May 25 2022 at 05:31):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (May 25 2022 at 05:35):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (May 27 2022 at 22:46):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (May 27 2022 at 22:50):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (May 29 2022 at 22:01):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (May 29 2022 at 22:05):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (May 30 2022 at 17:27):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (May 30 2022 at 17:30):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Jun 01 2022 at 04:01):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 01 2022 at 04:02):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 01 2022 at 04:06):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 01 2022 at 04:07):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 01 2022 at 04:09):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 01 2022 at 04:10):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Jun 01 2022 at 04:11):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jun 01 2022 at 04:11):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Jun 01 2022 at 04:11):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Jun 01 2022 at 04:15):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jun 03 2022 at 06:00):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jun 05 2022 at 09:01):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 05 2022 at 09:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jun 09 2022 at 08:11):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 09 2022 at 08:15):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Jun 16 2022 at 07:15):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jun 29 2022 at 02:31):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 29 2022 at 02:32):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 29 2022 at 02:35):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jun 29 2022 at 02:35):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jun 30 2022 at 07:16):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jun 30 2022 at 07:20):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

Eric Wieser (Jul 12 2022 at 22:32):

I think nale has dropped maybe 5-10 jobs on me today

Eric Wieser (Jul 12 2022 at 22:32):

It would be great if we had an easy way to tell which runners were failing every time

Rob Lewis (Jul 13 2022 at 19:38):

I've seen a few dropped jobs too, are you sure it's Nale?

Rob Lewis (Jul 13 2022 at 19:39):

At least, I've seen a few dropped without a message and without a log, which I think means the runner is just disconnecting, right?

CI runner status bot (Jul 17 2022 at 19:16):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 17 2022 at 19:20):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:01):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:02):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:04):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:05):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:05):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:07):

@github-runners-maintainers lane.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:11):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:12):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:14):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 18 2022 at 11:18):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:19):

@github-runners-maintainers lane.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:20):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:20):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jul 18 2022 at 11:20):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Jul 19 2022 at 10:56):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 19 2022 at 10:57):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

Johan Commelin (Jul 19 2022 at 10:57):

Yes, I know. The entire department is offline!!!

Johan Commelin (Jul 19 2022 at 10:58):

I'm working on a mobile connection atm.

CI runner status bot (Jul 19 2022 at 10:59):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 19 2022 at 11:00):

@github-runners-maintainers nela.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 19 2022 at 11:02):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

Johan Commelin (Jul 19 2022 at 11:03):

Yay! We're back online

CI runner status bot (Jul 19 2022 at 11:05):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jul 19 2022 at 11:05):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jul 19 2022 at 11:05):

@github-runners-maintainers nela.vm.uni-freiburg.de is online

CI runner status bot (Jul 19 2022 at 21:51):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 19 2022 at 21:55):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jul 20 2022 at 18:16):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 20 2022 at 18:17):

@github-runners-maintainers lena.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 20 2022 at 18:25):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jul 20 2022 at 18:25):

@github-runners-maintainers lena.vm.uni-freiburg.de is online

CI runner status bot (Jul 22 2022 at 11:31):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 22 2022 at 11:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

CI runner status bot (Jul 24 2022 at 00:01):

@github-runners-maintainers nale.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 24 2022 at 00:05):

@github-runners-maintainers nale.vm.uni-freiburg.de is online

CI runner status bot (Jul 26 2022 at 07:36):

@github-runners-maintainers nael.vm.uni-freiburg.de appears to be offline

CI runner status bot (Jul 26 2022 at 07:40):

@github-runners-maintainers nael.vm.uni-freiburg.de is online

Rob Lewis (Jul 28 2022 at 17:12):

I've seen a new spurious build failure a couple times in the past day or two. The build stage fails and GitHub just reports
Error: The log was not found. It may have been deleted based on retention settings.