Dunkel is dead

[inducer]

Currently, lots of messages like this in the dmesg:

[Mon Oct  3 13:52:42 2022] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[Mon Oct  3 13:52:42 2022] EDAC sbridge MC1: CPU 0: Machine Check Event: 0 Bank 7: cc00418000010091
[Mon Oct  3 13:52:42 2022] EDAC sbridge MC1: TSC 0 
[Mon Oct  3 13:52:42 2022] EDAC sbridge MC1: ADDR 2031c14940 
[Mon Oct  3 13:52:42 2022] EDAC sbridge MC1: MISC 150481a86 
[Mon Oct  3 13:52:42 2022] EDAC sbridge MC1: PROCESSOR 0:406f1 TIME 1664823350 SOCKET 0 APIC 0
[Mon Oct  3 13:52:42 2022] EDAC MC1: 262 CE memory read error on CPU_SrcID#0_Ha#0_Chan#1_DIMM#0 (channel:1 slot:0 page:0x2031c14 offset:0x940 grain:32 syndrome:0x0 -  OVERFLOW area:DRAM err_code:0001:0091 socket:0 ha:0 channel_mask:2 rank:1)

cc @rckirby

[inducer]

They seem to be happening every few seconds or so.

Rebooted it, to see if that helps.

[rckirby]

Seems not to have come back up? ``` ***@***.***:~/2022-nsf-transform-nonlocal$ ssh dunkel channel 0: open failed: connect failed: No route to host stdio forwarding failed kex_exchange_identification: Connection closed by remote host Connection closed by UNKNOWN port 65535 ```

[inducer]

https://www.complang.tuwien.ac.at/anton/failing-memory.html has a description of someone troubleshooting a similar issue.

[inducer]

@lukeolson What's the latest here? @kaushikcfd will transfer the GPU out of dunkel to keep it usable this afternoon.