[BIC-announce] The state of computer systems ...

[Sylvain MILOT]

Hello everyone,

The past few days, beginning with the last power shutdown, have been
trying to say the least. We lost another IO-brick on yorick sunday night
but I managed to revive it by telling the (L1) controller to ignore
environmental control! Oddly enough, (and with great joy :) all 3 fans on
that io-brick started spinning normally and the beast rebooted happily and
went on merrily until this morning. Remember that neither bullcalf nor
yorick are on service contract anymore so things are not as simple as
placing a call to SGI. It could be that this io-brick is really dying but
I'll have to wait a while longer to know for sure.

The problems this morning are related (to a dying io-brick?) to NFS
and the fact that both bullcalf (especially) and yorick are under heavy
attack by the likes of Oliver Lyttelton - a man with a deadline! Just to
say that the PBS queues are extremely busy and that this situation is
probably not ideal for systems that have already suffered the loss of two
io-bricks and two cpu bricks. The load average on bullcalf hovers at 75
and I've disabled logins to it for now.

As I'm writing this, NFS is acting funny between shadow and
some disks on CXFS (managed by yorick/bullcalf), namely
/data/aces/aces1/. The latter is under heavy utilization and a
failing/sub-optimal io-brick plus a system that has already suffered the
loss of another io-brick and that had to be reconfigured to accomodate
this loss ... well NFS going awry might just be a manifestation of this.
The good news is that there's no sign of data corruption.

For now, bullcalf and yorick are happily crunching data and things are
holding ... we'll see how far they go ...

---
Sylvain Milot (sylvain at bic.mni.mcgill.ca)
              (trinity at bic.mni.mcgill.ca)
Brain Imaging Centre
Montreal Neurological Institute
Webster 2B, Room 208
Montreal, Qc., Canada, H3A 2B4
Phone  : (514) 398-4965, Fax: 398-8948
Mobile : (514) 712-1768
Office : 527 Av Des Pins O., Room 204
         Montreal, Qc., H2W 1S4