key's blog

Willers Downtime 7PM PDT Oct 7th, 2009

Planned outage - July 31st, 2009 11pm PDT

Willers will be down for file system maintenance at 11PM PDT, July 21st, 2009.
This may take a few hours.

Outage Dec 13th, 2007

Willers experienced hardware failure (NIC) after it was moved
at the colocation facility on Dec 13th. We are live on a new
interface as of Dec 14th.

Downtime Nov 3rd, 2007 at 11AM PDT for approx. 2 hours

Willers will be down on Saturday, November 3rd 2007 at 11AM PDT for
approximately 2 hours for hardware maintenance.

Yet more disk problems

Disk problems continue to plague willers. We've replaced the
users disk. As a result everyone's home area has moved from
/export/home[bcdefg] to being either /export/homeh or /export/homei.
If you have anything installed that is using the /export/homeXXX as
part of the path, fix it to use /users/<your-username>/. We will
continue to shuffle home areas around to try to balance the I/O load,
so these directories will change in the future. We have also
recabled the system disk.

There was some filesystem corruption. Two bad blocks were encountered
while moving the users date.

Downtime Sept 21st, 2007 after 1730 PDT

Equipment in the colocation facility needs emergency service
and Willers has to be moved to access the equipment. Downtime will
be tonight (Friday, Sept 21st at 1730 PDT). Downtime is estimated
to be 15 minutes.

Outage due to disk problems

We had a disk controller lockup that hung the system disk and resulted in some file system corruption in /var/mail. We are still looking into the situation.

A reminder to users that there are NO backups. See the "Lack
of Service Guarantee":/docs/policy#guarantee and "Lack of
Backups and Disaster Recovery":/docs/policy#backups for
more details.

Willers Downtime July 28th 2007 at 11AM

Willers will be down for potentially several hours on Saturday, July 28th 2007 starting at 11AM PDT. We will be swapping out hardware in an attempt to address the stability issues we have had lately.

ssh problem fixed.

We had a problem with default sshd configuration and with DenyHosts
clamping down on some hosts from member companies. We have fixed
the sshd configuration and we've believe we have cleared the
denyhosts entries that were put there by mistake. If you
are still being blocked, please contact admin@employees.org

OS Upgrade complete

We have completed our upgrade to FreeBSD 6.2 and are working through some issues. Contact admin@employees.org with new issues and help@employees.org with questions.

Syndicate content