downtime: August 13-20

We will have downtime sometime between August 13th and 20th, to re-cable the server. The exact date is not yet known. Downtime is expected to be approximately 6 hours.

mail problems -- poor reputation due to compromised user account

Today we have a poor IP address reputation because a bunch of spam was sent through employees.org via a compromised user account. The account password has been changed and the email queue cleared of the spam. However, it will be a little while before our email reputation improves. Until that happens, mail will be delayed (queued on employees.org) or possibly rejected (bounced to the sender).

Sorry for the inconvenience.

UPDATE, 21-Feb 14:37. We now have a neutral senderbase reputation and are no longer listed on the bl.spamcop.net blacklist. The mail queue has mostly drained.

Update on banjo.employees.org disk problems.

So it seems that bringing the system down to single user mode and letting fsck do its stuff has solved the immediate problem of terrible performance.

But the longer term problems still exist. Both of our RAID arrays are running degraded. Another disk failure and we could lose the data.

Current plan is to ditch the hardware and move the entire system to a VM on a server that being generously donated. Work is in progress on bringing up this VM and migrating everything over.

Stay tuned. More to come.

-steve

Twitter account for Employes.org status information

To augment http://status.employees.org we recently created the
twitter account @EmployeesORG to report system status. You can follow
@EmployeesORG on twitter or see posts at https://twitter.com/EmployeesORG.

Web mail now supported (Roundcube)

We support web access to email, via the link https://www.employees.org/roundcube. Use your normal employees.org username and password to login.

migration complete

DNS records have been updated, and Banjo is the new employees.org server.

If there are any problems, send email to admin@employees.org

Status.Employees.Org

During outages of employees.org, you can visit http://status.employees.org for status updates. status.employees.org is hosted on a different computer in a different datacenter.

Welcome to employees.org

Employees.org is now re-opened for logins.

We have done our best effort to recover user data off of /dev/ad4. I remind users we do NOT back up user data and the disaster recovery plan is to wipe the system and start anew. The latter almost happened today. If this is not acceptable to you, you may have outgrown what Employees.Org offers and should consider a commercial service.

There was some file corruption from /dev/ad4 that resulted in Mailman bouncing a lot of messages.

Syndicate content