Skip to content

Service recovery Humanities IT Services

Due to a fire in our main datacenter, most of the services hosted by Humanities IT Services are currently down.

For information on the status of other UU services, please visit the liveblog page on uu.nl.

What are we working on

We are currently working on restoring critical systems and creating a general 'offline' page informing users that our services are currently down and providing contact information where necessary.

The future

We are currently waiting for power to be restored to the datacenter. Once we have power, we will need to verify that our hardware is working properly and then begin working on restoring our services.

We currently do not have a timeline for when we will be able to restore our services.

Contact

If you have any questions or concerns, please contact us at our regular email address. Please note that our team is busy, so we may not be able to respond to your email immediately.

Timeline

Monday recap

We've been able to get the first few emergency pages up and running (most importantly, diergeneeskunde.nl) and deployed servers for other critical services.

However, we're still blocked by a DNS issue. We hope that ITS will be able to resolve it by tomorrow morning at the latest.

Monday morning update

We're currently hard at work coordinating our recovery plan internally and with ITS; at this point it seems likely that we won't be able to start full recovery until Thursday at the earliest. This is due to delays in restoring power at the datacenter.

In the meantime we're also working on deploying temporary infrastructure to get critical services back up, as well as a general status page for the UU wordpress environment.

However, we're currently blocked by a DNS issue that ITS will have to resolve before we can proceed with bringing the temporary infrastructure online.

Expected delay

We've received word that NorthC is not going to be able to restore power to our equipment tomorrow.

We will evaluate this news tomorrow and see what can be done in the meantime.

Final update for today

NorthC has indicated that it may take 72 hours to restore power. This means that we may be able to start bringing up infrastructure again on Monday at the earliest. However, full restoration will take longer.

As such, we still cannot give a concrete timeline at this point. We hope to have a more detailed update on Monday.

In the meantime, we have been unable to get the temporary landing page running due to unforseen issues outside our control. We're hoping to get this resolved soon, but we can make no promises at this point.

Initial site visit by ITS

We have received word from ITS that they've visited the datacenter. While the building is still without power, our hardware does not appear to be affected by the fire.

We are still waiting on NorthC to provide us with a timeline for when power is restored and when we can start starting our servers.

Restoring critical services

We have started working on restoring critical (internal) services in our alternate datacenter.

This will include a temporary landing page for users informing them of the outage.

Fire under control

The fire department has declared 'brand meester', which means the fire is under control and is no longer a risk of spreading to other sections of the building.

Backups verified

We have confirmed that our off-site backups are present and accounted for. We expect little to no data-loss to occur once our servers are restored.

Initial incident

On May 7th, around 8:30, a fire broke out in one of the technical rooms in the NorthC datacenter in Almere, where most of our servers are located. By order of the fire department, power was cut off to the datacenter around 9:10.

As a result, most of our services went offline.