Due to a fire in our main datacenter, most of the services hosted by Humanities IT Services are currently down.
For information on the status of other UU services, please visit the liveblog page on uu.nl.
We're working on our plan for restoring full services once our datacenter is back online. We've reached out to our service owners to figure out which services need to be prioritized.
Power has been restored to the datacenter. We're currently waiting for our colleagues at ITS to verify and restart their systems. This may take some time, as they have a lot of work to do and are prioritizing stability over speed.
We've provided an initial timeline in this post. We're still dealing with some unknowns, so a more detailed timeline will follow once we have a better understanding of the situation.
Matching/Osiris
We've received a few mails about the status of the matching forms. Please check the main Utrecht University site for information regarding matching.
Humanities IT Services
If you have any questions or concerns, please contact us at our regular email address. Please note that our team is busy, so we may not be able to respond to your email immediately.
We've been able to restore our core services ahead of schedule.
Some services may appear to be working, but we haven't been able to thoroughly test them yet. We will post an update as soon as we can; but in the meantime we kindly ask everyone to be patient and wait till we checked that everything is working as expected.
Our colleagues at ITS are currently hard at work with the initial startup of the UU infrastructure at the datacenter.
We've discussed our plans for tomorrow with ITS and are in contact with them about their progress and the next steps for us. We're still expecting to start working on bringing services online tomorrow.
We cannot guarantee our services will work reliably when they first start up, so we kindly ask everyone to be patient and wait till we and/or ITS confirm that everything is working as expected.
NorthC has restored the power in the datacenter itself a little ahead of schedule. For now, this has no impact on our timeline.
We're currently preparing for tomorrow, and have decided to stop any other recovery efforts for now. This is to properly plan out final details and preserve our staff for the long few days ahead.
We've created an initial timeline for restoring services in our datacenter.
Tomorrow, Wednesday, NorthC will restore power to the datacenter, after which ITS will need to start up their infrastructure. They have indicated that they expect to start doing so after 17:00.
Hence, we plan to start up our infrastructure on Thursday, May 14th. We will initially focus on our own core infrastructure before moving on to other services. We've been in contact with our stakeholders and have made an initial prioritization list. With any luck, those will be online by the end of Friday. Full restoration may take a few more days after that.
ITS has managed to resolve the DNS issue, which allows us to bring up several emergency landing pages and other services.
Most sites in our wordpress environments should now be redirected to this page, while some others will be directed to dedicated info pages for those sites. Note that this only applies to sites under the uu.nl domain, and some select others.
We've been able to get the first few emergency pages up and running (most importantly, diergeneeskunde.nl) and deployed servers for other critical services.
However, we're still blocked by a DNS issue. We hope that ITS will be able to resolve it by tomorrow morning at the latest.
We're currently hard at work coordinating our recovery plan internally and with ITS; at this point it seems likely that we won't be able to start full recovery until Thursday at the earliest. This is due to delays in restoring power at the datacenter.
In the meantime we're also working on deploying temporary infrastructure to get critical services back up, as well as a general status page for the UU wordpress environment.
However, we're currently blocked by a DNS issue that ITS will have to resolve before we can proceed with bringing the temporary infrastructure online.
We've received word that NorthC is not going to be able to restore power to our equipment tomorrow.
We will evaluate this news tomorrow and see what can be done in the meantime.
NorthC has indicated that it may take 72 hours to restore power. This means that we may be able to start bringing up infrastructure again on Monday at the earliest. However, full restoration will take longer.
As such, we still cannot give a concrete timeline at this point. We hope to have a more detailed update on Monday.
In the meantime, we have been unable to get the temporary landing page running due to unforseen issues outside our control. We're hoping to get this resolved soon, but we can make no promises at this point.
We have received word from ITS that they've visited the datacenter. While the building is still without power, our hardware does not appear to be affected by the fire.
We are still waiting on NorthC to provide us with a timeline for when power is restored and when we can start starting our servers.