Two Weeks On: Your Questions About Data, Servers and Timelines, Answered
Posted on
On September 23, Russian missile strikes hit both of our data centers in Kyiv. Since then, you have sent us a lot of questions: in tickets, in chat and on forums. Many of them are the same questions, asked by many of you.
We collected the ones that came up most often and answered them here, directly and with the numbers we have today. This way everyone gets the same answers, in one place.
Dedicated servers: where things stand today
Every server we have checked so far is working. Based on that, we expect the affected share to be small: about 90% of servers should be unaffected, and 99% of customers should get their data back.
The queues are moving:
- Drive handover: about 85% of customers who requested their drives have already received them.
- Replacement servers: about 42% of those who requested a replacement have it.
- Full service restoration: up to 10% so far.
Our entire virtual infrastructure (cloud and VPS) has been restored, and not from backups. Some dedicated server customers have already been restored or moved to a replacement server.
Dedicated servers: timelines
You have the servers and a new facility. Why is it taking this long?
What holds things up is space, not hardware. We did find space, and all of it went to bringing cloud and VPS back online. That facility is now completely full.
New racks for dedicated servers are being prepared, and we expect them in about two weeks, although that doesn’t depend on us alone. Moving servers abroad also takes time and involves factors outside our control.
So how long will it take?
Two to three weeks. About two weeks for the new racks to be ready, then about five days to install the servers. Not all of this depends on us, but on our side we are doing everything we can to speed it up.
How many DC1 servers need relocation, and is there a priority list?
- All DC1 servers need relocation. The exact number is internal, so we don’t publish it.
- One facility is already full, and we are waiting for two more.
- Once a facility is ready, we need about five days to install all the servers.
- There is no priority list for restoring service: we physically move everything in one go.
- Replacements do follow an order: by the date of the request and as matching components become available.
If DC1 servers are intact, why are they still offline?
In DC1 the cooling system was destroyed. The servers are intact, but they can’t run there. DC2 was destroyed completely, so the two are not in the same situation.
Some DC1 servers have already been relocated and are working.
DC2: water, inspection and your drives
DC2 was badly damaged. What happened to the equipment?
All of our equipment has been taken out of that data center and moved to safe storage. Visually it is undamaged. There are traces of water, but that should not affect the data. We have already checked most of the storage media, and all of it works.
What happens to servers that came into contact with water?
Every server that came into contact with water goes through technical servicing and checks before it is put back into operation. If any server hardware turns out to be damaged, we move its drives into a replacement server, so your data stays with you. Hardware is not a constraint: we have enough of it in stock.
What if the inspection shows the drives are damaged?
We have checked a large number of SSDs, and so far 100% of them work. We haven’t reached the HDDs yet, but we expect the same picture there.
Have you actually read data from the recovered drives?
Yes. Every storage device we have checked is undamaged and working. The rest are in our storage and visually undamaged. The wait comes down to space: we can’t connect all servers at once, so customers are in a queue.
Do you realistically expect customers to get their data back?
Yes, we do. We expect 99% of customers to get their data. We believe all SSDs are intact, and most HDDs. Some HDDs may have been affected by the shockwave, but we haven’t come across any so far.
How can you be sure the data is still there?
If the data were lost, we wouldn’t be telling you we expect 99% recovery. Our entire virtual infrastructure has been restored, and not from backups. Customer dedicated servers were in the same racks as that infrastructure.
Getting your data before your server is back
Can I get my data before full restoration?
Yes. Many processes are running in parallel right now: restoration, inspection, connecting servers, and assembling replacement servers in other locations.
The vast majority of customers who asked only for their drives have already received them. Handing over data takes longer, and we are working through those requests step by step. Everyone who contacted us about their data is on the waiting list.
I only need a few files, not the whole server.
Please open a ticket and tell us which files or folders you need. A request for specific files is processed faster than a request for a full disk dump.
Can I verify the integrity of my data myself?
If you are in Ukraine, yes. Please contact us to arrange it.
Replacements, moving and colocation
Can I get a replacement server instead of waiting?
Yes. Replacements are handled in the order the requests come in and as matching components become available. Please request one in your ticket.
Can you move my VPS to the Netherlands?
Yes. Everyone who asked for this has either been moved already.
What about colocation customers?
For colo it’s simple: send a courier and we will hand over your hardware. We can have it ready the next business day.
Billing, support and communication
What about my invoices and compensation for the downtime?
If you are moving to a replacement server or continuing your service, your invoices will be recalculated accordingly, and you will not be charged for the days of downtime.
Other billing and compensation questions are reviewed individually, depending on each situation: your service and what you choose next, whether that is restoration, a replacement or your data only. Everyone who has already asked about this has received a reply from our team and has been added to the review list. Because of the high number of requests right now, this review has its own waiting list.
When can I expect a reply?
Our technical team works on recovery and all related processes 24/7. Our sales and billing departments work during business hours. You may well get a reply from them outside those hours too, as the whole team is working closely together right now, but please keep in mind that we are handling a large number of requests, so ticket response times may be longer than usual. Thank you for your understanding.
Where can I find updates about my server?
We understand that waiting without news is hard. Here is how we approach it. We publish a public update when there is new information we can stand behind, and we would rather not post just for the sake of posting. General information is published here on our website. Anything specific to your service is answered in your ticket, because that is where we can look at your exact server.
Coming soon: Warsaw
We are opening a new location in Warsaw, Poland, and the first servers launch in about two weeks.
Warsaw places our infrastructure inside the European Union, right next to Ukraine. For customers whose users are in Ukraine and across Central and Eastern Europe, it means one more location close to their audience, alongside our three existing ones.
- Dedicated servers and VPS. Both will be available in Warsaw, with a range of plans. We will share the details closer to launch.
- Moving from Kyiv. If you would like your server moved from Kyiv to Warsaw, we can do that. Just let us know in your ticket.
What’s next
We keep working at full speed, and despite everything, we keep growing.
As you already know, all cloud services in Ukraine are available to order. As for dedicated servers, some have already been restored in Ukraine, and we continue working on the rest. We are actively restocking servers in all our locations, but new stock sells out fast, so please check current availability with us in a ticket before ordering.
Thank you for your patience and for staying with us.
VSYS Host Team
