DDos attack - Servers up again
Incident Report for Kraken.io

Saturday night around 11PM someone using a hacking tool to find attack vectors for our API with a fake IP address (see https://en.wikipedia.org/wiki/IP_address_spoofing for more context) usually assigned to the African IP address registry.

The effect of this was limited but kicked of a fatal chain. In our processing queue for the API, there is a pool of NodeJS based workers sitting behind nginx load balancers. Now those load balancers tried to keep up with the incoming requests and at some point simply could not due to hard limits set for the amount of requests a load balancer should be allowed to serve. In the end the load balancers stopped serving requests since the were simply not allowed to open further connections to the API workers. So we set the workers free and allowed more connections to be handled by each, monitored the situation for a while and decided, all is fine in Kraken.io Land.

Spoilers: it was not. An hour later requests were piling up again and the API was refusing to service anything. It turned out requests were hugely delayed because of unanswered DNS requests. Since we use a private DNS infrastructure this was rather weird. Internal requests were being answered within acceptable times just fine. But: external requests were no longer answered. At this time, most of our systems started reaching the End-Of-Life for cached DNS requests. For the outside world, this culminated in kraken.io no longer being available, while for our private infrastructure the rest of the world ceased to exist.

Further investigation showed that our own DNS servers were no longer receiving DNS zones from upstream servers, e.g. us asking for s3.amazon.com on our upstream servers got denied. A few cables further the issue was finally spotted: the DNS systems within the data center - to which we turn when requesting information about public DNS zones - were no longer available.Mitigation efforts undertaken:

  • load balancer scaling has been changed to automatically adapt to incoming load instead of hard defaults
  • our pool of DNS servers for external requests has been expanded (edited)


IP address spoofing

In computer networking, IP address spoofing or IP spoofing is the creation of Internet Protocol (IP) packets with a false source IP address, for the purpose of impersonating another computing system.(92 kB)

Posted Nov 11, 2019 - 18:29 CET

This incident has been resolved.
Posted Nov 10, 2019 - 01:23 CET
A fix has been implemented and we are monitoring the results.
Posted Nov 10, 2019 - 01:17 CET
We are continuing to work on a fix for this issue.
Posted Nov 10, 2019 - 01:16 CET
The issue has been identified and a fix is being implemented.
Posted Nov 10, 2019 - 01:16 CET
We are continuing to investigate this issue.
Posted Nov 10, 2019 - 01:07 CET
We are continuing to investigate this issue.
Posted Nov 10, 2019 - 01:06 CET
We are currently investigating this issue.
Posted Nov 10, 2019 - 01:04 CET
This incident affected: Kraken.io API, Kraken.io Web Interface, Kraken.io Storage, Kraken.io Homepage, and LiveChat Chat widget.