Saturday night around 11PM someone using a hacking tool to find attack vectors for our API with a fake IP address (see https://en.wikipedia.org/wiki/IP_address_spoofing for more context) usually assigned to the African IP address registry.
The effect of this was limited but kicked of a fatal chain. In our processing queue for the API, there is a pool of NodeJS based workers sitting behind nginx load balancers. Now those load balancers tried to keep up with the incoming requests and at some point simply could not due to hard limits set for the amount of requests a load balancer should be allowed to serve. In the end the load balancers stopped serving requests since the were simply not allowed to open further connections to the API workers. So we set the workers free and allowed more connections to be handled by each, monitored the situation for a while and decided, all is fine in Kraken.io Land.
Spoilers: it was not. An hour later requests were piling up again and the API was refusing to service anything. It turned out requests were hugely delayed because of unanswered DNS requests. Since we use a private DNS infrastructure this was rather weird. Internal requests were being answered within acceptable times just fine. But: external requests were no longer answered. At this time, most of our systems started reaching the End-Of-Life for cached DNS requests. For the outside world, this culminated in kraken.io no longer being available, while for our private infrastructure the rest of the world ceased to exist.
Further investigation showed that our own DNS servers were no longer receiving DNS zones from upstream servers, e.g. us asking for s3.amazon.com on our upstream servers got denied. A few cables further the issue was finally spotted: the DNS systems within the data center - to which we turn when requesting information about public DNS zones - were no longer available.Mitigation efforts undertaken:
In computer networking, IP address spoofing or IP spoofing is the creation of Internet Protocol (IP) packets with a false source IP address, for the purpose of impersonating another computing system.(92 kB)