GrayINT - Detection and Localization of Gray Failures via Hybrid In-Band Network Telemetry

As the underlying infrastructure, data center networks (DCNs) provide an important platform for cloud computing and next-generation networks. However, due to the influence of many factors, such as network, hardware, and software, various failures may occur during the operation of the DCN, which brings great challenges to its reliability. Particularly, a more insidious type of network failure, known as gray failure, presents a significant threat to system stability. It is often precipitated by subtle, underlying failures which can be difficult to detect. In response to this challenge, in this work, we propose GrayINT, a gray failure detection and localization system that leverages In-band Network Telemetry (INT) technology to monitor all paths within the data center network in real-time. GrayINT offers an efficient and comprehensive framework for detecting and localizing gray failures, enabling rapid identification of device and link failures and corresponding responses. To verify the feasibility of our design, we build a fat-tree data center network on a virtual network testbed. A large number of experimental results demonstrate that our system can monitor device and link status in real-time, achieving gray failure detection and localization within seconds.