An engineer from a South African internet service provider accidentally deleted critical network settings while hurrying out for a smoke break, resulting in the largest internet outage on the African continent at that time.
Decades ago, one of South Africa's leading internet providers faced an unprecedented crisis caused by what seemed like a trivial mistake made by an employee. An engineer named Paton worked as a "core network engineer," a position that demanded high levels of responsibility and attention to detail.
The company where Paton was employed played a crucial role in maintaining internet service not only in South Africa but also in neighboring countries. The provider's DNS servers were authoritative for thousands of domains, including the national top-level domains of several African states.
One day, Paton was assigned to update network blocks
A network block is a part of the IP address space allocated for use within a specific network or subnet. and propagate them through BGPBGP (Border Gateway Protocol) is the main routing protocol between autonomous systems on the internet that allows the transfer of routes between different networks. to partners and transit providers. This involved editing access control lists (ACL)An ACL (Access Control List) is a list that defines the access rules to network resources for various users or groups of users. that regulated users' and domains' access to certain network resources. Usually, Paton performed this task meticulously, but this time his colleagues called him out for a smoke. His desire to join them made the engineer rush.
Upon returning from the break, Paton found the office in complete chaos. The network operations center was inundated with calls from irate customers. It turned out that the largest internet outage on the African continent had occurred.
The situation worsened when an unidentified individual claiming to be a hacker contacted a local tech publication and announced their involvement in the incident. This news spread quickly, creating additional problems for the company's management.
The investigation revealed that there had been no compromise of security systems. In his haste, Paton inadvertently replaced all existing access control lists instead of simply adding the new network blocks. This led to the failure of the intricate internet traffic routing system for a significant part of Sub-Saharan Africa.
After the incident, Paton not only restored the ACLs and updated the network blocks but also developed the company's first change management protocol—a set of rules and procedures governing the process for implementing changes in IT systems to avoid incidents or outages.
Source: TheRegister
Comments (0)
There are no comments for now