Cisco Outage
On Thursday, October 10, 2019 a network outage hit Cisco Systems affecting their entire network. A Cisco spokesperson said in an interview with
TechCrunch that the disruption was traced to an internal system change. The problem, however, isn’t the outage it’s the absence of a disaster recovery plan. The outage lasted for about five hours and prevented users from accessing Cisco’s learning portal, downloading software, signing on, and slowed down response time to support tickets. The company’s blog also suffered from the outage. WordPress’ install page appeared on the blog and could not be accessed. Downtime started at about 2pm GTM and lasted for approximately an hour. It wasn’t a security threat per say but it was an odd sequence of events that could provide insight as to the internal flaws inside the company. The company tweeted later in the evening, “We have traced our disruption to an internal systems change, and are in the process of being fully restored.” HTTP 500 error displayed on the website. The curious aspect of this story is the message that presented itself on the failed page. “Please contact the server administrator, it-webmasters.cisco.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.” Cisco hasn’t opened up about what exactly happened and how, saying on twitter: “Cisco is continuing to address the disruption to our IT systems.” However, there are things to be learned from this event. This outage was a huge issue for Cisco, a large company, but it’s much more common among smaller tier organizations. A business needs power to run, obviously. However, unexpected outages create situations from which it’s difficult to recover.
Outages aren’t Uncommon
Lloyd Banking Group, a British retail and commercial bank branching across England and Wales, also experienced an outage in February of 2019. Customers could not log in to their bank accounts. Furthermore, IT outages in the banking sector are becoming more of an issue, not only for banks but for their customers. An outage prevents customers from accessing valuable financial information and regaining control after such an outage is a logistical nightmare. If a large companies such as Cisco or banks like Lloyd Banking Group can experience outages then it can happen to anyone. Outages and downtown can mean a host of services rendered useless and inaccessible. That is why it’s important to get your downtime as low as possible. 9/10 data center professional professionals say that company management is more about lessening downtime than it was in previous years. Cisco stores its bits and updates firmware internally. In other words, it handles its IT processes internally. Publications like Tech Republic suggest the company would be better served adopting a cloud platform. If Cisco had the company blog hosted on the cloud or separated from production systems, they could have communicated their network status during the outage.
The Cloud and Disaster Recovery
Cloud solutions for financial businesses is not new, but financial cloud computing has one particular advantage that mitigates downtime. See, if an outage impacts a business, it should have a plan for accessing its data. Institutions who adopt a financial cloud service have the advantage of accessing their files and backups from anywhere. Cisco failed to plan accordingly beforehand and there was no way to communicate to its users and customers of the outage, leaving many confused and disgruntled. Hosting their blog on the cloud could have prevented this. If your facility is compromised by an outage, recovering data stored in house requires waiting for the business to regain access to the facility and hoping nothing was lost. However, a hybrid cloud approach to disaster recovery reduces risks of downtime. Some cloud providers, like Nerds Support for example, offer Disaster Recovery as a Service. Meaning, that there are financial cloud service providers, not all, that assist with developing a strategy and recovering data stored on the cloud servers.
Disaster Recovery Plan & Downtime Prevention
The cost of downtime rose 60 percent from 2014 to 2016 according to the Aberdeen Group. In other words, cost went up from $164,000 per hour to an average of $260,000 per hour across all industries. Although costs are incredibly high, downtime prevention is often overlooked. As we’ve seen above it doesn’t matter how large or small your firm is, it’s incredibly important to have a business continuity and disaster recovery plan in place. If you or your staff don’t know what to do in the event of an outage, things could spiral out of control. In the UK the Treasury committee launched an inquiry into banking IT failures. Last year, TSB bank in England also experienced an IT outage so disastrous the CEO, Paul Pester, was forced to quite. Even if an institution has the most advanced cybersecurity and backs up their data regularly, they’re still at risk of experiencing downtime if the infrastructure and environment isn’t monitored properly. Another reason that downtime is a problem for your firm aside from the financial loss is data loss. Critical data loss often occurs during downtime. Over three-quarters of executives experience serious data loss caused by downtime, according to a 2016 poll.
Is your Firm Vulnerable to an Outage?
Ask the following questions to determine whether your business is properly protected and has the necessary oversight:
• Is there a time when the digital environment is unmonitored? • How long has it taken to recover from downtime in the past? Was it past a certain time frame? • Are there sufficient IT personal with the required experience to manage software maintenance and respond to threats? Some businesses and financial institutions have the resources needed for constant monitoring. However, those who don’t should consider adopting a financial cloud infrastructure. Cloud service providers should provide 24/7 support, detailed service level agreements and experienced systems engineers. You can avoid downtime when you have the proper tools, guidance and expertise. Achieving maximum up-time is possible with the cloud. Visit the
Nerds Support website to read more about Downtime prevention, cyber security, or cloud compliance.
