PIERRE-State government's Internet and technology network suffered a full outage last weekend.
The system went down for about 20 hours. The first trouble came about 9 a.m. Saturday with reports of intermittent problems such as slowness and unavailability.
Within the hour, people using the Internet to visit state government websites arrived at blank page views.
Businesses couldn't record motor vehicle sales for the state Revenue Department.
Behind the scenes, bigger problems ran throughout state government's system. Staff for BIT and a vendor worked throughout the rest of the day and all night.
ADVERTISEMENT
Services completely were restored at 5:30 a.m. Sunday. It was the longest outage ever for state government's system.
What caused the breakdown remained a mystery as of Wednesday.
David Zolnowsky, commissioner for the state Bureau of Information and Telecommunications, said in an email response to questions that an interface within the storage infrastructure stopped working.
"This failure caused statewide systems and services to stop. Web content, business applications, wireless access and work drives were not available. Email remained operational," he explained.
Zolnowsky said redundant services built into the storage system didn't operate as designed.
"Beyond the failure of the interface, the details remain under investigation. An anomaly still under investigation triggered the system to invoke data preservation protocols," he said.
Web services and business services share the same underlying storage system.
"A failed hardware component was replaced to bring the system back to full operational status. Upon replacement, multiple internal processes were automatically invoked to verify system integrity," Zolnowsky said.
ADVERTISEMENT
The vendor is Hewlett Packard Enterprise. He described the company as "a reliable and responsive technology partner."
Zolnowsky said BIT is "actively engaged" with the vendor's engineers and management trying to understand "why the interface failed and why that failure cascaded through the redundancies that were supposed to prevent such a failure."
