Shock, anger and action as Kakao’s tech fails dramatically

The dramatic failure of Kakao services over the weekend left users wondering about a company on which they have become dangerously dependent and the government hinting of intervention into what has become critical infrastructure.
 
Kakao servers were knocked out as a result of a fire at a Pangyo data center that started Saturday afternoon. The company’s messenger services and services offered by related companies were unavailable, degraded or intermittent for anywhere from 10 hours to 2 days as backup systems proved slow and ineffective.
 
Time stood still for many people in Korea as they were cut off from friends, colleagues and customers.

If Kakao’s services have become national infrastructure, “then the state must take necessary measures for the good of the people,” President Yoon Suk-yeol said Monday on the way to work.
 
KakaoTalk, the company’s messenger app, is used by 90 percent of Koreans and is becoming increasingly popular overseas. Business is conducted with it. Companies employ it to keep workers in touch with each other. Institutions, such as hospitals and the Post Office, use it for first line communication.
 
When it went down — an event communicated by the company via Twitter — lovers were cut off from each other and small businesses were unable to take orders.
 
Related services, such as the Kakao T ride-hailing app, Kakao Games and Kakao Bank, were also affected, though to different degrees.

In the wake of the almost complete collapse of its core business, Kakao’s management has been vague about its disaster planning, declining to reveal the number and locations of its other server farms.

It is not clear whether the company employed a “hot-site” backup plan, whereby mirrored servers stand by at the ready and can be brought online in a short period of time. With cold-sites or warm-sites, recovery takes time.

When asked whether Kakao’s sites were hot or cold, company public relations representatives were not able to provide an answer immediately.

Namkoong Whon and Hong Eun-taek, Kakao co-CEOs, explained in a press release Saturday that while the company had taken precautionary measures, the possibility that all servers would go down at once was unforeseen.

“We have backed up our data in multiple data centers across the country and have a dualized system to deal with glitches,” the statement said.

In a dualized system, if one server fails, another kicks in.

“We started the dualization procedure immediately after the fire. It is rare for one data center to affect all services, and it is taking longer for us to restore systems completely.”

The fire broke out in the SK C&C building, which houses servers for Naver, SK Telecom, SK Broadband and IBM. Naver reported partial disruptions of some services, including its search engine, news, blog and online shopping, but they recovered several hours later.

“Naver keeps a certain portion of its data centers vacant,” said a tech company executive. “The company leaves its servers turned off so that in case of an overflow, it can be redirected to other servers. But the problem with Kakao is that its data was concentrated too heavily in one place.”

Kakao has no data center of its own and is currently colocating its servers at server farms owned by others.

The fact that the backup did not immediately kick in could be because Kakao runs its servers very tightly without room for a contingency plan, according to tech executives. KakaoTalk reported numerous minor glitches this year.

Kakao started construction of its first data center last December, in Ansan, Gyeonggi. It is scheduled for completion by the end of next year. The company signed an agreement last April to build a second data center, on Seoul National University’s Siheung Campus in Gyeonggi. Construction is set to begin in 2024 and finish by 2026.

President Yoon told local reporters Monday morning that the government will look into whether Kakao has monopolized some services — and whether changes are needed.

He “respects the rules of a free market that guarantee a company’s freedom and creativity,” but “only while the market runs on a reasonable and fair system.”

“The majority of people who use Kakao would have suffered inconveniences due to the service freeze over the weekend,” Yoon said. “It is a network run by a private company, but it’s virtually the same as a state-run network in the eyes of the people.”

The Yoon administration will establish a team under the National Security Office dedicated to preventing a reoccurrence of the massive disruption.

Members of the Science, ICT, Broadcasting and Communications Committee of the National Assembly agreed to summon the leaders of companies involved in the meltdown. The Financial Supervisory Service (FSS) said it is also looking into the failure.

Most major Kakao services resumed as of Monday 9 a.m., including KakaoTalk, Kakao Map, Kakao Pay, Kakao Webtoon and Kakao Map, but minor glitches were still reported.

Users could message and send pictures to each other on the KakaoTalk app but could not use the “cupboard,” which stores data sent in the chatroom in the past. The KakaoTalk Channel service, a marketing and advertisement page for companies, was still being fixed.

Kakao Bank said its service was fully recovered Monday. Following the blaze, some of its services were halted, including logins, alerts and money transfers through KakaoTalk. Kakao Pay and Kakao Pay Securities also generated errors for users. They were still recovering as of Monday morning.

The impact on the financial apps was not as big as for the KakaoTalk messenger app because they have separate data centers.

Kakao Mobility’s Kakao T navigation and taxi-hailing service was restored but not the rental service for electric scooters and bikes. The Daum portal site was also back, but email was unavailable.

“The Daum and Kakao email and Talk Channel services are taking time due to the complexity of the restoration equipment,” Kakao said in a notice Monday morning. “Many services are stabilizing, but we ask for your understanding that some may be delayed depending on the traffic.”

Some Kakao companies promised users compensation.

Kakao Webtoon and the Melon music service will extend subscription benefits by three days, Kakao Entertainment said Sunday.

Users who paid to read webtoons and web novels on Kakao Webtoon and Kakao Page will be given three additional days to view the content that expired during the malfunction.

Regular Melon subscribers, who pay on a monthly basis, as well as those who bought temporary benefits, will be given three extra days of service.

“All services have been restored as of 7 a.m., and no data has been lost or damaged,” a notice on Kakao Webtoon said on Sunday.

Kakao Mobility promised compensation, but hasn’t come up with a plan yet. The company, which operates Kakao T and a navigation service, is still trying to restore full services.

Kakao Games is also discussing compensation plans.

The situation may be a little more complicated for the KakaoTalk messenger. Although almost everyone in Korea uses the app, the fact that it is offered free makes it difficult to calculate exactly how much people have “lost” due to the glitch.

“We will look thoroughly into what damages have been caused and how we can compensate after we get through to the bottom of the situation,” said Yoon Seung-jae, PR manager at Kakao.

Kakao established a command center led by Hong Eun-taek, co-CEO of Kakao. The task force will be divided into three teams, investigating the cause of the malfunction, managing the current crisis and compensation.

The company will set up a separate channel where people can report damages.

BY YOON SO-YEON, JIN MIN-JI, PARK EUN-JEE [yoon.soyeon@joongang.co.kr]