"Sagrada Familia in the IT industry" … A project of Mizuho Financial Group (FG) that was so ridiculed. The complete restructuring of the core banking system, which is directly linked to banking operations, has been a long-cherished wish for Mizuho FG for 19 years. This book takes the form of going back to the past from the present when the big project was completed, and approaches the whole picture of the huge project based on interviews with a wide range of stakeholders from the top of the bank to field members and IT vendors.
Part 1 introduces the whole picture of the new system MINORI and Mizuho FG's new digital management strategy. In the second part, we will look at the 30 mistakes and causes of the large-scale system failure that occurred in 2011, which was repeated by the management team and the information system department. In the third part that follows, we will return the hands of the clock further and look back on the first large-scale system failure in 2002. Then it becomes clear why it was such a long-term and huge project.
Please come back to the beginning after reading the last page. You will be able to feel the "growth" of a huge organization. It's not someone else's affair. How was Mizuho FG, which failed many system development projects and caused two large-scale system failures, able to rebuild its company and succeed? The essence will be a learning experience for any company or business person.
The main points of this book
Mizuho FG completed the construction of the world's largest core banking system in July 2019. It was also the moment when Mizuho FG's long-cherished wish was achieved after overcoming two system failures.
The two system failures were not just faulty in the information system department and system development companies. The root cause is the lack of leadership of top management, who neglected IT systems and postponed investment decisions.
Mizuho FG has developed a large number of IT human resources through the system renewal project. Mizuho FG is in a position to take the initiative in the digitization of the financial industry, which is already in full swing.[Must read point !!] A huge project for the construction cost of 7 Tokyo Sky Trees
Too heavy waist
The project to build the accounting system "MINORI" of Mizuho Financial Group (FG) was completed in July 2019, spending eight years and costing more than 400 billion yen. A core banking system is an information system that supports banking operations such as deposits, loans, and transfers, and is the heart of a bank. By the time of its full restructuring, Mizuho FG has experienced two stages of system integration and two large-scale system failures.
In 1999, the former Dai-Ichi Kangyo Bank, the former Fuji Bank, and the former Industrial Bank of Japan announced the integration. From April 2002, the former Mizuho Bank for individuals and small and medium-sized enterprises and the former Mizuho Corporate Bank for large enterprises have been established. This is the predecessor of Mizuho FG. After announcing the business integration, the company first aimed to reorganize the core banking system unique to the former three banks into two. However, the reorganization of this old core banking system did not proceed slowly, causing the first major failure on April 1, 2002. In the end, the reorganization was completed in December 2004, more than two years behind.
Mizuho FG repartitioned system development and aimed to complete the renewal of the new core banking system by FY2011. However, the preparation was carried over from 2006 to 2010. The Great East Japan Earthquake struck in 2011, when full-scale system development began. A large number of donation transfer requests flooded the old core banking system, causing a second large-scale system failure.
Why couldn't the system be renewed by the second major failure? Mizuho Bank used the dilapidated system of the former Dai-Ichi Kangyo Bank, which was developed in the late 1980s. I had a sweet perception that such an old system would "still be used". Without some chance, it is difficult to judge a huge system investment of 400 billion yen. For Mizuho FG, the second large-scale system failure triggered a full-scale move to a new system.
What is MINORI
MINORI is Mega Bank's first core banking system that fully adopts SOA (Service Oriented Architecture). By combining applications and their functions as independent "services", the impact of system changes can be suppressed as much as possible. As a result, the application development period can be shortened by about 30%, customer management using a single customer number can be performed from anywhere in the store, and centralized transfer request bookkeeping can be changed from a batch method to an online method for each case. At the same time, it has become possible to enhance functions and reduce costs.
Management of 1000 participating vendors
About 1000 IT vendors participated in the development of MINORI. This is equivalent to about 10% of Japanese IT ventures. The four major vendors, Fujitsu, Hitachi, IBM Japan, and NTT DATA, played a particularly important role.
In development, the requirement definition of "AS IS", which is a standard system development and follows the current business flow, was completely prohibited. I wanted to avoid the user department becoming "just as it is". The user department was asked to take an inventory of banking operations from scratch and then think about the ideal business flow.
The wisdom and know-how of each vendor were gathered by establishing a special conference body at the beginning and end of the project, and discussions were held on the optimal system. Even so, the project was difficult and the development completion time had to be postponed twice. It was also a manifestation of Mizuho FG's determination to never cause a system failure.
Erase the remnants of the 80's
One year system migration
No matter how good the quality of the system is, if you make a mistake in the transition phase from the old system, the difficulty until then will be a bubble. Mizuho FG made every effort to move to the new system, shutting down ATMs a total of nine times, and taking a full year.
With the transition to MINORI, the administrative procedures of the branch offices have changed significantly. From the first day of the transition, in order to run the work smoothly, about 17,000 clerical staff were given classroom training, and all 400 stores rehearsed 6 times each. The instructors who visited all the stores were mainly women in specific jobs (general jobs) who worked at the sales offices. Until then, employees from branch offices often acted in response to instructions from the headquarters, but many said that they became the driving force behind the MINORI project, which was a major turning point in their careers. They are now struggling to be indispensable for the launch of next-generation stores.
To next-generation stores
Mizuho FG has set a goal of reducing IT costs by a cumulative total of 72 billion yen in its five-year plan from 2019. The floating costs are intended to be invested in the future, mainly in the digital field.
At the same time, we will work on reforming sales offices and office work. The bank's business flow depends on the core banking system. The inefficient business flow tied to the old system remained on site. What was particularly problematic was that the application received at the branch office basically had to be processed at the branch office. Therefore, the branch office required a lot of clerical staff.
With the introduction of MINORI, customer information management has become possible for all stores, and most of the clerical staff can be consolidated from the sales office to the office center. In the vacant store space, there will be a corner for recommending financial products and a corner for meetings with corporate customers. It's a next-generation store.
In this way, the shackles of the 1980s disappeared from Mizuho Bank.
Offensive Digital Management Strategy
The banking industry is gaining a presence with new players such as AI, fintech and cashless. The Financial Services Agency is also willing to digitize financial services. In such a financial industry, whether Mizuho FG can exert its presence depends on being able to master MINORI. This is because MINORI makes it easier not only to reduce costs but also to put new business ideas to practical use.
In addition, in order to make a digital transformation, it is necessary to have abundant IT human resources who have been trained in battle. In that respect, Mizuho FG, which has experienced the world's largest system development project, has developed IT human resources who will be responsible for the future.
Z Holdings and LINE, which own Yahoo, announced a business integration in November 2019, but both are technology companies that are trying to bring new financial services centered on payments. Mizuho FG, which has a close relationship with these two companies, is in a position to take the initiative in the domestic financial field.
Second large-scale failure immediately after the earthquake
Black box system
The trigger for the development of MINORI was a large-scale system failure after the Great East Japan Earthquake. It all started when a large amount of donations exceeded the processing limit. A big problem occurred when processing transfers after 3:00 pm at night so that they could be deposited the next day.
It is common to set an upper limit on the processing that can be done for one account. This is because the system can be used safely and without waste. However, the person in charge at that time did not even recognize the existence of the person who was enrolled at the time of the start of operation 23 years ago. Data that exceeded the limit caused an error, and it took 8 hours to restore the data alone. Due to this loss, the obstacles will spread further from the next morning.
The automatic processing stopped, and I had to manually enter the execution command. It is a situation where mistakes are likely to occur. However, Mizuho Bank did not incorporate an abnormal scenario into the system and did not prepare an operation manual. In the midst of a failure, it is impossible to create a scenario with no mistakes without testing, and at the same time, manually enter execution instructions without making mistakes.
As a result, the effects of system failures extended to the total shutdown of ATMs. It was 10 days after the occurrence of the system failure that the unloaded transfer process was completely eliminated. During that time, the number of transfers for which payment was delayed exceeded 2.2 million in total, including those addressed to other banks and own banks.
Not just the responsibility of the system department
The "System Failure Special Investigation Committee," a third-party committee established by Mizuho Bank after the failure was resolved, found 30 inadequacies such as delays in management decisions and mistakes by system personnel. The FSA also pointed out that this obstacle was not just a problem for the systems sector.
Most of the direct causes were due to the system department, but the root cause was the disregard for IT and lack of understanding of Mizuho Bank and Mizuho FG management. The core banking system needs to be renewed from a long-term perspective. However, he has postponed large-scale renovations because he hates the large cost and risk of failure.
After that, Mizuho FG worked on recurrence prevention measures for a year. The symbol is the creation of a data flow diagram to grasp the overall view of the system. Until then, specifications for each system had been prepared, but this was the first time that we focused on the data flow and visualized the entire system. Since then, this flow chart has been revised every time the business flow changes.
The beginning is the first large-scale system failure
Leader IT neglect
In the first place, anxiety about leadership began several years before the first major system failure occurred in April 2002.
It was in August 1999 that the former Dai-Ichi Kangyo Bank, the former Fuji Bank, and the former Industrial Bank of Japan announced their management integration policy. He explained that while pursuing thorough rationalization through integration, he will aggressively invest about 150 billion yen a year in IT to compete with the US banks.
However, the integration of information systems, which is the key, has been difficult from the beginning. The powers of the three banks antagonized, and they could not unify the operations for integration and show a clear management strategy in the first place. The subcommittee was entrusted with the examination of the integration policy without clarifying the management decision. Since the field department is attached to the system that they have developed for many years, they desperately try to keep their own system. In the end, it fell into a barren functional comparison, and the composition was such that Dai-Ichi Kangyo Bank, which pushes Fujitsu machines, and Fuji Bank, which pushes IBM machines, are in direct conflict.
With the intermediary of the Industrial Bank of Japan, Mizuho Bank's accounting system eventually adopted the Dai-Ichi Kangyo Bank's system, and the accounting peripheral system and information system of Fuji Bank. The information system review work, which took four months after the announcement of the integration, created a decisive gap between Dai-Ichi Kangyo Bank and Fuji Bank's system personnel. After that, the three banks developed only their respective areas of responsibility and did not have a person in charge of overlooking the whole.
Fall from the first day of release
Then, at 8:00 am on April 1, the defect surfaced immediately after the start of use of Mizuho Bank's core banking system. The accounting system of the former Fuji Bank has been separated from the ATM cooperation network between the metropolitan banks. When the amount of data to be processed increases and multiple conditions overlap, a system malfunction that was previously invisible may occur. Therefore, we have to repeat the test for various cases many times. However, due to the delay in system integration, we could not spend enough time for testing.
Although we managed to suppress the problems such as ATMs associated with this, we could not resolve the delay in the account transfer processing well, and the maximum number reached 2.5 million. Not only Mizuho Bank staff but also IT vendor employees were mobilized to re-enter the data without sleep. Mizuho Bank's fund transfer failure processing will take until April 18.
Recommendation of reading
Mizuho FG President Tatsufumi Sakai is said to have visited the data center first after taking office in 2018. The phrase "the system is the heart in the financial industry. It has a very high priority" echoes. When I read this book, it's not hard to imagine the tension and struggle in the field, so it's exciting to think how much the words and actions of the top management encouraged the people in charge of the field.
Most Japanese companies have a "core system". Many companies built a core system around 2000 and continue to use it as it is, and it is safe to have a system failure at any time. It is also said to be the "cliff of 2025". The attitude of the management is questioned as to how to face and build the aging core system. It can often be read from the attitude of Mizuho Bank clarified in this book.