Canada’s Phoenix Payroll Catastrophe

This is a story about one of Canada’s most amazing blunders. It's a story about a multi-billion dollar failed project.

First, what is Scrum?

• Scrum is a framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value.

• Scrum is a framework founded on empirical process control theory which has been used to conduct work on complex products since the early 1990s. Worldwide, in varied marketplaces/circumstances, and often in extremely difficult conditions, teams which employs it have proven that Scrum helps them to deliver extraordinary business results and to create excellent work environments.

Scrum’s design implies the following:

1. First, Scrum describes Iterative & Incremental Development patterns which are ideally suited for Product Development.

2. Second, Product Development includes activity aptly described as complex adaptive problem-solving.

3. Third, Scrum encourages a bias toward action. (As opposed to a bias toward analysis and planning in light of conjecture. I.e “Upfront planning”.)

4. Fourth, the best way to mitigate risk, improve predictability, and ensure high-quality work product, is to trust self-organizing, cross-functional teams who employ brief cycles of planning, implementation, reflection, improvement. (Plan + Do + Check + Act. In other words, Iterative & Incremental Development.)

5. Fifth, waterfall is inappropriate. Especially in IT which was its intended purpose, but speaking more broadly, it’s inappropriate for any complex work. Even Sir Winston Royce, the author who coined the term, asserted that sequential/phased delivery “is risky and invites failure” ^source, especially in large software projects.

And the reason for this article, the message I’d like to relay is: the waste caused by employing Critical-Path Planning techniques (such as conventional Project Management patterns like the 'Waterfall') is a problem for the world. The cost of failed and challenged projects worldwide is estimated, annually, in the trillions.

Now, the Story of Tremendous Failure: “Phoenix Payroll System of Canada”

In 2009, the federal government of Canada hatched an idea to update all payroll systems for federal public employees. But rather than update the systems, it was decided they would build a brand-new system from scratch and then decommission all the old systems. (One system to rule them all!) Bureaucrats got behind the plan to make government “smaller and more efficient” by centralizing the payroll operations into a single office and a single IT system.

If you listen closely, you might hear the faint echoes: “A flick of a switch”, they said. “We’ll save millions”, they said. “We’ll launch the new system with just the push of a button”. And some fool decided to name the new system Phoenix – it would “rise from the ashes” they said.

In 2011, the government awarded a 30 million-dollar contract to IBM. (And in case you didn’t notice the dates, it took 2 years just to decide on a vendor.)

By 2015, that budget had long been spent (and extended and spent again!) but the system still hadn’t seen the light of day. IBM was reporting critical problems with the system. But an election was pending!! So, in April 2015, Prime Minister Stephen Harper signed the deal to construct a new headquarters in New Brunswick (promising 500 jobs) and he demanded that the Phoenix system go online by December.

In January 2016, news broadcasts about the disaster filled the airwaves and, as if it wasn’t already bad enough, a massive security flaw was exposed that allowed widespread access to employees’ personnel records. The personal details of 300,000 employees were leaked.

Then, against all logic, the Canadian government was determined to “enjoy the savings” promised by the new system and they proceeded to lay off thousands of payroll staff across the country. (Given the disaster, why would the government proceed with lay-offs at this time? The simple answer is that the business case for the project was approved on the basis that these staff would not be needed; so, for senior directors of the project to recieve their bonuses…et cetera.)

But Wait, It Gets Worse!

March 9, 2016, was Phoenix’s first payday. How did it go, you ask? Thousands of employees didn’t get paid the correct amounts. Some were paid too much, others too little. Income taxes were calculated incorrectly. Many didn’t get paid at all. Within weeks, employees were dipping into their retirement savings, credit cards, and defaulting on their mortgages to make ends meet. An incredible disaster.

The government’s FAQ, below, provides hints of the myriad problems caused by the system.

Just imagine the avalanche of problems leading this question:

“I was underpaid by the system and we had to withdraw my spouse’s RRSPs to keep our house. Now we have to also pay income tax on those RRSP withdrawals?! Are you kidding me?!”

By June of that year, the government started to throw people at the problem. 100 employees were hired in Quebec; the government started sending letters to recently laid-off payroll staff to return to work; and lawyers sharpened their pencils.

Then the numbers started flowing in…oh, the wonderful world of estimates. “The defects could cost $20m to fix and will be done by October.” Then $25m. Then $50m. Senior officials then started “stepping down” or taking leave for “family-related reasons”. Then $70m previously earmarked as savings were reallocated to fix the mess. Then another $140m. Then another $142m. These funds were to hire staff. (Hush, let’s not even talk about the next decade of litigation!!)

As of August 2017, the most recent publicly-available report, only 49 per cent of payday transactions were being handled accurately by the system.

A Hard Lesson to Learn

Here’s the lesson to be learned by this disaster: in 2016, federal government officials were recorded saying they encountered “unanticipated complexity” in rolling out the new payroll system.

“UNANTICIPATED”!?!?

That is an incredibly stupid statement. It implies that the decision-makers involved believed that they could anticipate all the complexity. That statement implies that in some alternate reality, if only they had conducted more analysis, if only they had produced a better plan, if only the project managers could see deeper into the unknown unknowns of complexity, just maybe they’d have devised a plan of such perfection that millions of lines of code, at the flick of a single switch, would instantaneously and perfectly execute for all eternity.

But that’s not reality.

Anyone who has ever worked on a multi-million-dollar software product knows that analysis leads only to assumptions, and assumptions are risky. If we’re smart, we do it differently:

1. We plan for frequent delivery of small-batch, high-quality, well-tested code;
2. We avoid long-term plans based on conjecture;
3. We do not segregate the learning of requirements from implementation of those requirements;
4. We do not segregate the implementation of those requirements from testing those requirements.

And, mind you, this is a story of just ONE failed project. One of hundreds of projects in Canada. Of thousands worldwide by governments and enterprises every year.

Why Am I Angry?

I am angry because we know how to prevent these disasters!

The results of Scrum are known (to some). By studying the results of tens-of-thousands of projects worldwide each year, research from Standish Group, Forrester, Gartner, KPMG, VersionOne, Scrum.org, Agile Alliance, Project Management Institute, Forbes, Harvard Business Review, NASA, IEEE, (basically everyone!!) it is well-known that Iterative & Incremental Development methods (such as Scrum) are more effective, less risky, and produce better outcomes than sequential/waterfall approaches. Yet, Canada's treasury board still awards contracts based on fixed-scope, fixed-timeline budgets. The fiscal habits and stage-gating of government projects is 100% to blame for these failures.

The Chaos Report by Standish Group is an oft-cited survey of +10k projects worldwide which compares success rates of Agile versus Waterfall methods. Had anyone in the Canadian Federal Government cared to look, they’d know that the best way to mitigate risk in such a complex environment is to employ Scrum.

Scrum is important to the world because it describes a way of working which helps organizations/teams to drastically reduce risk and successfully deliver complex projects and products. Consider the trillions spent annually to resuscitate failed projects; consider all the ways those funds could be used in the world.

If only the money wasn’t continuously drained into a Waterfall.

This article also appears at:

• LinkedIn