Vendor or Partner Dependencies: A Common Bottleneck
The cursing could be heard outside the room again. The partner had botched another launch, and everyone’s pager was ringing off the hook. Helplessly, the team assembled and did the normal ritual checking the APM for all systems, and low and behold the external payment processor that bank forces us to use is down again. This time the intern won the bet for MTTF at 60 minutes. No one really wins this bet.
“I’ll call Jerry,” said the Scrum Master.
Jerry was the architect and program manager for the payment processor. He was as great guy, even when his voicemail picked up, he seemed chipper. The official title was DevOps Manager, accountable to make sure the systems were always up and running. It was a simple role, after QA validated the code, he moved the batch of code to the release branch and deploy.
Over the past couple of months, the batches were getting bigger which forced Jerry to blindly deploy code, and hope.
“He’s not answering, he’ll call us – he always does, I’ll text our VP. In the meantime, I’ll expense us pizza for lunch while we wait.”
A half hour later, 2 pizzas arrived for the team carried by a familiar face, the VP of App Dev, Carolyn. “So let me guess, Jerry’s not answering, they did a big bang release, and we’re queuing up payments, but no one on the business side has noticed yet because we’re still writing business, just not getting paid?”
The room was silent, this happened every first Friday of the month. The same thing. The same problem. It was a routine surprise failure. Strategically, the partnership with the bank made sense, in lieu of more favorable terms for working capital. Last year, an agreement was made to ensure the company got the working capital required and the bank would process all payments for the next 6 years. This was free pizza party 9 in the past 30 days, each taking longer and longer to resolve, getting closer and closer to impacting sales and ultimately cashflow, and the team sat idle for days – just waiting in case they needed to jump in.
To not miss a coaching opportunity, Carolyn asked the group, “While we’re here, let’s play the hypothetical game of being the CIO of the bank. What would you do?” asked Carolyn.
“Retire.” Said someone in the back, and the room broke into laughter. The Scrum Master threw out post notes to everyone to write down their ideas, and the room went silent as the team worked toward fixing the hypothetical system.
After the noise of pens quieted, Amit, one of the senior developers on the team stood up and started reading off the solution. “Upskill the developers,” “Deploy more,” “Test More…”
“Unless we help our partners, the next 6 years will be risky.” Carolyn interrupted, “Who wrote the last one?” “I did,” responded Carl. Carl had joined the company a few years ago during their digital transformation. He was a vocal leader and help drive change back then. “They’re no different than we were when I joined. Carolyn – weren’t you the DevOps Leader here? And if we don’t help them, there’s no technology in the world that can fix it on our side. We’re tightly coupled with them – so let’s help because I don’t think we can fire them. Either we can sit idle 10 days a month waiting for their storm to pass, or we can jump in and help!”
Carolyn was surprised by the idea, but it made sense. There was 5 years of strategic partnership left, and it was trending toward disaster which neither side desired. “Carl, can you join me for my one-on-one with our CIO next week?”
Dependencies take all sorts of forms; and a common one is vendors or partners. When dealing with any dependency you isolate and control it, but that only gets the organization so far. Eventually, the dependency will become the bottleneck. If you can’t fire your vendor or buy your vendor, the only thing you can do is partner and help your vendor or live through the turbulence together.