The Bug That Wasn't Ours
Payments through Xendit kept failing, whether adding a card or completing a transaction, with nothing in our logs to explain why. It took a full day working alongside their engineers to find out.
While working on Horego, QA flagged something strange with the Xendit SDK during checkout. Adding a card would fail. Completing a payment would fail. Nothing in our own logs pointed to a clear cause.
The first instinct in a situation like this is to assume it's your own code. I wasn't convinced. Sometime earlier, I had asked our backend team to start logging critical events into Kibana, mainly so issues like this could actually be traced with data instead of guesswork. That decision paid off here. I went in and started tracing logs specifically around Xendit's calls, trying to find anything that pointed away from our side of the integration. Eventually I found it: the pattern in the logs strongly suggested the failure was originating from Xendit's SDK itself, not ours.
I filed a support ticket with the evidence. It didn't stay a ticket for long. Xendit's team came back wanting to pair directly, and our VP of Engineering trusted me to lead that collaboration on our side. We got on a call together, screens shared, going through it live.
We traced it down to a conflict between the Xendit SDK and Microsoft Clarity, which we had running for heatmaps and session recording. The two were interfering with each other in a way neither team had ever tested for.
That was really just the start of the day, though. Once we had a working theory, it turned into a back-and-forth of debug builds: Xendit's team sent over a patched SDK, we tested it, found it wasn't quite right, and they sent another. By mid-afternoon, we had confirmation that a proper fix was ready and stable.
It wasn't a bug in our code, and it wasn't really a bug in Xendit's code either. It was two SDKs that had never been tested against each other, quietly overlapping in a way neither team had reason to expect. The fix shipped that same day as Xendit Android SDK version 4.2.3, released January 24, 2025, addressing what their changelog describes as a "3DS authentication edge case," the same window our issue fell into. It rolled out to all of Xendit's merchants, not just us.
I think about this one more than some of the flashier fixes I've shipped. There was no dashboard graph to point to, no feature to demo. Just a full day of two teams passing debug builds back and forth, ruling things out one at a time, until the real cause finally surfaced, and got fixed, on the same day we found it.