• A Call to ARMs: Bringing Observability 2.0 to Mobile (Part 1)

    The Observability-Free Zone

    For most companies, “mobile observability” is a misnomer. I say this because the data that most mobile devs have to “observe” their app in production is laughably simplistic. The vast majority make due with basic crash tracking and precomputed metric aggregates as the only lens into how their app is performing in the wild. 

    Those who want more than the bare minimum, like actually-useful data to hunt down ANRs or network request latency from the client’s perspective can pay vendors that provide SDKs and UI that will track user and app events in greater detail. Some of these products are better than others, though very few provide the capabilities that backend SREs are used to when they are troubleshooting production issues.

    For a long time, this kind of basic production monitoring was good enough for most companies who ship mobile apps. Simply knowing whether a new version has caused crashes to spike, or whether the P50 of cold app startup is below some arbitrary value, was seen to be enough. If those numbers looked good – by some definition of good – the app was considered stable.

    But increasingly, people are becoming unconvinced, as users complain about app quality issues that simply don’t show up in those wonderful databoards they have that supposedly tell them how their app is performing in production. Because if all you have are aggregate crash counts and a handful of percentiles, your app isn’t really observable.

    So when I flippantly say that mobile observability is a misnomer, that’s what I really mean: observability is much more than just having graphs of a few key metrics. It is, as Hazel Weakly puts it, about having the ability to ask meaningful questions and get useful answers that you can then act upon. 

    If your tooling can only tell you that some amount of people are experiencing slow app startups but doesn’t give you the ability to figure out who they are or why just them, that’s not observability to me – that’s just monitoring.

    Some folks who are hip to this turn to vendors like my current employer to provide mobile app performance data in production that is actually actionable. Others achieve similar results by building or assembling all the pieces on their own, like my previous employer.

    Still, even as commercial and homegrown mobile observability solutions become more prevalent, most mobile devs continue to be stuck in the dark ages, even as their SRE colleagues rack up eye-watering bills for logging traditional backend observability. 

    A large part of what has caused the arrested development of mobile observability is the lack of demand. Small, already overworked mobile teams aren’t out here demanding more problems to solve, no matter how passionate they are about performance.

    But I think things are about to change: true observability of the 2.0 variety is coming to the mobile world en masse for all who want it. Not only is the demand increasing at a breakneck speed, the ecosystem of tooling is mature enough for mobile solutions to not only work within their own silos, but also enhance existing observability data collected for the backend. 

    To me, mobile is the final Infinity Stone of the Observability Gauntlet.

    So how did we get here? It’s simple, really: good old supply and demand.

    The Demand Problem

    Mobile teams are chronically under-staffed. By that, I don’t mean they are always small – they are just always asked to do more than what their staffing levels can support in actuality.

    To some degree, this predicament is understandable: mobile platforms and ecosystems these days are so sophisticated, an outsider might think that you can ship, maintain, and add features to an app with just a small team of relatively junior devs.

    And they’re not totally wrong. From powerful platforms and tooling, automated functional and performance testing frameworks, robust CI/CD pipelines, and hands-off distribution channels like the Play and App Stores, shipping a v1 of an app has never been easier. However, shipping v1 is just the start.

    The work that consumes most mobile teams after v1 is maintaining a stable user experience as the app and the world changes around it. Adding new features, sure, that takes time, but supporting new devices and OS versions without regressions, all the while features are being added, sometimes haphazardly to meet deadlines, can be deceptively tricky, and not always accounted for when calculating staffing needs.

    This is because the execution environment of mobile apps is so unpredictable, it takes an outsized effort to properly plan, create, and maintain the battery of automated tests that are necessary to ensure that most – not even all – code paths and workflow are properly covered.

    Even when you leave out the different combinations of hardware and software an app has to run on, factors like network connection status, battery, available system resources like CPU, memory, disk, etc. means that unit, integration, and performance tests have a lot of combinations to cover. 

    And that’s before you introduce the chaos that an end user can have on how an app runs. Or the seemingly arbitrary decisions that mobile OSes make that further adds to the entropy. 

    “Oh, you think your background threads are going to finish running when the user takes a phone call? Sorry, there isn’t enough free memory, so I’m just going to kill your app. I sure hope you handled the case when your serialization to disk can be interrupted mid-stream!”

    Simply put, if your mobile test suite does its job well, maintaining it as your code base changes is going to take up a lot of your time. If it doesn’t, production bugs and regressions are going to keep you even busier,

    The hamster wheel can feel draining for mobile teams. Complaints of features taking too long to ship will inevitably lead to a Sophie’s Choice between not writing enough tests or having to fix production bugs later. The last thing teams like this need is tooling that tells them their apps have more issues than just what their Crashlytics dashboard shows.

    Even those who crave more product data to help them debug issues tend to look for tactical solutions for specific problems, to help them fix the bugs that they already know about. For that, the status quo is perfectly fine.

    So what’s changed? Why is there suddenly demand for real mobile observability?

    First of all, the demand has always been there. It’s just been… silo’d. Some folks in the industry have realized that it’s essential for mobile apps to be truly observable – SLOs involving user workflows don’t make sense when you don’t include data from the apps themselves. Latency on the client are measured in seconds – shaving a couple hundred milliseconds in the backend will barely register if the request is running on a 2G network.

    Big Tech, my friends, have understood the importance of client side performance for years – they’ve just built all the tech in-house rather than use vendors. How do I know this? Because that’s what I spent a couple years doing back at Ye Olde Hell Site, before, you know…

    This is where, in a previous draft, I spent a thousand words or so talking about the cross-platform, OpenTelemetry-esque production client tracing framework that I helped conceptualize, build, and rollout, but that’s a tangent I’m skipping for now. Suffice it to say that companies with mobile performance specialists have been all over this. Slack even blogged about it

    And now, what motivated the early adopters will begin to motivate others: because upper management was made to understand its importance.

    You see, busy mobile teams no longer need to be internally motivated to better understand how their apps are performing in production – they’ll be explicitly told to do so by their Directors and VPs, by folks who want to know how mobile apps directly contribute to company KPIs so that they can more optimally allocate their engineering budget.

    All this, because money is no longer cheap, and engineering orgs need to justify their existence and prove their value to the bottom line. The end of ZIRP is the catalyst to the beginning of real mobile observability.

    The End Is the Beginning is the End

    For the uninitiated, ZIRP stands for “zero interest-rate policy”, and during that period (which ended around 2022), the cost to borrow money was very low. The effect it had on tech is that VC investment became abundant as rich people wanted a better return than traditional vehicles. This led to big funds looking to put money into tech startups at a rapidly increasing rate, and buoyed by successes by the parade of unicorns that made so many people wealthy, the money kept on coming.

    In those halcyon days, VC money flowed freely, especially for darling tech companies on the come up. R&D had free reign to spend as long as the company or engineering were perceived to be heading in the right direction. Whether this meant staffing new teams to stand up new products or signing large vendor contracts that provided very specific services, you don’t have to go that high up in the org chart to get approval for projects with significant financial commitments.

    But coming out of COVID, with a macroeconomic climate that featured rising interest rates, the costs for VCs to invest got a lot more expensive, so they’ve become more discerning. With the taps turned off, tech opulence turned into austerity, and that started a domino effect of budget cuts and layoffs. The industry bled, and the new normal is that if your project or team can’t justify their existence or provide a high enough ROI, you may not be around for long.

    While that may seem antithetical to the addition of a new line item in the budget for mobile observability, it is actually quite the opposite. The reason is that mobile performance has always affected app usage, but it’s just not a very exciting thing to back. 

    When competing for attention and money with sexier initiatives like new products and features whose importance and hype are tied directed to the clout of its pushers and their fancy slide decks, it’s really hard for something so relatively uninteresting to be prioritized – if anyone was even pushing for it in the first place. 

    In a vibes-based stack-ranking exercise, something boring like “slow apps make people use them less” don’t tend to end up near the top. But in a world where prioritization is actually data-driven, where you are asked to show your receipts, initiatives that can demonstrably affect the bottom line tend to win out. In that environment, people will go out of their way to find proof of ROI and efficacy for their projects.

    And where would you find better ROI than in a key part of your customer’s journey, a part in which you have limited performance data if any, where a whole class of issues creating friction for your users are invisible to you? Mobile performance is low-hanging fruit galore, and adding observability in your app will bring you truck loads.

    When you ship a regression in the app that slows down certain workflows on a mobile app, it will not be directly reflected in your dashboards if they are based solely on telemetry generated from your servers. Those well-calibrated SLO alerts that your SREs rely on to detect emerging incidents? They won’t fire. 

    If the only telemetry you have for your apps in production are crashes and pre-aggregated metrics, you will have so many blindspots where app performance regressions could be killing you on the margins. Even if you see your KPIs drop, you wouldn’t even know that they were caused by your app being materially slower for some percentage of your users in production because you lack the data to diagnose that.

    So yeah. Every mobile team should want this. Every SRE team should demand that their mobile team use this. The question is: how? If you’re not a big tech company who can throw people and money at the problem, how can you ease yourself into mobile observability without being tied down to a specific vendor’s solution?

    I’ll discuss this further in Part 2. Hint: it starts with O and ends with penTelemetry.


  • On Marvin the Album

    I came here bang out 2500 words on one of the best birthday presents I’ve ever received: the 30th Anniversary release of Frente‘s Marvin the Album on vinyl. I was going to talk about how it was the first album I ever bought myself, on cassette, from the long defunct Music World in Coquitlam Centre.

    But then I realized I would never finish it because I would have so much to say about how important it was to me, how well I know the songs, and how they would be the band I want to see the most, out of every band in my 1200+ CD collection. I would probably bawl if I’m ever in the same room as Angie Hart as she sings the first line of Girl.

    I would also have to talk about how they’ve more or less disappeared from my life, as their second and last album was released in 1996. Even though I was scouring the early internet for singles and EPs long after that for b-sides and covers, I don’t think I’ve actively thought about them much over the last 20 odd years.

    Of course, Angie’s band Splendid appearing on Buffy the Vampire Slayer was a huge deal for me. I also liked that record, plus all of Angie’s solo work that I’ve listened to. But Frente as a thing was in the past for me, and not actively on my mind as a band that I missed. It’s just been so long since they were an active band.

    When I heard about their reunion shows in Australian when Marvin turned 20 (?), it had already happened. At that point, I thought I had missed my chance to see them. I was disappointed, but as I said, having not been actively thinking about them an ongoing concern, it garnered little more than a shrug and “too bad” from me.

    So earlier this year, when I mentioned to Danica about how I would like Marvin on vinyl, being on my nostalgia binge brought on by the 30th anniversary tour that Sarah McLachlan did for Fumbling Towards Ecstasy (which is the other album that hold this level of importance to me), I didn’t even bother looking to see if it existed. 14 year-old me would’ve been disgusted. 34 year-old me would’ve understood.

    I soon found out it existed though: on my birthday this year, when I opened up Danica’s present to me. There it was. And IT WAS SIGNED! Splendid. Newly 44 year-old me was absolutely delighted.

    But even then, I didn’t listen to it right away. I had birthday drinks to go to that night after work, and seeing a small group of my friends in person, something that is fleetingly rare these days, was still how I was going to spend that evening. Between that, the resulting hangover (well-made cocktails still slap), a pub quiz (more friends!), and some leftover work stuff I was itching to get done, I didn’t really have a chance until last night to take a trip down memory lane.

    When I finished up my part of Eliot’s bed time routine, I went into my office and ripped opened the plastic. The cover and linear notes were familiar, but not exactly the same as the what I was used to on the cassette and CD. And I was really familiar with the linear notes on those formats for this record. Really familiar. I didn’t have much recorded music of my own when I was in high school, so the albums I had, I knew every part of them.

    I dropped the needle on the record, switched the input on my amp, and put on the connected headphones. The crackle comes through. Just in time. And then Angie: A girl is the word / that she hasn’t heard. Oh I was so back.

    It’s not that I haven’t listened to that song and album on and off over the last decades — I just haven’t listened to it with such deliberateness in a long time. That’s what this whole vinyl thing is for me, to listen to music intentionally as the only thing I’m doing, to literally put the needle on wax and wait as the right groove is hit and sound is produced. Latency is a gift.

    The first time though was jarring. Girl sounded exactly as I remembered, with a bit more crispness on the vocals due to the listening format/environment. But when I was expecting the end of that to go into Labour of Love, it went to Accidently Kelly Street instead.

    You see, this version was effectively the original Australian release, with the original running order, and the Bizarre Love Triangle cover tagged on the end. The US/Canadian version I was used to had a different running order, as well as a couple of songs that were swapped out. I was familiar with 1-9-0 and Out of Sight because I had heard them from other releases, but they weren’t part of the Marvin experience for me, so they felt out of context when they came on.

    But a different set of songs and running order didn’t materially alter my experience. I was still listening to Marvin the Album with intentionality for the first time in years and years. And everything I loved about it and them came flooding back.

    It was magical. Transcending. The nostalgia sweet spot was worked with precision.

    So of course I googled them afterwards to see if they did a tour when this was release a couple years ago. And of course they did. Last year. Australia. I had missed the chance again. But this time, I was bummed. Really bummed.

    The google tunnel led to their socials, and then a glimmer of hope: they were working on new music! So maybe, in 2025, a reunited Frente (probably just Angie and Simon Austin?) might need to do some promo for a new record, perhaps some small shows in their native land?

    The dream is not dead yet.

    Anyway, where I was? Yeah. I don’t have time to write that post. Instead, I’ll just say thing:

    I love music and Marvin the Album fucking slaps.


  • On Oasis, Nostalgia, and Monoculture

    Oasis was not a huge band in Canada. Wonderwall was a big hit, but they were largely just another Britpop band to all but the British people who moved here. Part of that is because they didn’t really become a thing in America, where they were more known for the antics of the brothers Gallagher than for their music. Growing up in the 90s, they could’ve easily passed me by, growing up in Vancouver, not materially more relevant than Kula Shaker or Spacehog.

    But for me, that wasn’t the case, because all my best friends growing up were *super* into them. Like, “scouring the import bins of HMV or whatever record store was in town for the B-sides” into them. Definitely Maybe, (What’s the Story) Morning Glory?, Be Here Now, and the aforementioned B-sides soundtracked many car rides and marathon Dreamcast sessions.

    In my small friend group, the release of Be Here Now, was a huge deal, even though everyone agreed that D’You Know What I Mean? was a bloated mess. When Liam finally wrote his first song, Little James, on Standing on the Shoulder of Giants, we all said “nice try.” I even knew the names Meg Matthews and Patsy Kensit without having read a single British tabloid. So despite where I grew up, Oasis was a thing in my life.

    And the funny thing is, back then, I didn’t even like their music. I’ve probably said on a number of occasions about how their songs were simple and boring, or went on about how big of a asshole Liam is. When Manchester City was relegated to Division 1, and then Division 2, I laughed so hard. And yet, I couldn’t get them out of my life. The songs off those first two records, especially Definitely Maybe, were drilled into my brain.

    Musically, trad-y rock bands just wasn’t my cup of tea, but a not-small part of my disdain for them was simple teenage contrarianism. All my friends liked them, so I’m going to go out of my way not to. But as I got into my 20s, my tastes expanded, and I grew fonder of all of those familiar songs. It got to a point where I couldn’t deny that their blend of straightforward songwriting, mind-numbingly basic lyrics, and a knack for melodies and killer choruses actually make a pretty compelling package. They don’t live up to the hyperbolic boasts of Noel and Liam, but as a mainstream rock band, they did what they do really well. I liked them enough to go see them when they finally came back to Vancouver in their later years, and genuinely consider myself a fan.

    So when Noel and Liam announced they were getting back together for a tour next year, something in me clicked, and I felt compelled to fly All Around The World (heh) to see them play their native land, where they were legitimately massive, one of if not the most representative band of an era of music in the UK that was literally named for it. There are very few bands I’d do this for. There are many other bands whose music I like a lot better that I wouldn’t do this for.

    That’s because to me, Oasis were more than a band, more than songs. Not in an “oh their music was so special to me” sort of way, but more because they were an integral part of my formative years, and the thought that I could relive a small part of it was incredibly compelling. On top of that, to be able to experience their reemergence amongst other people for whom Oasis were also an important part of their growing up, sharing OMG vibes, it’s not something I could pass up.

    Make no mistake: this is absolutely a nostalgia hit for me, one that is increased by a factor of 10, as I’ve never experience Oasis as part of a monoculture before. In my life, they were huge, but for most of the people around me (save for my friends), they were just another band. No more important than Travis or Pulp. It’s analogous to (association) football, really, where being really into it here in Canada is kind of a weird, non-mainstream thing, but totally normal and mainstream over there.

    Like if a British NFL fans goes to the Super Bowl: what if your niche interest at home was the biggest thing in the world?

    Having gotten tickets to one of the Edinburgh shows means that I get to experience Oasis with a bunch of 40+ year-olds who also vividly remember Euro 96. And I suspect my motivation is not dissimilar to many of the people who braved the ticket buying mess and scooped up one of 1.2 million tickets sold this weekend for one of the their reunion gigs in the UK and Ireland next summer.

    People didn’t fight the internet and fork out their hard-earned money in a cost of living crisis just to see some band play music that was big almost 3 decades ago. I suspect that many, like me, want to reach back in time to experience a piece of their past. It’s not (just) about the music, man.

    So all of the online takes from people with no fondness or connection to Oasis, about whether the tickets are worth it, or whether they deserve all this hype given their uneven output as a band, are totally missing the point: all of us want to go back to 1995, to a time when Manchester City sucked ass (among other things), even if it’s just for one night. Artistic merit or worthiness doesn’t even come close to factoring into why we want to see this so badly.

    It doesn’t really matter if Blur had a more consistent catalogue, or if Radiohead had a greater influence on music. They aren’t Oasis. 1.2 million tickets sold, with many more wanting to buy them, is proof that they are like no other band of that era. Being away for 15 years surely helped with the demand, but the hoopla grew organically, without much lead time or marketing. The demand is there because Oasis is just different. Don’t make this old man explain it to you. IYKYK.

    Oasis was a thing in the 1990s. A massive, massive thing. At least in the UK. If everything about the 90s are coming back now, shouldn’t the self-proclaimed biggest band in the world be part of that? And for at least for a couple of months in 2025, they will be.


  • On Mobile Observability, the Niche of Niches

    Hi. It’s been like 9 months since I last posted. Two young kids, one with special needs, will do that to you. But thanks to recent bed time routine changes, I have some enforced time out of the apartment and in front of a computer every night for the foreseeable future. So what am I going to do with that time? Navel-gaze. First, lets go meta, in an unnecessarily long and winding way, as is my style.

    The worst thing about ADHD for me is my incessant need to learn. No, that’s not right. It’s my incessant need to go deep in interest areas that is beyond reason for a neurotypical person. Not only do I like niches – I LOVE the overlap of niches, a good deal of that interest owing to the possibility of discovering something new, to gain novel insight simply because I’m examining well-worn paths through new lenses.

    Football finance and Watford FC? I can never hope to achieve the level of expertises as true experts in those areas, but when you slap two circles together of folks who are interested in each, the overlap you get will be tiny. And that’s where you’ll find me. Hi: I’m your Venn diagram intersection.

    This is perhaps why I do what I do for work. How well do I know Android? Decently well in some areas, passable in others. Experimentation and data analysis? I know enough to get by. Mobile app performance? OK, THERE I can claim some specialized domain knowledge. What about collecting perf and stability telemetry on mobile devices in production? Well, that’s just I’ve been doing for the last 9 year for money.

    Well. If each of those related topics are a circle, I’m at the centre of that overlap, and overlap that has its own name: mobile app observability. To say that my work experience and domain knowledge makes me suitable for the field is a given. Whether I’m good at it is up for debate. I think I am pretty OK, but I’m a bit biased.

    What’s not up for debate is that spending time exploring this niche of niches is so absolutely my jam. Measuring and improving mobile app performance in the aggregate, whole-user-base level, and deriving insights from that, is such a challenging, fascinating, and impactful problem to solve. To me anyway. Not only that, it gives me the opportunity to go DEEP in a subject area, something that, due to life circumstances, I’m generally not able to do for fun these days. So what do I do to scratch that itch to deeper than deep? Work. I’m sure my employers don’t mind.


  • On Talking About Football Finance on Social Media

    It turns out that 280 characters of not carefully worded chunks is not a great medium to talk about football finance. Specifically, getting folks to understand what you’re trying to say requires a level of precision that I’m not willing to invest when firing off takes on a mobile phone, especially when there is an opinion embedded in said takes that some folks don’t agree with.

    This began first thing Friday morning, when I checked my email to see that Companies House has sent me a notification about Watford registering a new charge. It turns out that the club had arranged loan with Macquarie in which we get an immediate cash injection secured on the remaining payments for Ismaila Sarr due from Olympique Marseille over the course of the next couple of calendar years.

    I yawned, fired off a tweet about how I was surprised we haven’t done it yet and wondered what we needed the cash for, then didn’t think much of it. This is the type of transaction we’ve been quite used to given the aggressive loan repayment schedule and our revenue and cashflow being dramatically reduced thanks to relegation. Getting what football finance expert Kieran Maguire calls “posh payday loans” is par for the course: we need cash now, so we’ll pay a bank like Macquarie a cut of those future receivables in order to get the money now so we can meet our financial obligations. It’s not great, but this is what a relegated club has to do if it wants to pay down debt as well as field a competitive squad.

    The next day, right before the Leicester game, I saw some concerns online (or what I perceived as concerns) about this, and how the club (via Scott Duxburry) said over the summer we’d be debt free by the end of the year, and taking this loan out puts into doubts the validity of that statement. I, perhaps stupidly, replied (this was someone I followed and whose opinion I respected) saying that this wasn’t a new loan per se, just exchanging future receivables for a smaller lump sum now to get additional cashflow for whatever reason.

    What followed was odd. I had a couple of people relying to me, apparently annoyed at me for saying this wasn’t a new loan. One of them was has been replying to my posts for a couple years at least, always with something critical or nit-picky to say. If you’re a Watford fan on Twitter, he might have done it to you too. He kept saying I was wrong without actually addressing my points, and then started his usual tone-policing. He even corrected my grammar on an unrelated posted for improper conjugation on a collective noun! The last straw came when he replied to another tweet I sent asking folks to mute or block me if they don’t want to see my tweets (thanks, algorithm), saying I was playing the victim card. I then took my advice and blocked him so he won’t ever have to be annoyed by my arrogance or whatever. Funny thing: his replies probably made the algorithm inject more of my tweets into his timeline, which made it more likely he sees tweets from me that he won’t like. Like Thom Yorke said, you did it to yourself.

    The other person replying was actually pretty reasonable. He said I was wrong for saying it’s not a new loan, and he’s right. I mean, it’s a new charge on Companies House, so of course it’s an new loan in the technical sense.

    What I was trying to say that it’s not as if Gino rang up Macquarie and got them to wire £10M that is secured on something we don’t want to part with – like Vicarage Road. This is a new financial obligation, but we already know how we’ll pay for it – the future transfer fee instalments for Sarr from Marseille. We don’t have to find new cash to pay for it – it’s already on the books. It matters that the security isn’t Vicarage Road. Now THAT would be a problem, as defaulting on a loan like that means the stadium is Macquaries. This? The worst that can happen is that Macquarie takes the legal rights to those instalments, which is effectively where that cash is going to anyway.

    What I *should* have said instead is that in my opinion, this new loan isn’t something I’d be overly concerned about given this specific charge will be settled by payments owed to us from a Champions League club over the next two years, which is pretty safe IMO given that Macquarie is also willing to accept it as collateral.

    The reason I said what I said is because if someone concerned about the club’s financial situation reads that “Watford has borrowed £10M from Macquarie”, they might have a bit of a panic. And rightfully so if this means we are adding to our financial burden materially. But knowing that we are a club that has cash obligations greater than what we bring in, partly due to debt repayment to this very bank, I’m not surprised that we need money due to us in 2025 right now.

    I was expecting this, in fact, given Scott said we’d be “debt free” by this year’s end, which I take for clearing the refinanced consolidated loan from Macquarie that was once worth £50M, and it doesn’t include payables or Gino’s loan. The club, like me, was imprecise with their language, and as a result, has drawn the ire of some fans, much as I have, to a much, much smaller degree.

    So what did I learn? Be very, very precise when talking about technical details of something that people feel passionately about (Watford FC). If someone can blow pass your general sentiment and focus on aspects that you have verifiably gotten wrong, they will, especially when they disagree with your general sentiment, which I’d much rather they focus on instead. Whether this is or isn’t a loan isn’t the crux of what most people will have a problem with – whether we should be worried that Gino will piss this money away instead of paying down debt and effectively add to our future financial burden is.

    For me, I’ll need to see more to be worried about this, like splashing out £10M+ for players in January, or refinancing the existing Macquarie loan to extend the payment schedule. If those things happen, then I’ll inch closer to the panic button.