Software engineering is all about trade offs. Almost anything within the laws of physics and computers is possible if you pay for it. The currency is not only in dollars or engineer-days, but also in degraded performance in other aspects of the system. Cap theorem, etc. You also pay for it in increased complexity and reduce maintainability to handle rare-but-possible edge cases well.
But I guess that can all be mostly reduced to dollars and engineers.
Often, certain levels of performance guarantees are not impossible, just impractical. You want your benchmark requests to return at 50% of the current duration? Sure – but we’ll have to rearchitect everything from scratch and increase the server costs by 4x, as well as double the SRE team to handle all the spinning plates. Give us 2 years and 20 new engineers hired and ready to go in 3 months and you can have it. You want this in 1 year? Uh… 50 engineers, 10x server costs, and triple the SRE headcount?
It’s not that Twitter needed the ~3000 or whatever engineers and other tech and design folks it had before November of 2022 to maintain the service, but that many were in place to not only keep it running as it is, but to grow it in a way that would please shareholders without blowing things up. Yay capitalism! There are a lot of spinning plates at the Tweet factory and adding more without things falling over is… not easy.
This is because 98% isn’t good enough. For a service that is as well respected as Twitter (for its backend tech, at least), things have to work virtually flawlessly. “Eventually consistent” timelines with SLAs more than a few seconds can breed conspiracy – why am I not seeing what buddy over here is seeing? See the timestamped screenshot? Shadowbannnnnnnnnn!
That level of performance at that scale is very expensive. To build, to maintain, to improve. Remember the Fail Whale? The reason we were able to eradicate that is due to many, many, MANY hours of smart people building a system that is fast, resilient, and maintainable. It’s where a lot of the money from poorly targeted ads went to: just making sure everybody can tweet lots and everyone else can see it if they want. This is on top of a flawed but well-intentioned moderation system that folks just shat on without giving a thought about how hard it is to get right. Content moderation at scale is an unsolved problem, but Old Twitter was at least trying to do it – and devoting resources to do so.
What is unfolding today (July 1, 2023) at New Twitter is the result of callousness and a disrespect for how much work it is to get from 98% to as close as 100% as possible. The cost-benefit may not be there if you want to maximize profits, but that’s what you have to do to make the product as good as possible. Folks at Old Twitter were trying to do that, but it was hard in the context of a shareholder-value maximizing environment of a public company.
Jack Dorsey is not wrong about Twitter needing to be a private company for it to achieve its self-stated role as the public town square. But that’s not what the current administration is trying to do even though they still use that as a slogan. They want us to pay to see more tweets. It’s a town square, but designed for profit maximization, one that will gladly cut costs to make the product worse if it makes financial sense. But lets call a spade a spade.
If it’s not obvious to you before that New Twitter isn’t Old Twitter…. well, it’s probably still not obvious since this is just the same shit happening over and over again. But this is just one more brick in the wall in an increasingly fenced-off garden that we the users have helped build that probably cost more money that folks on the outside think it should’ve cost, one that isn’t about maximizing product quality and user safety if it comes at the cost of profits.
I guess it’s just capitalism at work.
Leave a Reply