Goodbye, Amazon!
8.75 years, and not counting anymore.
After almost 9 years, I have moved on from Amazon. And this is a very frank and straight from the heart account of my duration there.
The Berlin Days
I had joined Amazon in Berlin. A super sleepy team with extremely boring and meaningless work. I didn’t even get to commit my first code until well into 3 or 4 months into the job.
I was in a team that was tasked with re-architecting and implementing a ‘review solicitation engine’. In simple terms, it was something that would find out who bought stuff from Amazon in the last X weeks, and then send them emails to write a review of the product on the Amazon site. The legacy workflow was implemented as a few python scripts that used to run on cron triggers. The new service was to be implemented in ‘modern’, ahem, ‘Java8‘ and as a workflow, instead of a bunch of scripts tied together.
It seemed like a normal software engineering project in the beginning. But turned sour quite soon. A toxic team was to blame. The team had two additional SDEs who would never stop arguing. It was painful to work with the team. Even the SDM was helpless. Every single pull request ended in heated discussions, even over simple stuff like, ‘why tag code comments as [nit]’ or ‘how much detail would you want to put in an acceptance criteria of a ticket’. Getting them to do something as a team, was next to impossible. Many times, PEs were pulled into the discussions to just tell them ‘shut the fuck up, and do this’.
And that turned the project from super frustrating to extremely boring. The PE came in, and made a unilateral decision to work with a workflow already developed by some other team! And we were just expected to write ‘yaml’ configurations to tell how the workflow should behave. So, essentially, months of ‘heated discussions‘ on the potential Java service, reduced to a maintaining a number of yaml configuration files.
We still figured some good engineering things to do, afterall we liked coding! So we came up with templatizing the yaml files and generating those. Basically, use config files to generate more config files.
Later on, we started with experimenting reducing the spams sent to Amazon’s customers by using ML to identify most likely customers to write some reviews. And we did some additional enhancements and expansions among other things.
But overall, it was pretty boring. Meaningless too, as I despised the whole ‘send spam notifications’ to customers
The SCOT Years
I moved to Seattle to join a team in the gigantic supply chain organization. It was a team that identified vendors or suppliers from whom Amazon should replenish the inventories.
The work was better. Regular java microservices. People weren’t fighting much, which was good. But boy, was I in for a surprise! The abysmal state of code or testing or general engineering practices, were embarrassing. Code that was written sometime in 2008 or so, and had no clear owners and kept on passing around to multiple teams. Folks just came along, dumped their shit, and moved on. Design patterns? What’s that?
I soon learnt that Seattle had a completely different work culture. SDEs were more focused on ‘doing the most visible thing’ as opposed to ‘doing engineering things‘. Code optimizations, design patterns, testing strategies etc., always took a backseat. Folks were doing things that solved business problems ASAP no matter how it was done. Just add some patch work to already existing spaghetti mess and call it a day.
No wonder things were super messy. A deprecation of a service went on and on for years with no result. Changing an API signature would take 10 months of work. Tickets will remain open for months and then some ‘senior engineer’ will close them with an embarrassing comment like ‘if the issue still remains, please create a new one’. Of course, the ticket requester would have forgotten what was the issue in the first place.
I wouldn’t say it was the fault of the SDEs. It was the ‘culture’. SDEs who worked on business heavy projects were getting more visibility. Better chances of promotion if you learnt supply chain instead of engineering skills. Praises were showered on folks who wore multiple hats, i.e., SDEs who also did the work of a supply chain manager! SDEs would get promoted for writing a tiny lambda that created some business tickets, rather than engineers who spent much effort in rearchitecting in optimized way over AWS.
I realized I had to leave the org if I wanted to stay as an ‘engineer’. If I have to roughly estimate, 70% of my time was going in learning and making decisions related to supply chain and probably 30% was going towards real engineering work. I hated working with the decade old spaghetti code that nobody wanted to cleanup. If I would take the initiative to clean up that mess, it would have been a thankless job.
I tried to make some initiatives. Like, introducing Kotlin to the stack, or developing a generic test framework. I was shut down. ‘We are not in the business of making platforms’ or ‘Kotlin will be harder to move from Java’ (seriously?) were some of the remarks.
On top of that, the constant ass licking of upper level was nauseating. A senior manager literally said in an org meeting that ‘PEs/Sr. SDEs are great gods’. PEs or Senior SDEs were more Supply chain engineers rather than ‘software engineers’. I was disgusted by the incompetence of those. They would ask questions like ‘whats the benefit of Java 17 over 8 apart from being cool’, or ‘why do we need CDK when we can just have the resources created through console’.
So I left. I wanted to do engineering work. And I realized, the hard way, that I should stay at a place where ‘technology’ is the business.
Experiments at AWS
I first joined Route53. My team worked on the ZonalShift product that customers use to move traffic away from an availability zone to some other zone. Usually they use it during AZ impairment events. Like, IAD1 is down, then get out of that zone and move traffic to IAD6 or something.
It was what I wanted. Pure technology business. No bullshit supply chain decisions or sending spam emails to folks. I loved it. The focus on technology was much better than any other team before. We talked tech, instead of random business shit. People knew more than me. So it was an opportunity to learn from them, and I did a good deal.
Also the team was working heavy on Kotlin. CDK was given. I was doing a lot of those. Custom CDK constructs in typescript, developing features in Kotlin, talking to IAM teams, ASG teams etc., to onboard additional AWS services to ZonalShift. It was good work, and I liked it.
But, I still wanted something better. My learning in that team soon plateued. There wasn’t more fun things to do. And I wanted to get into rust, databases, system programming. Everything seemed very ‘high level’ now. It was much better than SCOT, but it wasn’t that fulfilling from my learning point of view.
So I moved again. This time to AuroraDSQL. The fancy, brand new database that was under development at that point of time.
I wasn’t in the core DB internals team. I joined the peripheries. I was in a team that worked closely with the control plane of the database. We owned the billing and metering of the database. When customers create a cluster, write some data, our service made sure that the db metrics (e.g., storage size, number of DPUs etc.,) are visible in their cloudwatch.
Everything was in rust. It was great learning experience. I also took part in some work that involved in getting a bit inside the storage engine. The team had very experienced developers who knew a great deal of DB internals, query engines, rust programming and so on. The team worked closely with rust team that actually develops some well known open source libraries and frameworks like turmoil, duchess, tokio etc.
This was everything that I wanted: DB, rust, system programming, distributed systems. I was already shining in the first few months of my joining. I looked forward to so much more! Formal methods, query engine optimizations, simulations, what not!
Until I wasn’t anymore.
The 5 day RTO
January 2025. Amazon declared war on any employee who had a life. Remote or 3-day RTO worked great for me. But 5-day RTO, that came as a kick on the balls.
Everyday, I lost about 2 hours in just driving. Add the time to get dressed, prepare things, stretch and breath, it was easily about 2.5-3 hrs of meaningless wastage of time and energy.
And it wasn’t just that. I had to make sure to at least reach office by 9:30 or so. Which meant, leaving home by 8:15 or at least by 8:30. But kids school timings were 9:00-3:15, which meant I had to arrange extra-hours for my eldest. Similarly my youngest, whose daycare timings were 9:00-5:00, had to be extended to 8:00-6:00.
This came as a big lifestyle and financial change. My kids, who eat their food slower than a snail crossing a mile, almost stayed fasting the whole time. The eldest, who used to come at 3:30 and have a late lunch at home, stayed hungry till 5:30. And both the kids had to wake up early, which effected their sleep quality every single day. They are not early risers, and everyday, I had to see them waking up with eye bags. It was distressing. Why they had to suffer because of a stupid rule from my employer?
Financially, extra hours at day care and school meant shedding extra money. Every month, I paid around 700 extra for that. And on top of that, I had to pay for the fuel and parking. I was losing money on things that I never wanted in the first place.
And all of it started to reflect on my work front. I was tired and exhausted by the time I’d reach home. I hardly worked for 4-5 hours in office and then, even if I wanted to to, I just couldn’t do much productive work at home. Meanwhile, fresh grads, or below-30 folks, staying near the office, easily spent their whole day in the office churning code.
There was no match.
I also thought of moving to Seattle. Maybe that would work? I spent a few weekends hunting rentals in that region. And I was of course disappointed. The lack of good amenities, super high rents, not-so-good schools, drug addicts and homelessness, shooting incidents etc., just didn’t feel right. Kids were having a good time where they were staying, with nice school and friendly atmosphere. Going to a drug infested zone, clearly would have been a wrong move for them.
Also, there was no assurance that Amazon wouldn’t lay me off or put me in focus or something after I move there. Amazon wouldn’t change anyway.
The Grind
Meanwhile, the grind at AuroraDSQL wasn’t getting any better. Deadlines after deadlines. No additional resources. Many folks, including seniors, left the org. Some left Amazon, some left to other teams. No-lifers could still sustain, working 14-16 hours a day. People worked over weekends like it was nothing. Do more, and more, and more - for some artificial deadline. I never understood who was waiting for those features holding their breath? But it was pushed. Get it done somehow!
I checked randomly. There wasn’t a single hour when I didn’t get an email from the code review system. Which comes when someone publishes a code review, or someone comments in a code review. I checked for a longer time window, like 2-3 weeks, and every single hour had at least a few mails: XYZ added some comments, ABC published a CR. 2 AM, 3 AM, 4 AM - every single fucking hour.
I wondered, what the fuck are these folks doing in their lives?
I obviously couldn’t do it. I liked databases, rust, distributed systems. But I liked my life more. I would rather read something and sleep, or maybe work on my personal projects, than work for Amazon.
I still could have done much, much more with remote or 3 day RTO setting. But with 5-day RTO, there was no way in hell I could have kept up with the no-lifers.
So, there went my ambition of getting into storage engine or query planner down the Nitro North’s toilet. I stayed in the peripheries and didn’t even bother to try to get into the core. It was just not worth it.
I finally found something that I thought would work better for me. So I left. Time will tell if that was a good decision. But I had to do it anyway. I moved to Aerospike.
The Goods and Bads of Amazon
There is no doubt Amazon has some great minds working for them. It’s a great place to learn engineering skills, explore diverse areas and domains of tech. The attention to detail, particularly in AWS, is something to be proud of. The deep dives, COEs, the microsecond level optimizations - all are great examples of high quality engineering.
But the bads, are just too many! Its a deeply hierarchical company. And its very frustrating to see incompetent managers or PEs or Sr. SDEs doing random and stupid shit. More so in Amazon Retail, compared to AWS. I have seen Sr. SDE asking question like ‘what is an integ test‘!
And with hierarchies, comes ass-licking. Amazon doesn’t lack asslickers at any level. SDEs try to keep a ‘friendly’ PE or Sr. SDE on their side, for promotions. Which is not a bad thing in general, but becomes toxic when they just do whatever thy lord says without questioning.
Same thing with quality. PEs get away from writing stupid, unorganized docs, with the excuse of ‘I put together this quickly to give some pointers to the team’. The same doc, if written by an SDE would require 5 rounds of refinement. There was clear pattern of ‘rules apply differently to different levels’.
The operational issues, they speak volumes. Amazon, particularly AWS, takes very seriously on ops. SDEs, oncalls, work like hell when a sev-2 knocks. But then, sometime, comes the holier-than-thou comments from the upper echelons. A PE would ask, “Why it wasn’t deployed on so-and-so region ? Bake-time? Why do we need bake time?”. The same PEs would then ask “Why there was a deployment on so-and-so region without a bake time?”
There was no winning here. Hierarchy wins. I am not saying that PEs are dumbasses. But the lack of empathy, high arrogance, farther away from actual ground reality, was very de-motivating. Amazon PEs need to learn that there are great engineers outside of Amazon too, or that, they can lead from the front while being a team player and not necessarily being ‘unavailable’.
I am sure there’s a good number of PEs who just stay unapproachable, fake busy, just to show that they are important! Power play, anyone?
I wouldn’t get into the crappiness of SDMs. They are just too many in Amazon. Its enough to say that I would try to stay away from SDMs who ever worked at Amazon.
Anyway, all that said, I am done with that. I wish it had worked out better. But it didn’t.
If you are an amazonian, and disagree with whatever is written above, shove it up your arse, and don’t reach out.