Rails Business
Brendan Buckingham and Ryan Frisch talk about developing with Ruby on Rails and how to leverage it to build a business.
Rails Business
Andrew Atkinson
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
What do you do when the database is maxed out, the busiest day of your year is six months away, and you still don't know what's actually wrong? Andrew Atkinson joins us to talk through that exact scenario at Aura Frames - the Christmas Day 2024 outage that set it off, and how they made Christmas Day 2025 a success.
LINKS
- Scaling Rails at Aura Frames: Splitting to 8 Primary DBs and Reaching #1 in the App Store
- From Christmas Outage to #1 App Store Ranking: An Aura Frames Postgres Scaling Retrospective
- You can't spell Shard without the word Hard
- Open Positions at Aura Frames
--Sponsor--
This episode is brought to you by Flagrant. Your partners for delivering thoughtful products & applications from any stage in your project.
Questions or comments, email us at railsbusinesspod@gmail.com
Today we got Andrew Atkinson, a software engineer, author, and consultant who's currently working at Aura Frames. Welcome to the show
Andrew:Thanks, Brendan. Glad to be here
Brendan:Yeah, glad to have you.
Ryan:What have you been up to? We we met a long, well, it feels like a long time ago, I guess it really wasn't, at the Sin City Ruby. And, you know, we've done some work together after that, but I haven't talked to you in, like, at least a year and a half, maybe longer. What have you been up to?
Andrew:Yeah. More or less the same stuff, you know, working with Postgres and Rails professionally. And then I think at that time I was finishing up the book, and the book's been now published for two years. and so it's been… You know, I kinda had like a promotional period that was pretty in-intensive last year, and then it's been-- year has been pretty chill with, you know, just kinda doing what I do, like writing about projects periodically online, and that brings in some traffic and, you know, that sorta helps with promotion. But kinda s- sales on the book side are just at a, a trickle level, and that's fine. It's fun to still have it as a, kind of a, a background investment area. But yeah lately I've have been working more as a, as an employee. I did a stint as running my own consultancy working with companies like Locable and other companies. And actually was able to find a really good fit with a company where I'd done some consulting work and there was mutual interest in bringing me on as an employee. So I've been working with Aura Frames now for about a year. A lot of the same sort of stuff, like working with Rails and Postgres, and uh, continues to be a good fit so, excited to talk about more of that today.
Ryan:And if I remember, I, I remember reading the, I think it was pr- it had to be one of your blog posts about the work you were doing at Aura, right? And, you know, they have the big influx of demand right around Christmas Day, right?
Andrew:Yep
Ryan:so, so what is it like know, trying to solve that problem, right? Like, i- is it… W- was it… I don't know how much you can talk about this or not, and feel free to deflect, but you know, was it a, is it a Postgres thing? Was it Postgres and, like, you know, scaling you know, the database? Like, how did you even, you know, sort of start to solve a problem with that much, you know, fluc- you know, dynamic… What's the word I'm looking for, right? Like, some days, like, the, the traffic is really low, and then all of a sudden it's absurd.
Andrew:Right. Yeah, like, sometimes like elasticity. Well, that's not exactly the surging demand, but yeah, there's lots of ways to talk about it, like, load or surging demand or, you know. But basically a four to five times increase on a single day, you know, which actually stresses everything, but most of that stress tends to be on the database. We need to capture all that right activity for in our-- in the case of Aura Frames, like new users setting up new frames on Christmas Day. Usually they've them as gifts. So yeah, historically for the company its you know, I think around 10 years of existence Christmas Day and Mother's Day are two of the big peak load days that really a lot of the architecture for the infrastructure is designed for, you know, to be able to handle. And then the rest of the year, the company or, you know, outside of peak events, which the company hopes there's more and more of course, like, because it means that the business is successful and healthy and… But there are those two big ones. Outside of the rest of the year the company also wants to, you know, have economical infrastructure costs, so the-- we need to be able to scale things down as well. So yeah, prior to me being part of the company I'd say the main pressure point has-- it seems like it was typically the database because it's not, you know, as easily horizontally scalable. Like as far as the web app instances we- we're big users of a variety of AWS services, and the web app instances, we can put in an auto-scaling group on AW- AWS and have those scale based on CPU so that we can bring in more instances as needed, and those can scale up and down. And that way, you know, we both have capacity when needed, but we can also scale down to control costs.
Ryan:Right. Yeah.
Andrew:with Postgres.
Ryan:Right
Andrew:so yeah the, the approach usually is like, you know, trying to understand what are the main that we might run into. And this would be true of anyone, you know, scaling Postgres. Like, it's or, CPU or memory pressure or whatever it might be, what- whatever the main bottlenecks are. And then you know, designing solutions that would those bottlenecks. So in the case of peak load, you know, we historically prior to… And I wrote about this publicly, so, I can provide these blog post links for the podcast as well. But and the company, you know, was able to take a look and review the posts as well, so they were fine with what was shared in them. prior to last year's Christmas, at the time of this recording, 2025's Christmas the Order Frames was still running on a single Postgres instance, a single primary for the main application API traffic. There were some other instances for auxiliary things like background work and things like that, but the main API was still a single instance scaled up to the biggest instance size that AWS offers, and with the fastest available storage type and, you know, the, the most that we, that the company could purchase for provisioned IOPS. So we could have much CPUs, memory and available to handle that peak you know, use vertical scaling to scale up in advance of that load, then afterwards when things settle down back to the normal levels, scale back down. And that, you know, that's a typical path is with cloud-based, Postgres is, you know, you're able to move instance classes like that, you know, whether it's AWS or another cloud provider. you know, that's pretty sensible because you just gotta click a few buttons and pay a little more money and, you know, you're not investing a huge amount of time in engineering hours and development and maintenance and s- that sort of thing. I think it's a pretty sensible path. But then, Yeah. What had happened at 20… Sorry, I said that wrong before. was a single instance through Christmas 2024, and then we put a new solution in place for last year. So
Ryan:Yep
Andrew:which was two Christmases ago,
Ryan:Ja, ja
Andrew:that's where Aura frames the, the, you know, again, on the largest instance available, peak load, lots of load testing in advance with simulated increased load from what production levels were the previous year, which is a, a cool capability that infrastructure team has built. Despite all of that, on the actual Christmas 2024, there ended up being… It was around a two to three-hour total time period where there was a mix of… There, there was basically an outage. And one of the main issues was related to replay of replica logs fast enough replicas and how… I wrote up the specific details on a Postgres oriented post, but the… I guess the summary would be a replication-related failure mode where data was accumulating faster on a dedicated replica volume and due to a kind of a more or less a misconfiguration, wasn't replicated fast enough and it caused the volume to fill. This is a, a failure mode that other large high write volume Postgres folks have encountered. And so there's various mitigation strategies and that sort of thing. And the team… I wasn't part of the team at that point but the team had a, a really nice write-up on, you know, internally on the, as a postmortem about what happened and what the resolution was and what the tuning advice, you know, the tuning config changes were going forward. but yeah, I mean, more or less it was that peak write traffic and being able to replay that to a an async replica fast enough without running out of disk space as those changes need to be replayed. And so, more or
Ryan:Is that like a buffer then? Is that how that works? Like it puts it in a buffer as it's writing and then that bu- like, I, I don't know if I'm using the right terms here, but that buffer filled up the available memory before it could, you know, put it into the database. Is that sort of what you're saying?
Andrew:I mean, it's, it has to do with Postgres' write-ahead logging system. So to make sure that all the changes durable, the changes are written to you know, buffers that are periodically then flushed to disk, and then those files are then replayed or, you know, replicated to a, a second follower instance, and those write-ahead log changes are replayed on that follower instance. The durability piece for the write-ahead logs would be more of the primary instance, but there was a config change with, I guess I should have refreshed my own knowledge of the nuanced details of this, but there was a config change in the version of Postgres used 14 on RDS related to how it preserved those write-ahead logs. And there was a benefit to it, but the cost to it was this possibility of filling up the, the storage space where the write-ahead logs are stored. And because this write-ahead log storage trans, you know, replication or replay is so critical AWS offers a thing with RDS called a dedicated log volume, and that was in use in this case. So there that… Actually, like a separate dedicated disk to store these write-ahead logs. But there was also a change in 14 that, meant that the, could fill the volume if in the case that the repla- the replica lag was too high. And so that's what occurred. So the replica lag, you know, we wanna keep a, a reachable secondary replica or follower database in sync with the primary. A lot of the businesses run using the replica, like things that don't need to access the primary database, could be admin database or other sorts of functionality. So that's continually kept up to date. But I believe what the team did at the time to sort of recover things was essentially rebuild a new replica. The replica that was in place wasn't gonna be caught up and in order to recover the system I think that's what I would do in this situation. I'd have
Ryan:Yeah.
Andrew:back and review the details, but es-essentially, and again, that would be, I guess, to, you know, to zoom back out a bit, there was a, a Postgres-related failure mode that not have occurred in the following year. there, there wasn't a tremendous amount of confidence that the following year's increased load, because the company's been… By the way, the company's been successful and business is growing annually, so which is great. So we have to plan for
Ryan:I think I got three frames now f- between the family.
Andrew:Yeah. Awesome. That's great to
Ryan:Yeah.
Andrew:Yeah, and I mean, the company continues to, you know, reach new retail locations and sell online and enter new countries. So we forecast increased frames, increased users, and increased activity, which all, you know, on the infrastructure team translates into more pressure on all the pieces. But for me specifically, more on the database is what I would think about. So we think about like, well, what are our new, our, you know, our new write volume at peak and that kind of thing. So, that in year, so now we're getting back to like when I joined in on the party. Last year Aura initially brought me in as a consultant to help review some CPU usage issues related to the single primary instance that, that wasn't actually the original failure mode on the Christmas day, but the failure mode Christmas day was something we discussed and was kind of in the background as context. We also found though that other activities like which are critical for Postgres like vacuum, running on big tables, there could be some, there could be some sporadic high CPU usage that would then have this kinda knock-on effect of increased latency and slow queries and timeouts and errors. And we did a bunch of work to try to diagnose what was causing that forming different hypotheses and testing them. The conclusion we came to was we didn't really reach a really strong of circumstances that were the cause that we felt like we could fix. And then the other thing that was happening at the same time was, this was mid 2025, so it's 2026 now. There was like a ticking clock that was like, we're very concerned as a company about the availability of Postgres through Christmas 2025, right? It's coming up. like, the discussion was like, let's try to solve this problem that we can reproduce in our load testing environment, but if we can't solve it, let's pivot towards a solution like sharding, where we can distribute this load to more than one Postgres instance, and we can kinda more redundancy. You know, we can split the workload basically. We can have, redundancy, but we can duplicate some of the same infrastructure pieces like that dedicated log volume I mentioned. we can now separate out our work to more instances that each will have their own CPU memory and IOPS resources associated with them. And that's ultimately what we did. We didn't really solve the path on the single instance, but there's also kind of an understanding of when are we gonna… When do we think that our workload is such that it's not really gonna fit in one more instance? And the company ultimately, and I think most companies, they want, you know, you wanna have a high degree of confidence going into a really critical sales period for the company. so if that means investing more work to have more compute available and over-provision things, the, the company was definitely more interested in that versus like, you know, continuing to iterate on different possible solutions beyond a certain point for that single instance setup. Does that make sense? Like, you know, basically there were two paths, and we had to choose one path.
Ryan:Yeah, I mean, it's a, you know, it's, it's a risk analysis, right? Like, we're not sure what the problem actually is with the single instance. We only have six months to solve a big problem. We need to move, and even if it means that we have to spend a little bit more in over-provision in the short term, that is better than losing customers, right? I paraphrase. I, I would imagine that's what they were thinking, 'cause I was not there.
Andrew:Yep, that's a good paraphrase. Yeah. And then we,
Ryan:Yeah
Andrew:we were out of to over-provision on the single instance because we were at the biggest instance available, the most IOPS available. So really to get, to be able to over-provision, we had to go to more instances. So then the question became like, well, what do we do? You know, how do we go to more instances?
Ryan:And are you able to talk about like the high level points of like, you know, what into that thought process, right? Like, you know, think about sharding, it's like, all right, I gotta have some sort of universal ID, and I gotta, you know, sort of split the, split the database according- or split the customer base accordingly, then my brain explodes. So
Andrew:I learned the I will start off by repeating the phrase that someone el- I don't know who to credit this to, but it's like you can't s- you can't do sh- or you can't spell shard without the word hard.
Ryan:Right. Right.
Andrew:It's like it's the devil is in the details, you know? It ultimately like on one side you have, you wanna expand capacity, and it's really, it's horizontally, right? It's more server instances running Postgres not vertically, like just moving to a bigger box, you know, with faster disks and more of them and stuff like that. So, We wanna go horizontally, but data lives in a table. So typically, when people talk about sharding, they talk about row-level sharding. So, you know, taking one big table and maybe putting half of the rows on one instance and half on the other. So, the primary task that the application or connection proxy or something needs to do is figure out, well, how do I divide these up? Where do I send writes? Where do I send reads? so you need, you know, you need like a sharding key, something that would say, "For this table, here's how I'm gonna divide up my data." Yeah, that's all pretty hard, and because, you know, not every table, depending on your schema design, might fit like a, a good pattern. And in terms of a high level, what we ultimately did you could argue is actually not sharding in a, a kind of a strict definition if the definition is row-level sharding. we kicked around the term whole table sharding or I also kinda like workload-based sharding, but it on what you're trying to communicate. Like, so with the whole table, basically what another engineer had done prior to me joining was kinda analyze, like here's all the top 10 tables by their actual workload, like what they're actually doing, inserts, updates, and deletes, and selects. And we also… They also tend to be the biggest tables. These are you know, tables with billions of rows that are not partitioned. And so we talked about, okay, well, Postgres lets us do table which we can use to break up a big table, and that can help in some ways, but we still are limited to that, partitions of that table, those smaller pieces being on one instance. So it doesn't really help us in terms of distributing the load. The tables are, for the most part, unpartitioned, and they're big. And so besides the that's happening on them that the application is doing, like the Rails app, there's also like vacuum needs to run. And if there's, you know, hundreds of millions of rows accumulated or tens of millions, that can take a long time to do. It can be of processing time to do the different things that vacuum does. so that's something to consider. Besides the foreground application activity is all the background work. And then there's things like, you know, over time you wanna adjust the indexes so they support the queries better. You might wanna rebuild your indexes so that, Old row versions are no longer part of the indexes. They can be smaller and faster to traverse. so you might wanna even rebuild your table, repack your table. So there's, there's , limitations to what we did, but what we found was if we just sort of make a mapping of here's our 10 biggest tables you know, what sort of additional instances would we need to distribute these to, to be able to get some real gains? And what we found was ended up with a total of going from one to eight total primary Postgres databases for Christmas 2025. what we did was we decided that for two of our biggest, most critical tables, they were actually gonna get their own entire server instance, which I had never seen before and I thought was pretty wild initially. Like, uh, really
Ryan:Yeah, that's
Andrew:table, like one, like, on, on a, a large instance that costs a lot of money. Like, but it's, you know, it's a critical part of the set of tables that are needed to be able to service API requests. And ultimately, by doing that, it's a way to chop up the workload to lots of instances, but not need to get into the of routing rows around. what we could do then I can jump into… I think two more interesting things from that are how we actually did the, the data movement, and then how we actually configured that from the Rails app. So I'm, I
Ryan:Yeah, I, I, that's pretty much the two questions I was gonna ask, and maybe on the second one I'll say, like, how do you-- Like, is it… You know, if you have to do a query, you know, that joins a table across two databases, like, how do you do that? Maybe that's part of the answer that you were about to get into. Or is it designed that, you know, the, the query load there is just low, and you don't actually have to do that that often, and that's why it was a good candidate to be on its own instance anyway. But yeah, l- l- keep going. You're, you're on a roll.
Andrew:Yeah. The big thing right away is we can't-- have to say goodbye to SQL joins across tables that are not in the same database. So, like, if we have table A and table B, before we could join them together on a common column value, and now if table A is on server one and table B is on server two with regular Postgres anyways, you can't just join. It'll just say, "This is a table I don't know about." If you try to, the SQL statement will be invalid. So we we moved to what some folks call, Ultimately, what we did was the databases were reachable from the Rails application, and ultimately what we did was move to two select queries, you know, to keep it simple. So it's like, you know, we select the primary key. In a join, it's like we have the primary key values, and we have or sorry, we have, from the source table, we have the foreign, or probably forward key values that we then would use to query on the second table for its primary key, column with those values. So essentially, that's what the application does, just makes two separate queries. And the cost with that is additional network latency. There's two servers involved now, where there was one before. It's two separate. But there's also some interesting advantages too. Like, you actually end up getting a simpler query that Postgres, you know, is gonna still parse and prepare and execute. But you can get so it's actually not kind of maybe kind of surprisingly, it doesn't necessarily mean that performance is gonna worsen overall. You do have more network latency between the two, it depends on if you can make each individual query… If each individual query is fast. found was the additional latency was not a blocker to proceeding with this plan. One of the interesting characteristics of this plan meant that, you know, we knew where we c- we-- what we couldn't do. We couldn't do SQL joins, we couldn't do foreign key constraints between two tables that are in the same database. We couldn't have common table expressions or sub, sub-queries that referenced tables that weren't in the same database. So essentially, the work kind of became… And I would say this was a bit before like, Claude-based programming. So this was more like 100% human-powered coding. But work basically became going through all the Active Record code and refactoring it to use these two, the two kind of select query approach. what we found was, which is a little bit of a segue, but fortunately Active Record supports something that at least reduces the overall work needed to be able to do this for the associations that are of a certain type. So for associations in a Rails app that are has many through associations Active Record added an option called disable joins. And what disable joins does is it changes the g- the generated SQL away from being a join to being two select queries
Ryan:So you're telling me someone else ran into this problem before and they had already solved it?
Andrew:Yeah, I think that work was… Yeah, I mean, more or less. Like, I think that work more or less was extracted from, Well, I know some work like that, some of the multi-databases support that Rails has, and some of the disable join stuff, to some extent was extracted from code at GitHub as they scaled their MySQL-based database stuff. I
Ryan:Yeah
Andrew:I've never been part of GitHub, but that's what I've read and
Ryan:That makes sense
Andrew:So yeah it's a framework capability, and what was really nice about that we didn't actually have that available to us based on an older Rails version we were running at the time. So we-- When we made that, back to that thing I mentioned before where we were at that fork where we needed to make a decision this would've been about a year ago now. We were-- We didn't yet have the new Rails version, and so I had started writing code that would that would like kinda conditionally branch and do the, queries. And what I found was, I kinda forgot how when you have an association on a Rail, on an Active Record model, you get all these like, you get like seven or eight different little generated methods that help with developer ergonomics, like as you check IDs and, you know, different little things. And what I found is I started to, needed to like re-implement each of those, and it was a big pain. I mean, obviously, we could try to do it in a general place and then include general code into models. And we were starting to go down that path, but there were like bugs to solve, and that was in addition to the unit tests. We were trying to preserve our unit tests around code as we refactored. so, at the same time, another engineer on the team was doing Rails upgrades and got us up to the version that had disabled joins, which I think was Rails 7. And I started experimenting with disabled joins, and it allowed us to like cut out a bunch of custom code that we were starting to do, which was a big
Ryan:Nice.
Andrew:And it, you know, it, had all those generate… It preserved all the regular Rails functionality and stuff. the disabled joins part was a big part of the solution. Maybe it was like I'd say somewhere between 25 and 50% of the solution because a lot of the code really just traversed associations and then did stuff you know, query code. And then maybe the other 50 to percent to 20, 75% was that was not a Rails association where we were accessing tables that would now no longer be in the same database. We'd drive refactorings there through unit tests. Which by the way, we did by… We had a, a Docker-based Postgres instance with eight databases locally, run unit tests, see a bunch of failures, start fixing stuff, like model by model.
Ryan:Yeah
Andrew:and that process played out over a set of weeks
Ryan:Interesting. So w-with this two, you know, query approach, right, does that mean like ordering and, you know, has to be done in Ruby in memory? And does that, you know, cause memory pressure? do you have to be like very cognizant of like what you are selecting on each statement, or is that a non-issue?
Andrew:No, it's definitely an issue. Yeah, we needed-- we, we generally need to be cognizant of any query that's not working with a fixed batch size, like a max batch size. And those definitely show up in like our plus metrics. There'll be certain API that'll be slow and some… Hopefully we catch this in load testing before anything goes to production. we'll see slow API endpoints. We'll dig into the SQL queries. We'll find or, you know, non-batched queries, and then we'll add batches. And that's been a activity on repeat like many times. And but we also kinda know where our like hottest paths are for the API endpoint. A lot of what the frames do is they wanna show like a slideshow of photos. Any code that has to do with that, like there's a bunch of different things you can do with the app, with the mobile apps and the frames, but our hottest paths are usually serving the photos, you know, to the frames. anything in there we go through with a fine-tooth comb and make sure that it's like, you know, we pick a batch from the first table and then we pick a batch from the second table, do what we need to do, and then that's in a loop, that kind of thing. So you know, you could pay a little more in terms of latency because you're batching and you're looping. But if you have all sub-millisecond queries even like as fast as sub, you know, like we generally ha- any sort of query in that path is for sure sub-millisecond at the normal levels, and is index supported and that kind of thing. Then, you know, even with that stacked up latency, you can still have performance that's scalable. So yeah. And but yeah, you can't-- If you had a query that, for example, joined to two tables and then ordered, like you could no longer do that. So you have to then work with the data that you have from the database, and anything that's left over to do needs to be done in Ruby. So yeah you'd, build a, array and then sort it.
Ryan:Fun.
Andrew:Yeah. Yeah, so I mean, a lot of what the infrastructure team does at Aura Frames anyways, as far as coding activities, is not really feature development, but it's really making features And if, you know, since we kind of own the database performance and reliability if there's a code that has to do with the data access, that would be something we would work on typically, or we'd help the product developer make sure that the design is scalable. but yeah, I mean, what I've actually, like, has increased my appreciation for the, the, monolith, which is what we still run that serves most things, is, you know, for new features that are not in the really hot path and they have what I would call baby tables of, you know, less than 10 million rows, like go wild, do whatever. You know, th- like performance doesn't really matter that much. But if you start to touch the hot path or you get big data then we start to care more.
Ryan:Yeah, that makes sense
Brendan:I, I assume that's like part of your review process before they write that code if you know it's gonna be in the hot path, or does it… Do, do you actually wait until a pull request is open and you're like, "Oh, wait a second, we gotta do X, Y, and Z," or whatever?
Andrew:Yeah, it's a good question. The-- I've, I know I've experienced it different ways at different companies too, but I'd say that we, you know, with the monolith, we still have lots of developers committing daily and shipping daily, and the company leadership really wants, really prioritizes that.
Brendan:Mm-hmm
Andrew:And so I'd say that it's a, it's usually not like a design up front. It's usually like try to catch it at PR time, somebody on the team, of the product devs know some of the infrastructure devs to tag on a pull request or to ping in Slack and say,"Hey, can you take a look at this?" But we also try to not be too blocky. Like, if they ship it… i, you know, I think, you know, the reality is that, like, there are sometimes performance issues, but we can, you know, we have different tactics to roll back quickly if needed. do a lot of things like a lot of orgs do, like feature flags and, if a product developer has a concern about the production impact of a change and less production kind of awareness or experience, then they can certainly feature flag and really cut down their audience to a particular feature. It could be employees only to start and things like that. And that's pretty common. So we have, you know, probably yeah, a dizzying amount of feature flags and stuff like that all over the code base
Brendan:And we always end up talking about AI, and it, it just also made my brain think like, okay, how does the AI handle this sort of thing? Like, just the writing of the code and, like, you guys staying on top of that. I imagine you're writing a lot of code, or at least have a lot of PRs that are AI written, or at least have some semblance of that, so you got even that to, to deal with on top of it? Or that's just kinda like what you're dealing with in general, you're saying?
Andrew:Yeah, it's been a really interesting year because it's kind of gone from like zero to basically like a year ago I was just occasionally prompting ChatGPT. I don't even know if I was using it at work. I mean, other than maybe kind of general research. I know that I would use it for like side projects. If I would use it for work, I'd be very careful to not anything specific. You know, it'd be like, "In Postgres, if you have a table you know, that's this many rows, and you write this kind of select query, like what are the performance?" You know, I don't know, like very vague.
Brendan:Yep
Andrew:And it went from like that to like Claude is everywhere in the company. You know, Slack and writing its own GitHub PRs. I've even had-- We even had a period of time where Claude was opening PRs and tagging me, and they're like, "Andrew, you changed this code six months ago. Like you're the owner of this, come and review it." You know, and
Brendan:Yep
Andrew:we've never let Claude ship anything, of course. Like, and there was, there was like a… You know, I think the company just deems it to be a very valuable tool was willing to take some risks but never like shipping unreviewed
Brendan:Yeah, right
Andrew:but like, yeah, I mean, basically trying to push the needle you know, I'd say on encouraging developers to, you know… The, the tool is provided to developers, so you know, hey, you can use this. You can, you know, the tool has access to our code base. The, the tool has access to our, database schema design. use it as much as it helps you get your job done. And so yeah, initially though it was, we've had a couple engineering-- We've had at least one engineering-wide meeting around like, you know, what some of these questions. Like, is it I think we consider it a tool and, you know, there's questions still about like how much priority to give to AI-generated PRs, we've, we definitely have been adjusting things. Like, I'd say currently it's more like adjusted back down from AI-generated PRs to being, mostly PRs that are opened maybe just at a higher volume from developers with more AI assistance. we're still trying to like have another human review those before they go out. I know I was listening to a previous Rails business podcast with Nate Berkopec, and he was talking about like the, how much-- what's the right level of understanding of the code.
Brendan:Yep.
Andrew:That's like a existential crisis I'm a-also going through right now personally you know, I always think that like anything to production like that I'm co-signing, I need to really understand deeply, right? Like, what was it before? What's the change that's being made? about failure modes. Have appropriate tests to show that it works correctly. It handles failure modes, like positive and negative tests. That's kind of the little talk I do run through with myself like at PR review, even if it's my own review reviewing of somebody else's code. But yeah, I mean, at the velocity where they're coming in and with the improving quality of the AI tools, I'm like, well, do I really… You know, what if I fudge that a little bit? Is that-- W-what if I like-- I, I have a pretty good understanding of this, but like what if it's just refactored in a week anyway from now and like the understanding I had like has already, is already stale and it's changed again? You know,
Brendan:Yep. Mm-hmm
Andrew:So I don't know. It's-- I-I'm still not trying to, you know, to be clear, I'm still not trying to-- I'm still trying to maintain the like, especially anything I write. But like with reviews of others' code, it's even more difficult. It's even more like to try to internalize the changes and after and provide a meaningful review, I find like, you know, it's always hard like without AI. So sometimes use AI to like help me it's still like a, a tricky question about I'm co-signing this, I'm kind of part of the stewardship group of this change going out, right?
Brendan:Yep
Andrew:You know, like, yeah, what's the level of understanding that is needed here?
Brendan:Yeah, that, that's a tricky one. Now Ry- Ryan, did you have any follow-ups on the sharding stuff
Ryan:i, I was cur-- that, so you, you talked about the Rails side of things, and you mentioned the other piece of it, which is how do you sort of migrate into that from a database perspective. I'd be interested to hear a little bit more about that too.
Andrew:Yeah. I found that to be interesting too, because, I mean, you've got billions of rows of data, which is terabytes of data to move. You can't just, like, snap your fingers and have the data onto instances. So we, like, we looked at the really clean… The ideal to me, clean way to do it would be logical replication in because you can have row-level control and you can, I mean, the big thing for us, since we were moving an entire table, was we could at least just move one table. We could say, "Table A, we need all of your data on this other instance that we're gonna move to," right? Going from one to two instances in the simplest kind of scenario. tried various forms of logical replication, including native and then AWS has pglogical that has some capabilities. And we ran into different barriers. So what we ultimately ended up doing was creating physical replicas time, which is very reliable and works well in Postgre- in RDS on AWS. So that would replicate the entire database. So it was it was very wasteful right away because what we would do is we'd say,"We want table A on this new instance." So we're gonna just copy the entire database, which has 300 tables on it or whatever. All we actually wanna end up with is A. So we had to unfortunately with that strategy, it was very reliable, but the trade-off was we had to provision a whole bunch of additional space than what we needed. But it worked well, and we ended up basically repeating that seven times. So we'd say, a replica. I wanna use this table on this other instance." And then that way we could gradually verify all the data was there. We could gradually cut over reads and then gradu- and then eventually cut over the writes. So what it ended up becoming was replica promotion process and then a cleanup process. another interesting part of the cleanup was AWS launched blue/green deployments a few years ago, what that lets you do is then we could… Because we had to provision all the storage we needed up front, and even as we deleted tables we didn't need the new instance, Which was a, a method-- a very methodical process to not screw up and lots of reviews and that sort of thing. But ultimately, that's what we did. We had a sec-second instance with all the other tables we don't need, so we delete all those. We still had paid for way more storage than we needed, so we could provision a new instance with the baby storage, you know, much… just table A storage needs. then we could use blue-green deployment, more or less uses logical replication to keep it updated and then we could cut over. so we could then reduce our costs by cutting over to an appropriately or right-sized instance after, then we just repeated that seven times. So yeah, it was-- it, it-- like conceptually to me, if you've ever done a replica promotion to cut data over, it's not that complicated. It was more just very tedious because we did it a bunch of times, and it was very um, that we didn't make mistakes. So that just meant a lot of reviews, a lot of rehearsals, a lot of documentation. And with all of that, it was-- it's very doable.
Brendan:I might have missed it so in case you already answered it, I just want to ask quick. So you, at this, the beginning when, you know, 2024 I think it was, you said you guys had basically like hit the highest provisioned instance you could get. So I assume you're like down like s- smaller instances now with the, all this replication and sharding. Is that right?
Andrew:That's the goal. It wasn't the goal for last year because the goal actually was more just the expanded… With the expanded number of instances, we actually did vertically scale those up. So we were actually way over-provisioned for Christmas 2025. then what we were able to do fortunately through all of that stuff, Christmas 2025 was very successful. There were no database issues through Christmas, which the company was very happy about. And what then the challenge or, you know, not challenge exactly, but less of a challenge became was, okay, well, we can't af- we don't wanna run these very expensive, very large instances with expensive storage throughout the whole year when our needs are much less. So then what we could do is scale everything down gradually. And we use, you know, infrastructure as code, so not everything's like manual, but there's a decent amount of stuff as a small team that is manual clicking around in the AWS console. but gradually what we did was move the instances all to smaller instances, and in a lot of cases, we could even move to less costly storage. And we could certainly reduce all of our um, IOPS needs and that sort of thing. So, so now we kind of have this collection of heterogeneous eight Postgres instances that make up the workload. And what we're looking at for the future then, like Christmas '20, you know, we actually think this architecture we have now will serve us well for at least a couple years. the next limitation that we could run into is the needs of a single table exceeding what we could then do on one instance. And so we wanna then subdivide that table through row-level sharding, and then ultimately move to homogenous instances so that instances can be more of like the you know, how you treat application server instances, where you could have, the same sort of instance class and ideally less costly storage. But the complexity then is like now you're managing a bunch of Postgres instances that all need to be patched and monitored and everything like that.
Brendan:Sure
Andrew:so yeah, that, that work will be like, kind of the future database architecture that would then build even more. You know, it would build enough headroom essentially forever is how we would imagine it.
Brendan:Yeah. Yeah, I mean, it's really interesting to hear all this 'cause, you know, there's always been the, the question, does Rails scale? And other… Obviously, we already talked about GitHub and Shopify and, you know, you guys clearly are doing a ton of stuff too. So, you know, I think that's kind of a moot point. That's not really a, a, a question in my mind at all. But yeah, it's so complex, right? Like, it's just it's taken us 45 minutes to describe all this stuff, and I still have probably, like, 30 questions. I'm sure Ryan has 30 questions, so I, I… Maybe this would be a good time to maybe segue into something else, though, but it be- I don't wanna just completely cut off this, if there's anything else you wanted to add about how this went, any, you know, any other things that would be worth sharing before we do segue into other stuff
Andrew:Yeah, I think I I wrote these in the blog posts. I'm a little more articulate, I think, about some of the finer points hopefully. And I remember just off the top of my head, I remember a couple of things reflecting back on it, 'cause it was kind of an intensive experience as,
Brendan:Yeah, I would bet.
Andrew:we had this like we had the clock ticking, right? And we wanted to be in a place well in advance of the, of Christmas,
Brendan:Yep
Andrew:meant at least a month ahead of time where we were able to load test and say like, "We feel pretty good about this." That was the ideal state. And we kinda got there, like, but it was very close. You know, the, the timing was close. And you know, I think like what I really appreciated was with the engineers before me, they had built a very extensive unit test suite. I think that was critical, you know, because any code, any active record changes that we changed, we needed to have confidence that we weren't causing new regressions. And then I think the engineering leadership team was like very committed to this, so they were really, you know, supportive of like, you know, do whatever we can to get that increased reliability that we want through the holiday season and did a good job of conveying the importance financially to the of that period of time. you know, we were able to make a lot of decisions quickly and then leverage the experience of other engineers that have been there for a long time that really knew, this, this area is really important, this area is not important, could prioritize well and stuff. And that stuff's like, that stuff's like hard to come by, you know, sometimes like on the team if you've got like a team that is not a long-tenured team or if you don't have good support from leadership and stuff. So I really appreciated having all that stuff, and I think that was a key part of the success that's necessarily the Rails code or the Postgres databases
Brendan:Yeah. Th- that makes sense. That's great to hear that it, you, you had all that there to help out. Switching gears a little bit, you know, you wrote the book, you mentioned it a, a little bit already. It's been what? Two years, I guess, is what you said since you wrote it. And that, that was my introduction to you. I think that I kinda reached out shortly after maybe it was in like the beta phase and, you know, you and I chatted about some stuff and, and, you know, you worked with us for a little while too. And so I, I'd be curious to kinda know, like, how's that going? Are you, you know, what, what are you seeing with the book? Are you adding chapters? Are you just kinda letting it ride, you know? What- whatever
Ryan:again?
Brendan:Yeah. Would you do it again? Exactly.
Andrew:Yeah I think I think I would… You know, I feel like I I feel very fortunate to have found, like, somewhat of a niche in, as a software engineer because I was reflecting myself. Like, I finished college and entered the workforce as a software engineer, like, 20 years ago, this year. So I feel like for about 15 of those years I didn't have any sort of, like, particular focus or, you know, I mean, which is fine. I mean, I think early career, you know, you're just doing what you know, you're just learning the field and doing things and whatnot. And e-everything's different now with AI, but we'll just put that pin in that for a moment. But, like, anyway I think a number of years ago, like, I just found that I was really excited about this kind of work, like scaling Postgres because it is hard and there's lots of nuance, and it's interesting, like, how it interacts with the client application. And the database is so important to companies. The data that customers are creating is vital, and it's critical and it's that sort of thing. So I've been enamored with that kind of work and have… The, the book kinda became a way to all of I knew and share it, which I also have always enjoyed. I write a my, my own personal website's been going for, like, 20-plus years. Just a lot of… Eventually, it just turned into a tech blog. I used to write about, like, random stuff, but , I kinda made it more, you know, career-oriented a number of years ago. Uh, So yeah, I mean, I think, like, in terms of writing and learning Postgres and the interaction, I don't s- I don't see that going away from me. It's just like a, the, the flame is still burning as far as, like, it's an interesting area for me, I like to connect with others about it. I'll be, I'm uh, uh, excited to have been picked to speak at the, a couple of upcoming events we can discuss. And I continue to, you know, like, plan to keep doing that sort of thing. So yeah it's really nice, I guess, like in, in summary, to have a niche. But going back to the book thing, the, you know, writing a book is extremely hard, and I think, like, with, unfortunately, with the introduction of AI and stuff, it's it feels like the motivation, which was already a lot of it was intrinsic motivation. It seems like now it's just gotta on- almost only be intrinsic motivation. It's like, w- if I write a book, like, is anyone gonna buy it or read it? Like, is it, you know, how do I What's the ROI on that activity? It was already not great, but
Brendan:Right
Andrew:And that doesn't mean writing books is not great, but it's like, it's somewhat for the author. It's somewhat for, like, hopefully, the author connects with the readers and feels like it's a way to, you know, serving a purpose or provide value to readers and that sort of thing. But if that sort of feedback loop of the readers just, I'm just imagining other developers being like, you know, prompting Claude, and I, I just think a lot fewer of them would say like, I'm gonna go buy a book and build some sort of foundational advice and spend three hours reading one chapter and practice these exercises and stuff." I mean, I'm sure there are people that will still do that, but, like, it just seems like it's changed so much that the, I don't see myself-- I don't know. I don't see myself writing like a, a new book, but, like, I do wanna continue to kinda cultivate this niche of Rails and Postgres, and it sort of bugs me that there's little things that keep happening that are new in Postgres that are making things I've written bit stale. And so, like, I gotta figure out how to do this. I don't know how yet because writing a book's so demanding i-in time, but I'd love to do, like, a either a revision or a slight title change of the book in the future to be practical. But yeah, that's kinda what I'm thinking.
Ryan:ironically, I feel like you know, reaching for a book and building that foundation is probably more important than ever, right? In order to
Andrew:maybe
Ryan:tell when AI
Andrew:and I don't
Ryan:BS'ing you or not. It'd be interesting, like, you have the GitHub repo that you sort of put all your snippets in, and I follow that and I get the notifications and I'm like, "All right, well," if I look at it, I can usually be like, "All right, this is probably where this came from."
Andrew:Yeah
Ryan:But it'd be in-- One just idea is to write a little summary with each of them. You do sometimes. Sometimes it's just a snippet, right? But it's like, know, we're all like-- Our attention spans are so short these days that it's like that's the bite-sized piece of how you can communicate these nuances. You know, so one idea for you of just sort of building on that and keep it going.
Andrew:Yeah. Yeah, and I do that's a good idea. I also-- I don't get a lot of response, but I like sharing out other even community or, I mean, things other folks have written too that are relevant to this space, like in a newsletter. So I have a,
Ryan:Mm-hmm.
Andrew:email newsletter that I occa- very occasionally update. It's not very frequent. And then I'm always look- there's other great sources too, Postgres Weekly and the Scaling Postgres podcast and Ruby Weekly and like, all of those things plus social media and whatnot. There's definitely a lot of interest in, you know, a lot of developers working on similar things that wanna learn new capabilities or how other companies solve things and that kind of thing. So, yeah, I mean, it's, it what I do have now is I have an appreciation for how demanding writing the book is, and it is something that would need to figure out how to do in a way that's a little bit less disruptive to my life. So just maybe like trickling things out would be a lot nicer than a big intensive period of time.
Brendan:It,
Ryan:it
Brendan:yeah. A- I mean, from my perspective too, like I still value your expertise and others' expertise. So if you've written the book, maybe you're not gonna make money on the book, but, you know, you do cons- some consulting. If that person's doing consulting, you know, we still go to Nate, we still go to you, you know, for even with AI 'cause, like, like we were already talking about, I don't trust everything the AI does. I think it still needs a person there. And I, I personally don't trust it on like big architecture. Like sure, if I'm greenfield, I'm gonna build something, I might just let it run, see how it goes. But if I've got a, you know, like, like Aura had, Aura Frames, like it- they had an architecture problem, and yes, it was pre-AI, but I think today they probably still would search out someone like you and with your expertise. So if nothing else, there's that. And you can also just ask your, your Claude, you know, "What would Andrew Atkinson do?" And get that far, 'cause you get all that, that data out there too, for what it's worth. But,
Andrew:Yeah, I think, yeah, I think companies still want to have, experts in different areas and owners, you know? And it's… thing- things aren't just changed once and released, like they have their own life cycle. And so I really like working on… Another thing I learned in my career journeys was I do like working on at companies like Aura where they're, they have a, a product market fit. You know, they have something where it's like a growth stage of like here's what… Here's a, you know, a very successful component of our business that we're growing into new countries and reaching more potential consumers and stuff like that. so you have… Like with that comes maybe not as much excitement in terms of like new features where things are changing all the time on the platform, but you have this responsibility to maintain the you know, to really not break things and maintain and continually improve, you know, performance and reliability and that sort of thing of what's there. And you know, I think like that sort of thing makes sense to have people on staff to do, you know, so they can build their expertise and keep, you know, keep that context that they have. Like, you use the word context like how AI uses it, but they have the context of the and the company and the goals that they can then use in their day-to-day work to, you know, continue to improve the platform. So yeah I… and that doesn't seem like that's going away anytime soon. So the, I hope there still are technical books and also maybe people of a certain generation, like they're just used… I know that's what I often do, like is, you know, at least what I used to do is like, you know, I'd wanna explore a topic and I'd buy a book. And if I'm really seriously invested in a particular area, like I would still up a book for that foundational knowledge. I also know too, like books go through so much more review and scrutiny. You know, there's a lot of editors and people involved in the production of a published book, that definitely counts for a lot
Ryan:Yeah, we got at least another 30 years of book buying in us, right?
Andrew:Hopefully, yeah, for at least for the authors that you know, wanna at least have some, a little bit of ROI on their activities
Ryan:All right, so you said you were speaking at a couple places coming up. You wanna share a little bit about that?
Andrew:Yes. Thank you. Yeah. Rails World is doing a thing that I hadn't seen before. I had the chance to go to Rails World a couple years ago as an attendee pitched Rails World, the main conference, on a, on basically talking about this work we did with Rails at Aura Frames. they thought it would be a better fit for this, the Rails at Scale Summit that they're doing, which doesn't have a lot of information publicly about it. It's a one-day summit. What I know is it's a one-day summit bringing together engineers from a few companies that are using Rails at a big scale. And we're gonna do short presentations. about all I know. So I'll be able to tell you more, but that's gonna be the day before the main Rails World event in Austin. So I'm gonna be headed to Austin for a short period. then Postgres has had a New York-based conference for a lot of years, and I've, been an attendee in the past, and it's been called PG Conf NYC. I believe it's been rebranded or… I'm not exactly sure on the details, but New York-based conference is called Postgres Summit, and it's in New York at the end of September, beginning of October. It's a three-day event. And I'll be presenting on the Postgres side with more Postgres detail there on our Aura Frames story. More of a typical, you know, 30-minute presentation. But also I, I really like attending that. I think it's, it's maybe-- Well, for me, it's been the best event in terms of the number of attendees and the technical depth of the topics. Includes a lot of Postgres core contributors that attend and present there. So they're always talking about… at this one, I expect to see a lot about what's coming in the next version of Postgres, which is 19, and then even, like, m-maybe groundbreaking stuff on, like, Postgres 20 stuff, which would be like 2027 things that they're working on, they present on. So that one I find super valuable, and it's sometimes it doesn't sorta, like, pay off in terms of real-world use for, like, two years after I learn something. But I've been able to learn things by attending that one that I put into use, which is cool. But yeah, those will be-- So Austin and New York in the, the next month.
Brendan:So you're, you're busy at the end of September it sounds like.
Andrew:Yeah
Brendan:Cool
Andrew:yeah, Au- Aura Frames has also been supportive of me um, writing on my blog about , this work that we did. We don't currently have at Aura Frames a, like a engineering blog. And so I said, "Hey, I write about this kind of stuff." And written about it there, and then they were supportive of, sending me to these conferences to present and so I appreciate that. And the… For what it's worth, the Aura Frames team is hiring, if anyone's listening to this that's interested in this kind of work feel free to get in touch with me, and can give you some more info
Brendan:Sounds good. Where can they reach you for that?
Andrew:If you wanna go through the official channel, you can, you can uh, email andrew@auraframes.com or you can just contact me through my website, andyatkinson.com. I've got a contact form there that just goes into my email
Brendan:Okay. A- anywhere else you wanna point people to before we jump off?
Andrew:I'm somewhat active on LinkedIn and Bluesky these days for social media, but otherwise yeah, that's it. Pretty much my You'll also find more info on the book. book's High Performance Postgres for Rails, for anyone that has not heard of it. You can find some-- I've got a book page on my site that has links to where you can buy it and some reviews and that sort of thing if you're curious about it and wanna see if it would be a good fit for you
Brendan:awesome. A- and for what it's worth,
Ryan:the
Brendan:yep, I have read the book, yep, as well. That's what I was gonna say. So
Ryan:Yeah
Brendan:definitely recommend it, so cool
Ryan:Well, thanks so much for joining us. We appreciate you coming on
Andrew:Yeah, thanks for having me.
Brendan:Yeah, thanks. Yep
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
Code with Jason
Jason Swett
Standing in the Fire
Very Good Software
Remote Ruby
Chris Oliver, Andrew Mason, David Hill
IndieRails
Jess Brown & Jeremy Smith
Startups For the Rest of Us
Rob Walling
Vibing with AI Code
Brendan Buckingham