Rails Business
Brendan Buckingham and Ryan Frisch talk about developing with Ruby on Rails and how to leverage it to build a business.
Rails Business
Rolling Your Own APM with Claude: Real Wins, Real Tradeoffs
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Ryan and Brendan talk through the growing pains of AI-assisted development — including fatigue, increased confidence in bigger code changes, and what happens when you stop paying for expensive APM tools and just build your own. Ryan walks through how he used Claude to identify memory pressure, fix N+1 queries, and cut his Heroku costs significantly in just a few days. They also dig into the real tradeoffs: maintenance burden, over-optimization, and why you still need to understand what Claude is telling you before you run anything on a live database.
LINKS
Questions or comments, email us at railsbusinesspod@gmail.com
Hey,
Ryan:Howdy. How's it going?
Brendan:Oh, you know, feeling a, a little worn out in some ways, but I don't really know if it's all the AI stuff, if it's just summer's coming and the kids are getting out of school, and I feel like I've gotta like do kids stuff all the time now
Ryan:Yeah, pr-probably a little bit of both. I mean, I feel like I've seen a lot about AI fatigue, and I can certainly understand it. Um
Brendan:Yeah. Yeah, in fact, I, that just reminds me, I, I saw sometime in the last couple days, Nunemaker posted somewhere, probably on all the socials, but I, I forget what he said, but e- exactly, but it was something along the lines of, like, he's been able to get a work-life balance back, and he, I think he was showing, like, his Git commit history, and it's like, know, he's reeled it in with AI a little bit, which was kind of a, a interesting signal, 'cause I know when we talked to him, you know, this has been s- two, three months now, you know, he was, "I can't get enough PRs out," and it's good to see that he's getting a little bit of a balance there, so.
Ryan:Yeah, I mean, I, I obviously can't speak to his experience, but, you know, that same feeling I'm feeling as well, right? Is like I, I am not feeling quite as overwhelmed by the number of PRs that are open and/or being closed or not. Um, you know, the, the number's still high, but I think my confidence in being able to push the PR through with less effort is becoming higher, um, 'cause I can do You know, I just pushed a feature this week, which was like a mon-- it, it was a monster feature, right? It was essentially a lightweight CRM component, you know, uh, deal pipelines, you know, the whole thing, a lot of JavaScript. And the amount of effort I put into reading the Java-- reading and understanding the JavaScript code was a lot less than I used to.
Brendan:Yeah
Ryan:Um, you know, I still QA via the, the web, right? And everything seems like it's working. That would've made me very uncomfortable two months ago, right? And now it's like, eh, it's gotten good enough, right? Like, yeah, maybe it's not the best JavaScript ever written, but it's functioning and it's not slowing anything down.
Brendan:Yep
Ryan:Um, but that… You know, maybe this is a good segue. Ended up rolling my own APM this week because I needed better visibility, right? For a variety of reasons. So I thought maybe we could talk about that a little bit.
Brendan:Yeah, definitely. I, I, I think I was thinking the same thing. We, we talked pre-show about this, and so I, I was also thinking that too. Like, what, you just said there about feeling more confident, my first question was gonna be, like, uh, well, obviously that probably led to building your own APM, but maybe just talk about that. Like, how'd that look? How… What, what made you decide at first? Or maybe is a, a good place to start, but maybe just also dive into the AI side of it like we were just talking about.
Ryan:Yeah. I mean, there, there's a lot of moving pieces here, but let's, let's back up. You know, I've had other APM tools in the past. Um, you know, the first one I ever used was New Relic. I used that for many years, and it got to a point where-- Well, A, it got super expensive, right? The price just kept going up and up and up, and I wasn't using all the advanced stuff, right? Like, literally, I think the only thing I cared about was queue time. So like, is my queue time good? Great.
Brendan:I think we have a whole episode
Ryan:Yeah, we, uh, we have. So I'm just recapping in case people haven't heard that, right?
Brendan:Yeah,
Ryan:So then I went on a, you know, a journey, I'll call it, of like, is there a better right size tool for me, right? And, you know, our apps. And I tried AppSignal for a while and, you know, I think that was an improvement over New Relic for me, right? And my-- There's, you know-- All these tools are great, right? They're super powerful, but, you know, there's different flavors, right? So, like, everyone has a preference. Um, I felt AppSignal wasn't as complicated. It gave me some more signals that I liked. But over time, again, the price crept up, and it got to a point where it's like, right, in between uses, this isn't, you know, worth spending on, right, consistently. Then I tried to move, you know… I, I, so I t- I toggled that off. They added a free plan. I talked to them for a while. I was like, Will this work? You know, same problem. You got a fire hose of data, right? I, I immediately hit every, you know, boundary they put in place for the free plan. That didn't work. I tried Honeybadger. Same problem, right? Uh, uh, right size price-wise, but, um, you know, with the amount of volume that I was pushing through, right, the price was gonna climb too quickly. Uh, so I just said, "All right. Well, let's see what happens if I don't have an APM," right? I left AppSignal still sort of installed in my app at the moment, and I could toggle it on and start collecting data, which, you know, doesn't help me for things that have already happened. But if there's a specific thing that I'm hunting for, then I can turn it on and collect some data, you know, pay for it while I'm using it, and then turning it off. Not the most ideal situation 'cause, you know, something just happened, I wanna be able to do a quick, like, what was it? What…
Brendan:Right
Ryan:So, like, that was it. It was price. It was fit, like queue time. You know, y- y- if you, um- Uh, Nate Berkopec, you know, puts out a lot about, you know, monitoring queue time, monitoring load time, response time. But I didn't have an easy way to get those metrics, right?
Brendan:Yeah
Ryan:So with Claude, I was like, "All right, let's see." Right? We had a You know, AI is causing traffic to go up to our apps. A lot of bots hitting us now, right? So our performance was starting to,
Brendan:said a ton of that as well.
Ryan:yeah,
Brendan:It's crazy
Ryan:performance was starting to tip in a not so great way, right? As things, you know, that can, can work for years and then traffic, you know, levels up and all of a sudden everything that was working fine gets a little sluggish. Um, I was starting to hit app timeouts, all of that. So like I, I just like, "All right, Claude, here's the situation." Right? And I-- we actually have two apps. So I did this on our, our legacy app, and I didn't build a dashboard or anything. I just said, "Claude, give me the queries you need to run. Help me figure this out." And I just sorta went. I was like, "All right. I see I'm, I'm getting memory pressure now, so like help me find places where I can, you know, quick wins that I can reduce memory pressure." And it was like, "All right. Here's the… Let's, let's add some instrumentation." And at the time I was like, "Oh, this is just gonna be why I'm doing it, and then I'll throw it away," right? Like I'm never gonna-- I don't need it after that. So I want-- I iterated for about three or four days. I was able to cut my memory in half
Brendan:Wow
Ryan:pretty much non-breaking changes. Like these are things that in hindsight are obvious, uh, but in the moment I was like, I didn't even know where to look, right? Which is a problem I've talked about a lot is like, it's like sometimes it feels like I don't know what I'm looking at, but Claude can figure it out, which is great. Um, then I started and said, "All right. Memory, memory's stable now," right? It's well under, you know, we're on a Heroku 2X dyno. We were on a memory, uh, Heroku Perf Medium dyno. So I went from a two and a half gig dyno to a one gig dyno, right? And I'm only using 500 megabytes, uh, on average, right? So like a huge memory profile in there, right? Across the board. Now that that was stable, I was still getting spikes like, all right, now I can start diving into the database queries, right? So I went to Heroku's dashboard and I said, "Here are the top, you know, slow queries. Find them." Right? And it gave me some middleware and was a-- you know, I, I deploy it. I'd run a query, it'd give me, you know, 20 minutes later and it's like, "Yep, here's an N plus one. Here's an N plus one. Here's an N plus one." And I just methodically went through and was like, "All right, let's fix these." Right? And through that process we went from I don't say consistent, you know, 30-second timeouts, but e-enough where it was a problem. Um, haven't had a 30-second timeout other than a blip here or there in, you know, a week now, I mean, 72 hours at least. Um, and the median, uh, the median, uh, percentile is, uh, 151 millisecond response times, and anywhere from, you know, the minimum 95th is just over 300 milliseconds to the maximum 95th is like eight, uh, eight seconds. So like, there's still some spikes that need to be dealt with, uh, but those are on sort of known endpoints that aren't hit a lot, not really affecting a large user base. So I'm sort of like, "Okay, I think we're in a good stopping point." The app is snappy, like compared to what it was. It feels good, um, to be clicking around in it, and it was a huge win. So I took all of that, and in our not legacy app, I said, "I wanna replicate this middleware, but I wanna do it in sort of an intelligent way, um, have a dashboard and do all that." And, you know, that was an afternoon's worth of work, and it hasn't had any noticeable impact on performance, right from a, you know, the middleware perspective. It has surfaced the offending endpoints, and I'm slowly working through that on the, on the newer app. You know, I've cut memory there significantly. And by the way, uh, was it-- I, I'm not actually sure of the pronunciation of it. Was it Gemlock? Is that how you-- The, the
Brendan:Uh, close enough. I don't, I don't really have a pronunciation I, that's different, I
Ryan:hopefully everyone knows what I'm talking about. Uh, that was a free win of about 200 megabytes right off the top. So if anyone's thinking about that, just enable it. It was,
Brendan:Yeah.
Ryan:uh, um, it was fantastic.
Brendan:I've heard it also called Gem Alic
Ryan:Yeah, GemAL. Yeah,
Brendan:like…
Ryan:I, I
Brendan:Yeah. case other people
Ryan:Yeah
Brendan:way in their head, they know what we're talking about
Ryan:It's, it's the memory allocator, right?
Brendan:Yeah
Ryan:um, 'cause I-- this, this isn't my area of expertise, but the one that ships with Ruby or is used on the server, however it works, um,
Brendan:Yeah, and I
Ryan:like glib, is glibc or something like that
Brendan:Yeah, and I, I mean, I feel like the Gemallec, isn't that one that, or Gem- Gemallock, whatever it is, like isn't that one that Nate talks about
Ryan:Oh, yeah.
Brendan:book and
Ryan:no, I, I totally ripped it off from him. Yep.
Brendan:Which
Ryan:an original idea.
Brendan:Oh, no, it, it's not even about that. I'm just saying that's something that he's also been talking about, so the fact that you could do it is great
Ryan:Yeah, so, um All right, so that, so that was the reasoning, pricing, whatever. Um, you know, I think some interesting
Brendan:know if you wanna keep going on or if you want me to ask a couple questions, 'cause I'm,
Ryan:Y-yeah, fi-- There, there's one other piece, but we can come back to AI. So ask the questions you got. Let's go
Brendan:Okay. Yeah, so I mean, first sounds like an awesome win because you reduced the cost of your application, and now you're no longer paying for an APM. So that's pretty cool. Um, now I just wanna make sure I had a, like a clarifying question. So you did all that initial work you talked about was the, the first app, and you're still working through the Locable app, or you're pretty much through that one too?
Ryan:Uh, no. So I've added all the instrumentation to the Locable app, right? I have a, I have an admin dashboard that I can go look at it, and it basically gives me, you know, top 10 slow queries, and I can drill down into it and see exactly what requests, uh, were slow and where it was slow. So was it in the DB? Was it in the view render? You know, what action? Was it memory? Was it query? And all of that is like, it's all on a dashboard. I click a button to get the detail, and then I, there's a single copy button that I can do the copy, and I paste it right into Claude, and I say, "Claude, help me fix this." And,
Brendan:Yeah
Ryan:you know, it does-- Claude doesn't always get it right. Sometimes it takes a couple iterations, um, but I have the data to tell me exactly where it's slow, and Claude is pretty good. Every once in a while, it got tripped up on, you know, it thought it was this thing, and we'd deploy it, and then, you know, uh, it would fix part of the problem, but then the real problem would, you know, become more apparent. It's like, "Nah, I kind of whiffed on that last one," but it helped us find it, so let's finish. Um, r- r-- So with the, with the Locable app, I have applied, I've, uh, I've applied a lot of memory fixes at this point. The N plus one queries that we have are a little bit more challenging to fix. And unfortunately, our slowest endpoint is one that is gonna take a little bit of effort to apply the fixes to, um, which I'm gonna get to, but it, it's gonna be a little bit more, I gotta be patient on that one.
Brendan:Yeah
Ryan:I've knocked out a ton of small ones, right? Um, I got a little bit more work to do right there. And, uh, actually, our, our biggest offenders are actually in jobs, which is probably why they ended up over there in the first place. Um, but, like, I have a huge memory allocation problem in like two or three jobs that are like, uh, you know, column reporting style jobs that run in the background that, you know, some of that memory is just gonna be necessary, um, because the, the report is big. Uh, some of it is like, all right, we're loading way too much stuff at any one time, right? We just need to find each somewhere, right?
Brendan:Yeah
Ryan:so I'm, I'm working through those, right? But I would say I'm like, I'm like 20% into that journey on the local app, right? And I've, and I've already seen like good returns, but there's a ways to go there, right? Like another four or five days worth of work probably to get to a point where, all right, like this is good enough for now.
Brendan:Yep.
Ryan:Um-
Brendan:Yeah, 'cause that, that was another question I had then. So just to be clear, you, you were able to reduce the size of your dyno for the, the non-Locable app, but Locable's still on the same architecture, so you, you haven't seen a cost savings there yet I- imagine?
Ryan:Well, let, let me try to qualify it. So, uh, the legacy app is we went from the 2.5 gigabyte dyno to the one, you know, the, you know, the, the perf-m to the standard two,
Brendan:Yep
Ryan:right? So that's $250 a dyno to $50 a dyno. And I'm running a few more dynos, but overall, like without the auto-scaling component, which we don't auto-scale as much anymore'cause I fixed the spikes, right? But, you know, we're, we're probably gonna save at least, you know, 500 bucks a month on that. And we're, we're a small app, so like, you know, that's meaningful to us.
Brendan:Right
Ryan:you know, the other thing I noticed a-as part of this work is that our cache was under-provisioned, so we were, you know, the LRU cache was ex- uh, expiring things faster than it could ca- actually cache. So like, we were paying for caching, but not getting any benefit for caching.
Brendan:Not actually doing it.
Ryan:Um, so I had to upgrade that, so like that ate into some of the cost savings, right? That was a,
Brendan:Sure
Ryan:hundred dollars more a month to jump up to the tier that we needed. Um, but that obviously made a huge difference too in the performance. Uh, you know, so there's a little bit of an offset there. With Locable, we've, uh, we've been on standard 2X for a while, right? So like the cost savings will be less on, uh, the newer app. For us, we'll still save some money,'cause if I can get rid of the spikes, we won't auto-scale as much, right? Like,
Brendan:Yeah
Ryan:can run on as little as two dynos, right? So it's like, you know, small beans. But on average, we're running, you know, eight to 10, right? And that number will go down if I fix the spikes. By how much, I don't know, right? We're still running at a high, um, uh, like, uh, pool u- thread, thread usage, right? So I, I could probably d- you know, go with s- more dynos at the base and spread out the load a little bit to be able to have some headroom. But, you know, right now it is purely a memory and, uh, response time, you know, and mostly query memory and query optimization problem at the moment for the Locable app. And that's just-- I just need to work through that.
Brendan:Yeah. And, and then of course there's the savings of not running AppSignal, I guess, that we already talked about a little
Ryan:that was I think the last The last monthly bill we had was approaching a thousand a month,
Brendan:Yeah
Ryan:like, I mean, we-- obviously I've turned that off so that that savings is already realized. But, you know, if we're factoring it into the overall equation for anyone else who's thinking of making the change, you know, put that in there.
Brendan:Right
Ryan:The, I'm trying to remember. There was one other facet that we were-- that you mentioned at the beginning about AI, but I don't remember what it was
Brendan:Uh, yeah, I mean, I guess I was trying to ask how much the, just the, the confidence you had in AI led you to actually build this thing
Ryan:Well, yeah, I mean, knowing that I ha- I can look at a dashboard and quickly see if there's a regression, right? Is sort of the other part of, um, you know, error monitoring in general, right? Like, it may not be erroring, but if, if, if it's, you know, the response time jumps five seconds, then that's a, that's a bug, right? So like, yeah, q- you know, I'm, I'm now monitoring queue time and response time, you know, obviously daily, hourly when I look, but weekly on our KPI dashboard to, to see the bigger trends. And, you know, after deploy, I'm, I'm sort of watching the, you know, the charts to make sure that everything is stable. So that gives me confidence. Uh, I'm also pretty much only, you know, uh, deploying AI PRs with testing now. I mean, there's almost always tests. My, my test cov- coverage has gone up about five, six, seven percent, um, in, you know, say the last, well, I guess it was the last five weeks if I'm being precise. It was the last time I measured. Um, you know, so that, that's helpful to know that that's going up.
Brendan:Mm-hmm.
Ryan:Right. The, the other piece of this is like You know, this, uh, doesn't have anything to do with AI, but thinking about it from a business perspective of spending the money for an APM, it's like I needed it when I needed it, and in between it felt like a waste of money, right? Which means it's a little bit reactive and a little bit proactive. But, like, if I'm being reactive, I really don't need, you know, two years of data. In most cases, I need about 30 minutes, or if I miss the window, I need a day.
Brendan:Yeah
Ryan:And very rarely do I need to go back 30 days or more, right? Like, and if I'm being proactive, I really just care about the last hour, right? Or the next hour,
Brendan:Yep
Ryan:right? So like, I have very-- I d- I did this all in a data, it just, you know, these are just tables in a database, right? I have a slow request table, I have a slow jobs table. I've got a couple other, uh, tables that roll up to that, uh, you know, and then I have a reporting table. So like a, for the detail What am I-- the detail I think is 30 days, right? I'm, I'm keeping 30 days worth of data. Now with jobs, you know, I've got 10 different queues, hourly roll-up, you know, 365 days a year. Like, there's a, a finite number of database rows that that's ever gonna populate, and that number is nowhere near a scale problem, right?'Cause I just purge automatically every night. It's like, if it's greater than this, get rid of it. So the history, the roll-up history of like queue times that I'm tracking, you know, I'm keeping a year's worth on an hourly cadence in the detail. I, I think I'm getting this wrong. I don't even k- think I'm keeping it 30 days. It might even be like 24 hours, right? Like, whatever it is, like it's bounded. Like, I did the math, and that table cannot grow that much, right?
Brendan:Yeah
Ryan:A- and even if it does, like, because it's mine, I can just say, you know, delete all, right? Like, I can start over, I can delete, I can trim it whenever I want just from the console
Brendan:Or, or export if you really need
Ryan:Or export, right? So
Brendan:control.
Ryan:control over the data that I save. I'm being very aggressive in making sure that it's not gonna grow to this table that's gonna cause its own performance problems. When I was building the thing, you know, the prompt I gave was like, "Hey, this cannot slow down. Like, the monitoring cannot slow down the app," right?
Brendan:And that would be my biggest
Ryan:Yeah.
Brendan:Yep
Ryan:And we, we ran the benchmarks. We're talking like micro… Was it micro or micron seconds? The US, uh, uh, whatever that stands for. Um,
Brendan:Oh yeah, I don't even know that
Ryan:like it's smaller than a, a millisecond, whatever the next level down is, right? Like there, there is v- very little overhead in the actual recording of the data, and I don't-- I'm not recording everything, right? So if it's a 200 response, I, I have thresholds set up, so like if it's a 200 response and it's under, you know, two seconds, like I don't care right now. Like that's not the threshold I care about, right? I'm only recording if it's greater than two seconds, for example.
Brendan:Yep
Ryan:have hard caps, like I'm never gonna have more than 50,000 records in the detail table or 100 whatever the numbers, I don't remember.
Brendan:Sure
Ryan:but I set that all up, all configurable. The only performance impact that might happen is on like the dashboard view itself, and that's like any other endpoint, right? Like it will show up in the report. I can optimize it. You know, I'm the only one accessing it or, you know, one of a, a handful of people looking at it, so it's like, all right, I don't… You know, if there's a problem, just leave it alone until you fix it, right? And it's not gonna affect the rest of the app. Um, I'm pretty happy with it so far. I mean, it, it's paying dividends, and it feels like I solved the problem. Um, and from a business perspective, it's a no-brainer from a, you know, I've saved money, right? And I
Brendan:Right
Ryan:more actionable, uh
Brendan:Yeah, I, I mean it, it, it's making me think a lot about, you know, the conversation we just had with Scott last episode. I mean, this was an example he even threw out there is you, you b- in his case it was you're starting your, a new app and you build your APM tool into the app. You know, obviously you're building it later, but I don't… Obviously it, it's pointing out that… I, I don't even wanna get into the whole SaaS is dead conversation, but it is just pointing out that same kinda thing. Like, you are making a decision to stop using a tool that you have used for years and years, and lots p- of us have, and now you're building it because you're getting… And a couple things that I'm hearing. One of them is obviously cost savings, but one of them you just said was it's getting you the things that you actually want. Like, you can tailor it to your specific use case, and it's now useful to you, where the other tools were not useful enough.
Ryan:Yeah
Brendan:to me is almost the more interesting part, because I've felt that way you know, even like, uh, uh, productivity stuff.
Ryan:Mm-hmm
Brendan:Like, how many productivity tools are out there right now? Like, thousands. How many of them I ever personally found that work for me? Like, none. And so I'm always… Like, I'm been trying, we talked about the command center thing that I've been doing, and I still do it, and it's working pretty well for me'cause it's, like, tailored to me now. Because I can just easily… And it's not even an application, but it's, you know, can just tell Claude like, "Oh, let's do it this way now," or, "Let's do it that way now," and it does it ins- instantly. I don't have to wait for a support ticket or whatever else to happen. So-
Ryan:What are the cons though, right? Like, so I, I've told you the things that I like.
Brendan:Yep,
Ryan:Yeah.
Brendan:what I was gonna say, but then there's all the… Like for you, I would guess the number one con is now you have to maintain this thing
Ryan:Y- right. Yeah. I mean, I, I do. Um, and time will tell whether that is a burden or not.
Brendan:Right
Ryan:You know, given the aggressive sort of constraints I put on, you know, the size of the database tables and, you know, that sort of thing, I'm hoping it won't be a big burden, right? But like, when would I not do this, right? Like, if my scale was bigger,
Brendan:Yeah
Ryan:I had a, a, you know, data pipeline that required, you know, much more detail, that would scare me away, right? Like, I have tables in my app that are, you know, a pain in my butt because they're growing so fast, and that's a different problem to solve. Like I-- You know, the existing APM tools deal with that much more gracefully than I will, right? But yeah, so I, I think scale of company and size of the app have something to do with it You know, as far as, you know, SaaS is dead, like I don't You know, I'm not anti using an APM tool. What I was anti is that the tool didn't work for what I needed, right? And I was able to build something, and it, it's also okay if this is a throwaway, right? Like, if it works for the next three months and it helps me get the app in a good place, and then it becomes useless to me, then I can pull the code out and move on, right? Or, you know, if we become an overnight sensation and our traffic spikes and I need, you know, greater visibility, then I'd be happy to turn one of those other tools back on, right? But
Brendan:Yep
Ryan:for where we're at, this was a very specific-- I needed slow requests and bad, you know, over allocation, and I could tune it just to that, to your point, is like I just care about these two things. Um, you know, there's areas that are in the queue time stack that I don't pay attention to.
Brendan:Yep
Ryan:really tu-tuning garbage collection, right? Like,
Brendan:Right.
Ryan:there yet.
Brendan:Mm-hmm
Ryan:I actually tried, um, to do that with some of the fixes, and it turned out just putting-- turning Gemini lock on solves the majority of the problem. It stopped the, the swap thrashing that I was seeing. Um, I do have a much deeper understanding of the Heroku and cache dashboards that I use and what, you know, the lever is now. Um, I'm feeling good about that. I still had to look up how swap actually works and what it m-means. That still tends to trip me up a little bit, but I mean, for the most part, the swap memory is very minimal now, um, and steady. So like from what I understand, that's a good thing and not a bad thing.
Brendan:Mm-hmm
Ryan:yeah, I don't know. The, it's, it definitely gives me more confidence with the AI stuff, um, just to have the observability and money savings as a bonus, um
Brendan:Yeah, I guess what other cons do you have from your perspective?
Ryan:Um, well, I don't know if this is a true con. This is like a res- you know, if a, an interview cop-out question, uh, answer to that question, right, style. But, uh, it's very easy to get caught in the flywheel of like, what's the next thing I can improve? What's the next thing I can improve? Eventually, I had to say,"All right, it's good enough. Move on." Like, my business partner was starting to get, I think, a little bit antsy. He's like, "Okay, like, it's good. Like, there's other stuff we need to work on. Like, don't over-optimize here." Um, you know, so easy to just be like,"Here's the latest slow queries." Dump that into Claude and be like, "What's the next thing that we can pick off?" And it's a bit addictive, um, to be able to do it so rapidly. Um, that, that maybe is not just a performance thing. That's like any AI tool, which is, leads you back to the, you know, the fatigue. But
Brendan:Right
Ryan:I got into it for about two or three days, I mean, around other work. I'd let it run. I'd come back and be like,"Yep, that looks good. Deploy. Run the query again." And, you know, it was fun watching the memory drop. It was fun watching the response times go down, and I, I could still go, right, if I wanted to. So that, that's a con. What other cons? Um- It, it was a three-day-- I mean, I think the distraction probably would've lived for me either way, right?'Cause I had to solve the problem for our customers.
Brendan:Right
Ryan:you know, so on the pro side, I solved it a lot faster than I would have been able to otherwise. I mean, that could have easily ballooned into weeks or months of worth of work that I did in two days.
Brendan:Yep
Ryan:also did a lot in two days, and, you know, I probably did more than I needed to. Um, yeah, I don't know. Uh, har-hard to say. Cons I, I could see that it would be-- You still need to sort of understand this stuff. A, a while back, I did some consulting with, uh, uh, Andrew Atkinson on Postgres optimization, uh, right after the, um, Sin City Ruby Conference. We know we met there. I know you met him too. Um, so I d- I did a few hours of sort of training with him. Uh, and I remember as I was going through this work this week and, uh, late last week, I was like, I understand what Claude is telling me because of that experience. Like, I understood what it was trying to do, and that was invaluable because I could correct it, right? Just like any other Claude thing, right? Like, I would be hesitant to do this if I didn't have the underlying understanding. Uh, same thing for code, right? Like the whole vibe code thing. Like, you can get something useful, but I'm still skeptical that you could maintain that long term without any understanding of how coding or architecture or project management or UI/UX design actually works, right? Like, might be great right now. I'm still very skeptical that it could work over months, years. I'd be surprised if that were the case, at least at our current level of AI, right?
Brendan:Right
Ryan:Um, same thing here with the performance stuff, right? If I didn't know when to correct, I'd be a little bit nervous, right? Sometimes it was like, run this, um, you know, run this SQL on the live database. I'm like, "Well, let, let's, let's make sure I understand what's happening here before I wipe out my production database."
Brendan:Yep
Ryan:yeah, so that would be a con. Like, if you're not comfortable with it, I'd be very careful about what you run on console.
Brendan:Right
Ryan:you know, make sure it can be rolled back or that you have backups or that sort of stuff. Uh, that could be a con.
Brendan:Yeah.
Ryan:Yeah, I mean,
Brendan:I, and I've even thought of that too, like with the work I do, which is, you know, uh, it's a lot of reviewing, but what I do a lot with my reviewing is, you know, I will tell Claude to, know, write… I love how it like abbreviates it as a repro script, but it's reproduction. I'm just like, "Reproduce this for me," but like give me the SQL queries, give me the Rails console queries, whatever. And yeah, I feel the same way with just even doing those even on staging 'cause it's like, okay, I can at least look at this and tell it's not, you know, a bad query. It's not gonna be a big deal. But yeah, I, I really wonder the same thing about people that… And, and obviously people have nuked their databases that probably know as well as we do as w- also, or maybe they didn't. I don't know. I don't know the full details, but
Ryan:Yeah, I, I, I know when we talked to John, he said when he was setting up his whole command center for his company that he used a read replica with read-only access.
Brendan:Right
Ryan:that's probably a pretty good idea just in general. Um, like I, I didn't give AI access to anything, right? So like I had to run the command.
Brendan:Yeah, and that's
Ryan:but th-
Brendan:do too. Yep
Ryan:that doesn't mean I'm foolproof either. Like I could miss something, right? So like in reality, the better choice is spin up a read replica, access the data there. So even if you accidentally issue a delete all command,
Brendan:Yep
Ryan:all you did was get an error back saying,"No, this is a read-only database."
Brendan:Right, right.
Ryan:yeah, so that, that, you know, I didn't do that. Maybe I should. Um, you know, mo-most of the stuff I was running against the database directly was like explain, analyze, right? So like it's, you know, making sure I pick the right query plan for an index or had the index that the query plan wanted. I should rephrase that properly. Um- Yeah. I mean, the rest of it was give me the last 10 records from this table, group by whatever data I needed that Claude asked for, and those are fairly safe as long as you give it a scan.
Brendan:Right.
Ryan:Um, yeah, I don't know. I, I'm, I'm loving what I have so far. Or you can-- I can report back in a, a few weeks to see if it continues to be useful. I mean, the nice thing is I can iterate on it pretty quickly. So like if, if, you know, if there's instrumentation that I need that I don't have, it's pretty easy to add it, um, with Claude. Um, none of this I could have done pre-AI. Like it was beyond my ability. Like I, I wouldn't have known how to register the middleware to begin with. So that's fun
Brendan:Yeah, I mean, I, I would, I would argue you probably could have gotten there, but it would've taken so much longer it wouldn't have been worth the effort, so
Ryan:Yes, I'd like to think that I was a good enough researcher in Stack Overflow that I've could have found it, but you're right, it would've taking, taken so long that it would not have been worth it, right? The, the, the time value, uh, equation flips back in, in the APM bucket very quickly there.
Brendan:Exactly. Yep. Yeah. Well,
Ryan:Frank
Brendan:this is a good place to just start wrapping up. I mean, I think, you know, my, my summary is that it's a, a pretty cool thing that you did it. I, I've thought about some of this stuff as well. I think our scale is a little too big, to your point earlier, where it's like would have too many performance issues. But I, I think it's, it's a good, it's a good exercise at least in like what are things like this? Doesn't have to be APM, but like what are things that you could… Doesn't even have to be in the, it also doesn't have to be, you know, thinking about how to cut costs, but maybe you do need to cut costs, or maybe there's a tool that you're just not getting what you need out of it. So I mean, it, it's worth considering if nothing else. But
Ryan:Yeah, no, I, I think that's always a good question to ask, right? But from a business perspective, you have to ask it from, like, is it cheaper for me to just buy this off the shelf,
Brendan:Right.
Ryan:right? Um, ch- and cheaper doesn't just mean financially, right? Uh, obviously that, that's a com-component, but like time and flexibility of the tool, right? Like if it's not doing the, the exact thing you need, like is that more valuable? Like you have to assign a, a value to that. Um, but I, I, I'd push back a little bit like on the scale thing, right? Is it purely just a volume thing for your company,
Brendan:Mm-hmm
Ryan:right? So like you could, you know, get around that by having a separate database that it reads and writes from, right?
Brendan:true.
Ryan:'Cause that won't block the thread
Brendan:Right
Ryan:And that, that was like I, I tried setting Rails Pulse up originally that like, and I think they've actually since fixed the problem, but like I had too much volume for the way it was set up.
Brendan:Okay.
Ryan:Um, I borrowed a lot from that when I built my version of the tool. Um, but their, their solution to that was, well, don't put the reads and writes on the primary database, and it's not a problem, right?
Brendan:Yeah, that makes sense. Yep
Ryan:you know, so like, you know, I think there is a line where, you know, paying for an APM makes sense, but if it is literally just to identify slow queries and you're not using any of the other APM stuff,
Brendan:Right
Ryan:right, it may still be worth it to roll your own, um, with a, a separate database. But, you know, I don't know your business, right? If it's millions of rows, I probably wouldn't bother, right?
Brendan:Yeah, exactly. Yep.
Ryan:Um, but maybe, I mean, what's the-- what does a million rows a month cost you? Right. I-is it the equivalent of a full-time engineer? Then maybe it does make sense to roll your own.
Brendan:Exactly. Yep.
Ryan:All right. Good place to stop. Thanks for letting me, uh, share that. I was excited about it
Brendan:Yeah, that's good. Glad you were able to share it.
Ryan:All right. See you next time
Brendan:All right, see you.
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
Code with Jason
Jason Swett
Standing in the Fire
Very Good Software
Remote Ruby
Chris Oliver, Andrew Mason, David Hill
IndieRails
Jess Brown & Jeremy Smith
Startups For the Rest of Us
Rob Walling
Vibing with AI Code
Brendan Buckingham