Rails Business
Brendan Buckingham and Ryan Frisch talk about developing with Ruby on Rails and how to leverage it to build a business.
Rails Business
Nate Berkopec
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Nate Berkopec (Speedshop) returns to dig into what AI actually changes about Rails performance work — and it's not what the hype says. The through-line: stop conflating understanding code with verifying it, because the real leverage is building objective, tool-callable checks an agent can loop against — N+1s, characterization tests, jemalloc — instead of shipping more unreviewed slop. Nate makes the case that the future looks like Intercom's "software factory" (tiny, automatically-verified PRs, most merged with no human review) and asks why the rest of us are still hand-building 4,000-line monstrosities. Plus: why "intelligence" is the wrong word for what these models do, and a first look at Thread Pilot, his new automatic thread-pool tuner for Sidekiq and Puma.
LINKS
- The Complete Guide to Rails Performance Book
- On Rails Episode with Brian Scanlan from Intercom
Questions or comments, email us at railsbusinesspod@gmail.com
we got Nate Berkopec back with us today, and, really , glad to have you on again
Nate:Yeah, glad to be back, guys. uh, I've, I've become such a long-term listener now. Um, I listen to every one. Although, actually, I didn't re- listen to the most recent one with, with Ernesto, which I should have. Um, but, uh, uh, yeah, I'm, I'm listening all the time and I really enjoy this podcast
Brendan:Awesome. That's, that's great feedback. Glad to hear it.
Ryan:I, I love seeing the, the blog post that you come out, uh, two or three days usually after we launch with, with your opinion. I love that, and I'm gonna join the so ke-keep it coming.
Brendan:Yeah
Nate:why I like doing this stuff is it gets me like, you know, and fired up, and it's almost like to do the hallway track at a conference for an hour, and so it kinda, you know, tends to unblock things for me creatively. So, uh, thanks for
Ryan:Yeah. Yeah, my, my favorite
Brendan:welcome
Ryan:right, right after I, I can't remember if we had a guest on or if it was just me and Brendan talking, but we were talking about, like, the things we liked about, know, how 37signals business works, and like three days later I see your blog post come out and you're like, "Don't do everything 37signals tells, you know, talks about." And I'm like, "Oh, that's a, that's an
Brendan:Yeah
Nate:don't,
Ryan:take." I, I, I don't necessarily disagree with
Brendan:I, I think that was the, that was the, um, writing ideal code, I believe is the one you're talking about
Ryan:th- this was a while ago now, but
Nate:yeah. That actually was based on something you said. Yeah.
Ryan:Yeah. I was like, "Hmm,
Brendan:A- and the reason I, I think I found that one because I had people, like, sending me texts like, "Whoa, did you see your, your podcast has blown up?" And like, Nate Berkopec retweeted you, and, like, the 37 Signals guys did. And I'm like, "No, I actually haven't seen that yet." So that was, that was fun.
Ryan:for that. I appreciate it. And
Brendan:Yeah
Ryan:for anyone who followed that, uh, that, that tweet thread, I, I, I think I generally agree with everything you said in there, right? It's, uh, the i- the ideal is, like, great, but not everyone operates the same way that they do or can afford to operate the same way that they do, which I think was your sort of main point is, like, don't emulate it if you're not them, right? Come up-
Nate:of, it's part of my luxury as a consultant to, you know, have worked with over 100 companies at this point and to have seen so many different situations that, you know, I see a lot of indies, a lot of people in your guys' situations sort of like beat themselves up over code quality, over, uh, over trying to emulate a… what they believe is the ideal of, of companies like 37signals or others. And like you're just not in the same situation. You don't have the same resources, you don't have the same constraints, and you don't even have the same goals. Like 37signals is a, uh, what do we wanna say now? 20-year-old successful company with a lot to protect and a lot to, um, a to retain and keep
Ryan:Mm-hmm.
Nate:And most people in your situation are like to make it to the next six months or the next year. Um, and that, that-- those constraints produce very different code. Um, so yeah, I, uh… It's, it's, it's to me, it's these constraints, it's these systems that produce code, not individual to, to, to the, to a hu-huge extent, individual brilliance or, or, you know, all those people are all just geniuses and, and super, uh, super talented technical people. They are, and that is part of it, but the majority of it is, is the circumstances that you are-- that the business works in
Ryan:Yeah. A- and I, I think that i- is good grounding for anyone who's listening, right? At, at, at the same time, I like to look at what they do. I agree with a lot of the stuff they do, and then I have to make a decision of like where are, you know, me or whoever, you know, we're talking about, like where are you on your own journey, and how does it compare? And you have to be don't think I ever beat myself up. I wish I could spend more time on it, right? And I, I can't, right? And that's fine. That's just where we're at. But it's still an ideal that I can strive for if I agree with it, and I don't agree with everything put out there. Uh, there's a lot that I do, but, um, you know, for business and code, and those are You know, they talk about those two things quite a bit, which is good. I like that they, they work in the open, and they're very transparent, and it gives you the ability to listen and learn and then take the things that you like and apply and discard the things that don't work for you. So,
Nate:Yeah
Ryan:uh, anyway, um, I, I kept that tab open for a long time, meant to jump into the conversation, and then too much time passed. I was like, "All right, I'm not gonna, you know, go back and stir it up again," but maybe I will sometime
Brendan:Uh, that's all right. We were still part of it either way, right? We started it kinda So I mean, this is probably a good segue then to kind of our, our first question, which is, like, writing ideal code, you know, that comes from them sharing things. They're not selling pro- or, or they're not selling courses, but you know, you sell courses, you sell books. Maybe you're not going for ideal code, you're going for, like, speed and those things. But at the same time it's like, th- they're related, right? It's like w- you, you have the courses to try and teach people how to fix their, their performance problems, I guess the, maybe the easy way to put it. I'm curious though, like, how is, how are things going with your courses at this point, and your books, given the world we're in at this point?
Nate:Yeah. I mean, I have not really updated any of my courses in a very long time. The last thing I released was in Practice. Well, I mean, I think I probably… Well, yeah, I think it would've been Sidekiq in Practice, which was, like, four years ago? Like, it was three or four years ago or something like that. I haven't-- I don't think I've updated that at all except for bugs and typos since. Um, and The Complete Guide to Rails Performance came out 10 years ago, uh, came out in 2016, I've updated that a few times, but mostly just for sort of like factual accuracy and, and, um, things got deleted in Rails or, like, certainly I, I think I just deleted, like, the entire Turbolinks chapter I had originally written. Um,
Ryan:That's funny
Nate:really expanded onto it. Like, I never wrote a new Hotwire chapter. I never wrote a chapter about, um, um, Rails six multi-database support added by Eileen. Like, that's probably the biggest performance thing that's changed in the last ten years, and it's not in the, it's not in that book. so, like, I, I've done very minimal kind of changes to all of my written material. And, um, so as a result, like, all of my revenue graphs just look like, you know, big launch, sort of long, slow decay. And I don't know, I maybe make a few hundred bucks a month off of that stuff, um, which is perfectly fine and intentional. However, um, in the last year and a half, two years almost now… Yeah, two years, I've switched to spending most of my time on a sort of ongoing retainer basis with a large number of companies, that, that offering works really well for them. But it means that I'm not… First of all, I don't really seem, like, seem to have much time left over to, like, write, and I, I, I sort of nail, like, a particular customer with that offering. Like, it's very good for, like- Like six employees up to like 100 employees. And companies that are a little bit bigger than that, I have to figure out something that, that I wanna do with them. And then companies like you guys, like the-- this is why I love listening to Rails Business Podcast because it's like you're such like a, a, great, uh, representative of this section of Rails people, of like the, the indies. Um, and That like 50 to $200 online course was like the perfect point for people like you. It's the perfect offering. Like you have more money, you have more time than you have money. I also find that indies are, are more interested in investing in thems- uh, have some interest in investing in themselves personally. Like they, they want to just have the problem solved, but they are interested in i-investing in a way that they could solve this problem again in the future without my help. Whereas like a lot of the bigger companies I work with are like, "Just come in here and fix this thing. Like just, just, just, just fix the-- stop the bleeding, please." Um, and so the course was always like the, the entry-level, like perfect fit for people like you guys. and I, I, I was thinking about this the other week, and I was like, well Would I wanna, like, update all of that stuff? And, like, what-- would I do it differently? Would I, would I have a different format? I've thought a lot about, you know, putting together s-stuff in-- I, I kind of had some experimentation with the format in Sidekiq in Practice, where Sidekiq in Practice is book, but there's also, like, these interactive exercises that you do after every chapter. like, that was-- Anyway, it's, like, done through a, a CLI. Um, that was kind of, like, an experiment, and then I was like, "Okay, well, now in two thousand and twenty-six, like, would I change the format again differently?" And just personally in, like, my own and experience, um, you know, we're just sitting, talking, prompting agents all day, and I wonder if there's even, like, a different need now. Like, do we-- do people still want to read a book or do a course in two thousand twenty-six? Like, do we even have the patience for that anymore? Like, I, I don't know if we do. like, I don't believe in-- There was like-- There was some sort of talk, I would say, like six months ago, where people were like, "Well, you're gonna, you're gonna send your agent to school, and, like, y-your agent's gonna take these courses and come back a super performance master." And I was like, "I don't think that's gonna happen." And it really didn't. Like, no one's, no one's, like, sending their agent to a course. Um, it doesn't make any sense, right? Like,
Brendan:Right.
Nate:so I… And, and so, so what, what is-- what do we want in two thousand-- the summer of two thousand twenty-six as we, as we talk now? Like, a course? Is it a, is it a, is it prompts? Is it context? Is it, um, you know… What, what format are we trying to deliver technical skill improvement, like what I'm trying to do, but in this new AI landscape? Like, what do we, what do we think?
Ryan:What, what do you-- I mean, I have a lot of thoughts on this, right? But before I just spew them out at you, um, what are you trying to accomplish, right, from a business perspective with, know, this? Is this just a, a side thing that you enjoy doing? Are you trying to drum up business?'Cause that, that, that can frame, you know, the response, you know, to you and then also
Nate:Um, I think, you know, where this has fit in my business for a long time has been… had two purposes. One is… Well, three really. One is, like, it's the entry level of a ladder. So I have, like, basically a, a service ladder that goes from these, like, you buy these written mater-materials, like, upfront. It's 50 bucks, and that's it. Um, gets you into my Slack channel. Like, it… There's sort of like this, like… That's, like, my, my entry level of a ladder. The second part of the ladder now is the retainer uh, is $3,000 a month. And then really do this right now, but in the past, I, I have done, like, even higher-end consulting or training has always been a big thing for me. Like, you know, trainings in the five digits, um, o- on site or, or remote or whatever. So, like, that's what my, my ladder looks like, and the, the, uh, writing has always been the base of that. Um, so there's the ladder component. There's, there's definitely, like, a building authority component. Like, it's always been like,"Oh, Nate's the guy that wrote the book on Rails performance." So the more… I, I just think the more you write and the more you publish, the more, authority and, and inbound you build, so that's, that's always been a huge part of it. then, yeah, it's made, it's made money, not as much for me recently in the last, I don't know, five years. Um, it would probably be, don't know, less than 20% of my top line, 10% even maybe in some years. So maybe the revenue doesn't matter as much for me anymore, which might be an interesting lever.
Ryan:I'll say
Nate:but
Ryan:Yeah. So like, I'm your-- You know, I'm definitely in the, the, the group of less than six employees, right? Like we, we have, I think we have seven people on staff now, but I'm the only developer, and we contract out to two or three other people as resources allow, right? very small team, you know, for all intents and purposes, independent, right? Um, you know, for me, like love reading your books or the PDFs or whatever you wanna call it, right? But for me, that's learning. Like, I wanna level up my understand- like my baseline understanding of, you know, the levers to push because I still, you know, despite talking to you in the past and despite reading all your content and despite, you know, any course that you can think of, like I still get tripped up when I see some performance stuff. So like I, I know there's another, another level that I need to go. That said, with my day-to-day, I'm in the like prompt and agent, right? Like I've-- I, I think we talked about this last time we were on or, you know, maybe in between, but like I've now built sort of my own tooling, very lightweight tooling to give me just what I need for the agent, right? Like I've, I've got it down to like I can copy, you know, I get, you know, my N plus ones, my memory consumption. I can copy like the context. I can drop it into an agent say, "Hey, go fix this." And I can do that loop when it pops up, right? Like when I'm having a performance problem, I can identify it quickly, I can fix it. I don't always understand it, which is why I go back to the learning. you know, so when you said it, it, you know, AI context or skill, you know, that I could, you know, make my agent smarter, you know, with your know-how would be appealing to me. know yet if that would actually move the needle any more than just prompting the model, you know, outright with no skill, But, you know, I still struggle with the tuning of the, you know, the, the, the work, you know, number of workers, number of threads, you know, on something like Heroku, right? You know, I, I, I'm, I'm getting smarter at that, I, I think. Like I, you know, every time I look at it, I feel like I'm like: Okay, I, I understood what I did here, and then it breaks like three days later. I'm like, Well, did I really understand it or did I get lucky, right? So, all that to say is I, I think there's two separate tracks, right? One is, you know, when I'm in learning mode, you know, I want to level up my own understanding, and I think there are times where I've, you know, I'd want to reach for your retainer service, but it may be just out of reach, right? And, you know, like, you know, maybe if it was like I had a really big problem and I don't know, like and I needed you for like a month, right? That would be great, but like that's not a retainer, so I don't know where you land on, you know, popping in, popping out, right? Like it's hard for me to say, "All right, I'm gonna give you a retainer for 12 months at three grand," like'cause I don't have performance problem for 12 months out of the year. I have a performance problem for two weeks, and then, you know, it gets fixed, and then I move on. So like A lot of just different context to put back into the conversation. don't think I answered anything, but those are the things that popped into my head
Nate:What have you, have you been using any books or courses to learn anything over the last six months?
Ryan:I subscribe to, um, both Drifting Ruby and, um, Chris' GoRails podcasts. And I, I would say part of that is because I like that they're publishing content, and I just wanna support them. I would say it's sort of like what I just said with the retainer. When I need it, I need it, right? They, if they post a, a, a, an episode or a course or whatever you wanna call it that is interesting to me to learn, I'll watch it. I'm not religious about it. When I find the most value is simply it's like, "Oh, I have this set of problems," and they happen to have covered that set of problems. Now I can go and watch, like, a series of three or four videos. It's like, all right, this gives me
Nate:Yeah
Ryan:information that I feel like I can craft a prompt that actually will help me solve the problem, right? Uh, courses have definitely become less of… I just don't have the time to sit and watch it all the time.
Nate:Yeah, I gotta, busy avoiding the permanent underclass. I don't got time to read
Ryan:Yeah. That, that said, like, you did, I think, you, I think it was the si- it must've been Practical Sidekiq. I'm pretty sure I bought that and never actually looked at it, right? But I'm still interested in that. It's just like I gotta come back to it, right? So like these, like sort of these self-contained classes where you buy once, right? Even though, and this is a bad example, I never actually got through it. Like, to me that is… I'd be willing to invest in that because at some point I will actually go through it, right?
Nate:Yeah, that was my, that was my own thought and reaction as you were talking, was that Sidekiq in practice was designed really for you to sit down for a day or two and walk through it and walk away with a deeper understanding of Sidekiq. Whereas The Complete Guide to Rails Performance was actually, I-- not really at the time, but in retrospect, I realized it was more of like a reference manual. It was more of like something that, "Hey, I have N+1s. All right, I'm gonna read the chapter on N+1s from Nate's book," and you could kinda go back and forth with it. And it, the chapters didn't really build on each other, um, in the way that my Sidekiq, uh, course did. So, um, I've
Ryan:I think both are useful
Nate:that the reference manual
Ryan:Both are useful, right? For different reasons. Uh, just I don't know how you moni- like, if, if monetization is your thing, like, I think that's gonna be a tough, you know, uh, business strategy, right? I think both of those things are now getting harder to monetize just because of the AI. Um, but it's good for authority, I think, in, in, in all cases
Nate:Mm-hmm.
Brendan:Uh, I would just add too that, I mean, I've got both books, and I do use them basically as just reference manuals as well. Like I, like you just said, Nate, like if I have an N+1 or whatever problem and you've got a chapter on it, then I'm gonna go there first rather than like searching the internet or even asking Claude or, you know, any LLM. Although I probably could just say, "Hey Claude, what does Nate think about about this stuff?" And it probably would know, or at least most of it, so
Nate:Yeah. Yeah, I think a lot about You said something earlier about understanding, I think a lot about that word because it's something that comes up a lot on, on Twitter about like when peop- when-- in this debate about like, should we, should humans be reviewing code anymore? Um, is kind of usually how that's brought up. And around the human review of code, it's like people say: "Well, well how, how are you gonna maintain it if you don't understand it? If you don't understand it?" And it's always the word understanding. It's like someone has to have understanding of these changes. to me, it's-- I don't think it is… I guess, I, I guess I would add on to that is like I, you know, Tobi does not understand every single line of code that gets committed at Shopify. Um, does David, uh, I think probably. There's probably enough code going through Basecamp that he does not, and, and, and enough domains that he does not deeply understand every single line that gets committed. and routinely ask non or semi-technical people verify software work. Um, we call it QA. Um, and they don't understand what's been done in those changes, but they know how to verify the quality of the change that's been made. That's a whole different skill set even, you would say. So I, I think a lot in bringing that back to this conversation, about how agents change the work that we are doing in two thousand twenty-six. And I, I start to wonder if like my job is no longer to… You know, I think maybe I thought before that my job was sort of to like create, mm, like c- ideally like create a bunch of clones of myself, you know, and other people to like, to like download my understanding into their brain. And, uh, particularly now with agents, you're, you're, you're able to do things that you don't one hundred percent understand and, but you can verify the quality of the work without understanding. So what does that look like as an example? Um, N plus ones I think is an easy one. You… Understanding how, how and where the best, uh, uh, how to fix an N plus one, what change should be made in the code, what effects that has, is different from how do I verify that an N plus one Was fixed. Um, there's a couple of ways you could verify that N plus one is fixed. You could do it in a kind of manual way and say, um, "Given this test data, when I run this controller action, here's the log from Active Record before and after," and you can see, okay, in the before log, it goes to the post table three times, and in the after log, it goes there once. Um, and there's more automated tools like Bullet, and you can say,"Okay, before this thing raises an exception and after it doesn't." Um, so the verification and the understanding of the fix is actually different. I, in the past, I think actually what I've been frustrated with, with working with people is that they never verify, uh, their performance changes. Like, they're not actually… They're, they're putting things up which they may or may not understand, and then assuming that the verification is true, that it worked, and then deploying it, and then it's not. Uh, what agents force us to do, because we see that these agents only succeed eighty percent of the time, is to verify and externalize the verification process so the agent has to prove to you that these things worked. Um, however, so with all that out of the way, it is creating, um, a verification to require understanding. Um, creating a verification process is difficult. I think it's probably you both have and learned.
Ryan:And or expensive. Then
Nate:yeah. Yeah, sure.
Ryan:that, that's the other constraint that we have is like, I mean, there are tools out there that you can spend thousands of dollars a month on, depending on the size of your app, that will help you verify performance.
Nate:Right
Ryan:you can't afford 1,000 bucks an a- a month, as an example, then verification becomes harder, right? The, uh, not impossible. There's just, there's varying levels to it, and I'll, I'll also think, say, you know, I think there's varying levels of understanding that you need to verify, right? Like, you can have a deep understanding of the problem. You can have a, a shallow understanding of the problem, and depending on what the problem is, the shallow one might be fine to verify. Um, I would say it's like I don't have the level of trust in AI to say that I don't need any understanding. I want some basic level of understanding, but I think that's maybe my own problem and not necessarily a, an actual problem. I, I'm curious what you, you know, what you're seeing out there. Like, do you believe that, you know, AI is capable at this point of sort of doing the right thing, on its own, right? Uh, especially in the performance work, right? I'm, I'm curious what you see.
Nate:Well, here's what I definitely don't believe. I don't believe that you-- I think there's, like, a influx here. There's, like, there's a, there's a desire here for people to think like,"Oh, I need to lower my standards." Like, that's what, that's what, that's what this seems to mean to most people is like, changing how human review of code is done seems to, like, be a synonym for most people of I am going to ship work, and the trade-off is that I will get more of it. Like, that I will be able to increase velocity, and that's, that's the trade-off that everyone else is making is kind of the FOMO feeling, therefore, I need to do that. And I don't believe in that at all. Like, I don't, I don't want to reduce the… F-first of all, I think it's a false trade-off. Like, what the hell is the point of shipping more stuff if it's lower quality?
Brendan:Right
Nate:like, don't think that, that-- think that that's necessarily the trade you have to make. I don't think you have to ship things that are lower quality to get more velocity. Um, what I'm trying to figure out what are the ways that we can AI to Uh, with non-- a-any, I guess any way to ensure quality in a software change that is not just human review. Um, and I'm trying to spend less time for… Okay, as an example, I'm trying to spend less time, um, reviewing individual performance PRs. And the way that you can do that is have the agent demonstrate to you that it was faster. So here's my, you know, this is stuff I used to have to pull teeth from people to get them to do, is like, "Write a benchmark and show me the before and after results. Um, write a script that demonstrates the N plus one in the controller before and after and how you fixed it, um, like by actually running the code and like showing me the log." Like, agents are perfectly willing to do all this stuff, this drudgery that how you verify a change, um, that people were not willing to do before. And if I can do that, then I don't have to do that part, and my review becomes m- either, either it's-- it can become higher level and more like in-depth, or it can just be faster. And like that is what I'm trying to do. I think-- I just think we're not really being very, um, what's the word? I just think we're not thinking very deeply about what human review is for and what, software review at all is for. Um, what is quality? What is, what is, what is quality software and how are you ensuring that through a review process? And instead, we're just kind of repeating the processes of the last 30 years and saying that this is, it-- this is what is good and how it must always be done. Um, and I don't subscribe to that. So I guess, to come back to how you posed the question, am not in a world where I think that like The other, the other lens here is like, I guess is like AI is-- people are, "AI is getting smarter. It's getting smarter, smarter, smarter," and like the word intelligence. That word really bothers me because it's like that's not what I'm seeing the last six months. Like, they're not getting smarter, they're getting more capable. like I think this-- It's a good example is to think about the, um, the Hugging Face, uh, breach, I guess, that happened just this week as we are recording, where basically the paperclip optimizer became real and a OpenAI model like, you know, found some zero days to like, uh, hack into Hugging Face and like cheat on the evaluation it was under. Um, like that's not intelligent behavior That's, that's paperclip optimizer behavior. And I just think about these tools in a way that's like, how do I, how do I build a system around the paperclip optimizer such that it will mow my lawn and not chop my head off? Like, so I guess that's kind of what I was dancing around and ask- having-- talking about this question so far is like, is, is the fifty dollar product that Nate sells these days, is it less about, uh, building people's understanding in their brains? And is it like, how do I-- can I, can I give you something at that price point and range and like form factor that's more about like, it's more of a system? It's more of like, a s- a number of workflows that allow your agents to fix the kinds of things that I used to have to teach you how to do manually, but now these things can more or less run on their own because they can, they can, they can find a problem, they can a fix for the problem, and then they can verify the problem, was fixed all autonomously. Um, uh, know if there's like a good… There's really not like a comparable for that right now, I think.
Ryan:appealing to me, right, to have the, the recipe so that I don't have a better word for it right off the top of my head. But, you know, a, a lot of it is just, you know, you've built up enough brand authority over the years that I-- when you say something, I trust it. And maybe that's a problem, but it's the truth, right? Like, so, you know, if you're able to take that experience and say, "Here's the recipe that I follow now to, you know, verify the work," and, uh, that would be valuable both from a learning perspective, but also as a tool I can use. To bring it back to what we were saying before, it's like, all right, those are the two reasons I do something. I either wanna learn or I need to fix a problem. And what you just described theoretically do both. I'm, I'm learn-- I'm not learning code at that point, but I'm learning a thought process. I'm learning, know, a, a system of thinking. Uh, you know, so it's, it, it is abstracted up a level, and I don't know if anyone else would actually think of it that way, but I do. Um-
Brendan:Well, one thing I like to think about, 'cause I, I do a lot of code review, and I, I actually have been asking my agents more often to, to look at the code, but also tell me, like, how can you reproduce this, or what can we run? You know, so it's similar to what you were talking about, Nate, with, like, how do you get it to prove the N+1 is fixed.
Nate:Hmm.
Brendan:be kind of anything. And a lot of times it'll be like, there's really nothing to do here. But sometimes, and I've found quite a few times where, I can't think of a specific instance at the moment, but, like, it finds things that are either, like, little security holes, or maybe it is, like, an N+1 that wasn't even obvious yet, but then it kind of re- reproduces it and it finds it. So I've been pushing on that, but I've also been trying to think about it from the perspective of … And I've even told my boss this too, like, AI is great. We can write more code, and y- you already touched on this too, about how, sure, we could write a lot of code, but it might just be crap, and why do that? And I keep trying to paint the picture that we can't increase the number of pull requests or code we're pushing out until we fix the process of even getting the code out that we got done before AI existed. And that all kind of comes together, and we haven't figured that out yet. It's still a problem to, to figure it out, like you, you've kind of been describing here.
Nate:I think AI is an incredible tool when it has a process of-- an objective process of that can be accomplished through tool calling. So getting back to, like, the intelligence thing. Um, So we're all kind of reading these headlines this week particularly about how AI is solving these, like, math conjectures that, like, haven't been solved for a hundred years or something. not just like, to be clear to everybody, like, they're not, like, just, uh, s- burning, like, ten million tokens and then coming up with a solution. They're burning tokens, writing a proof in Lean, and then seeing what happens, and then-- And, like, it's, it's a, it's a, it's a, uh… They use, like, formal math conjecture proof languages like Lean or some other, whatever they use, I don't know. Um, and then going back and forth with a tool call and, and working their way to, "Oh, actually, this is proved now." and that looping is what is, is the powerful part, is the loop of burn token, tool call, burn token, tool call. And the fact that, you know, math is math and, like, if it works in Lean, it is correct. It may not be meaningful, it may not be a useful result, but it is formally correct, is what has allowed these-- this progress in mathematics. Um What-- the problem that we have in software engineering is we don't have, uh, a domain where it's like, if it's, if it's true in Lean, it's true. Like, what, what, what is a mergeable pull request? Like, there's probably a list of like 100 different items you can think of that would be like, uh, you know, does it have any bugs? Does it, uh, have… And, you know, there's probably, I don't know, a thousand different sub-classifications of bug you can think of. Um, is it performant? Is it secure? Uh, is it legal? Like, does it break any laws that we're operating under? Um, uh, there's probably hundreds of different things you can think of, and not the majority of those probably don't have like a tool call that you can loop over and say, "Is this true or not true?" And partic- like, I always, uh, I was kind of even before agents frustrated with the, like the maintainability people of like, what does it mean for software to be maintainable? And it's certainly not an objective decision, at least the way that these people have presented it over the last years, is like there's no maintainability score that you can walk back to and, and optimize. And I do think it's something of a failure of our profession that we have resisted quantification of certain pro-- concepts for so long. Um, and it has, it makes progress with agents difficult because there is no, there is no tool. If there's no tool, if there's no loop, if there is no, if there is no burn tokens, try thing, yes or no loop that can go through, then agents are not really that useful because they require a human being sitting there to guide it and steer it, which is not that much of a gain. Um, so I, I, I've s- I guess what I'm seeing in my work with people is that, um, there are certain problems in performance that fit that loop. N+1, the one I keep bringing up, is one that fits pretty well with that. Um, you can prove whether or not you fixed an N+1. That doesn't mean that every possible N+1 fix is good because it can fail the other 99 things. Like you can, you can… I cover this in my RubyKaigi talk. Like, you can pro-- you can, you can propose something absolutely ridiculous to fix an N+1. It's like, yeah, you fixed the N+1, but you added 300 lines of absolute nonsense. Like, I'm not gonna, like, I'm not gonna maintain that. So, um, think to me that is, that's the juice right now, is like figuring out what is verifiable what, is repeated work that can be verified without human input.
Brendan:Yeah, that, that makes sense. And it's got me thinking about some stuff we talked about pre-show and, you know, like, I think one of the things that you've been thinking about a lot with your engagements that you, you've talked about with us was, um, just the fact that, like, you're not solving just, like, one N+1 anymore. Like, you're looking to figure out how do you solve maybe all of the N+1s in a, a code base, right? Because now you have those agents to be able to look at, at least analyze more, push out more pull requests. Again, who knows if they're right. We gotta figure that part out.
Nate:Mm-hmm.
Brendan:yeah, I guess a- as far as, like, a, a question though, I mean, w- is that what you're seeing, though? Is that kinda how you're approaching engagements, is taking advantage of AI to be able to do more like that?
Nate:Yes, but I would say that I'm not-- I haven't gone as far as I wanna go with it. Like, I, I definitely think that LLMs have helped me to, um, extend myself across higher numbers of clients in ways that I absolutely could never have done in the past. Like, I think one example was is like, I have, I don't know what, maybe like 18, 20 different pull requests in flight across a dozen different clients right at this moment. And before AI, keeping those, um, pull requests open, and like ready to merge without conflicts would've been… That would've been like a full-time job by itself. But now I have a, I have a thing that looks at all my pull requests and says which of these have merge conflicts, and then if they do, I just fire those off to an agent, and I go do something else. And then they
Brendan:Yeah
Nate:It pushes them up, and there's no more merge conflicts. Um, and There's certain classes of performance problems where I think I've basically kind of gotten to the stage of, like, I have a prompt that works most of the time. Um, I don't really have what I would call, like, a real system or a factory around them. Like, I, I think that there's certain classes of change, you know, um, I think a lot about converting, um, tests from f- uh, factories to fixtures being the class of problem here. Um, N plus one is definitely another one. But, um, I've, I recently had an agent, um, go through all my previous reports and kind of like categorize all of my different recommendations down into like-- this is also a thing I've meant to do for like literally years, and like now I have an agent. This is, this is a perfect size task for them, right? to try to like get a list of like what are like the things that I've ever recommended in a report, and I think I ended up with a list of like two hundred things. And
Ryan:Cheers.
Nate:some percent of those I think well with, with agent work. jemalloc maybe like is another good one. Like, um, there's certain ways to, uh, like verify that an app is running with jemalloc. It kind of really depends on where it's deployed and how it's deployed, but like there's various ways to verify like is this app running with jemalloc or not. And it's not, it's not like unknowable. Like, there's tool calls for that, right? Um, and there's other problems like that, I think. Um, and I think the extent to which AI is a productivity boon is the extent to which we can do and make these checks and make these proposed changes to make the check pass without large degrees of human intervention. Like, I think the last thing we need is like another thing that opens up a hundred PRs you're never gonna look at. Like that's, that to me
Brendan:Right.
Nate:of the course you buy and never r- and never do,
Ryan:Right. Yeah
Nate:is, is gonna be like, you know, the, the agent tool you buy that opens a hundred PRs that you never merge. Um, and, uh, I don't wanna do that. So,
Ryan:I'm at least at plus 50 right now on that plus minus from, from baseline. I mean,
Nate:Yeah
Ryan:really hard not to get ahead of myself, but it's, it's kind of addicting to let an agent like a- attempt a problem that you
Nate:what's
Ryan:wouldn't
Nate:what's so bad about this stuff is like it feels like work. Like, um, I, I've, I think the slot machine comparison is quite apt in that there's like-- I mean, just think about how the fact for like 20, like 30 years, like this image of the computer programmer is like the… Remember that website Hacker Typer that would like, you know, fill up your screen with like fake, um,
Ryan:The
Brendan:Yep
Nate:y- you-- Like, that's literally what we look at all day now, right? Is
Ryan:Yeah
Nate:oh, it feels like, oh, I'm looking at the matrix and it's like this, these, these tokens are flying by. Like it feels productive like in a way. And the, and of course, people have talked a lot about like the variable reward and like how that's more addicting than a consistent reward and like, yeah, like it used to be, I think-- And I think the, the form factor changes here, but like the, there's also like these fashions and like I've noticed that things that people produce with agents, which then they realize like a month later were just, um, I think the word people use is work slop. Like it's,
Brendan:Yeah
Nate:a, it's an artifact that looks like work, but it doesn't actually do anything useful. So like, um, I think the big one for a while was like dashboards. It was like, everyone's gotta, they gotta have a dashboard for this, a dashboard. And like their agents are all making dashboards and, uh, like situation monitoring kind of like stuff.
Ryan:It's impossible to maintain it. It's
Nate:that?
Ryan:impossible to maintain those things, right? You get so many tools that are built in isolation, and then all of a sudden your, your whole day job is like, make sure the dashboard still works. You forget about the product
Nate:and I think most of the things you build, you realize they're just not that useful. Like, they're
Ryan:Well, that too
Nate:they're just not that good, um, and you never use them. I Yeah, I, uh, I definitely want to-- I, I think that that's, that's, that's very like, um, have a hammer, everything looks like a nail, is like, uh, you know, if you have this report-generating machine, suddenly everything needs a report, and like who's got time to read all those reports? Like I don't, I don't need, I don't need more reports, like I need more results. And this is something I posted about, but like I, I'm really not interested in AI automation anymore, which does not close the loop on something. Like it needs to do the whole thing end to end. Like I have enough… The, the amount of-- The solution space for like AI 25% of the way towards a solution is like infinite. Like, you could, you could have AI nowadays do like the 25% p- first pass on like literally anything. So like I, I don't find that novel or interesting anymore. Like what I want is something that can work completely hands-off without human intervention, and I think people are still underestimating this, the potential size of that problem space. Like I keep going back to the Intercom example. Like I, I, I think that, uh, particularly'cause we're on, we're in the Rails context here now, so I can kinda use that. But like I think if you haven't looked into like what Intercom has published from, um, uh, Brian Scanlan and, uh, so he's-- So I'm just gonna dump here if you haven't watched all this stuff. went on the Rails Business Podcast and talked, not Rails Business, the other Rails Business. He went on the, the Ruby on Rails podcast, the, the one, the official one with,
Brendan:Yeah. Yep. Right
Nate:that's, that's 100% you need to listen.
Brendan:It's a great one. I listened to it as well. Yep
Nate:publish a bunch of stuff on their blog, um, about how their AI software factory works. Um, and I would just go, like you got, you gotta re- Like I don't think people realize that a company the size of Intercom is like hundreds of PRs per week which were never reviewed by a human being. They're a SOC 2 company. Like the, the, the size of the, the compliance surface here is huge, and they figured it out. And like we still have people saying like,"Oh, you can't ship P- PRs without human review because it's not SOC 2 compliant." It's like, well, some lawyers down at Intercom figured that one out. and I, like they, they, so as an example of like the things that Intercom are doing, like they're, they no longer look at their Dependabot PRs. Those are all automatically done. Um, those are all automatically merged. Um, any, I think they said 92% of the PRs opened, um Are, well, let me get that stat right. I think it's something like three-quarters of the PRs that they merge today are-- have no human review. Of those, 92% are 20 lines or less. So I, I just think people are really behind right now, uh, in our community in thinking in terms of like what is possible here and, and also, like what is the shape of that work? So what did I just say? The, the PRs that are being merged are tiny. Like what they're not doing is what I'm seeing everyone doing right now, which is like building 2,000, 4,000-line PR monstrosities, which are, you know, are the result of a 20-hour interactive Claude session that nobody can understand or review, right?
Brendan:Yeah
Nate:this automated change that, uh, does a 20-line output, and it does something which can be reviewed in a way that's automated by a, by an agent. And I, I just don't see anybody really trying to, to work that way. And
Ryan:dream though
Nate:so much of our work is like is this repeat-- like Dependabot I think is the easiest example. Like why haven't you automated your dependency update process yet? Like what, what, what genius level of human review are you bringing to the table every time you open up a PR that says update JSON from 5.8.1 to 5.8.2? Like why isn't that just merged immediately? And why isn't that, that process of, of why, why can't I just hit merge immediately on this? Like why isn't that process done by an agent? Like
Ryan:Because you need a system
Nate:a bunch of big brain stuff to those PRs
Ryan:Yeah, but it, like I, I think the challenge is, you know, for us small people, is that you have to have, you know, a test suite. You have to have, you know, ways to verify, right? And if you're, you haven't done any of that work to build the foundation, then it's very difficult to take that next step. I agree that that's the ideal, right? Like, I mean, and that's the opportunity is, you know, to be able to do these small, verifiable, non-breaking PRs, but the infrastructure you need underneath it to support it is, A, I think unknown to a lot of people, right? Like, it's, it's hard to wrap your head around what is needed because not a lot of people are talking about it,
Nate:Yeah
Ryan:openly. I mean, Intercom thankfully is putting it out, and that's a great, a great resource. but for some of us it's like,"Yeah, that's great, but I don't have a test suite." Well, that's, that's a big problem to solve to then get to the next level.
Nate:I'll give you two quick-- I'll give you a quick thing before, um, the… I have two indies in my retainer right now, and
Ryan:Mm-hmm.
Nate:one thing I told them both to do is to use AI to write, um, what we would call characterization tests.
Ryan:Mm-hmm.
Nate:characterization test just says, "Given these inputs, the outputs of the code is this." And I've told them to write these at a very high level. One of them is like, I literally even wrote it or prompted the agent to write it such that it's like the output is like a HT- HTTP response. Like, it's like a, a curl. We're doing like-- We're, we're basically testing the output of this s- con-- this app with curl and saying,"When I put these things in, these things come out, and I don't care, not testing how it comes, like how
Ryan:Looks.
Nate:Yes. And these, these tests are useful. It's a ver-- It's a classic legacy code refactoring technique, but it's very brittlable, okay? I don't actually care that they're brittle because I can just have the agent delete them and like, okay, I-- now I understand what this code does, so like I can replace the characterization test with a proper, you know, unit test, controller test, whatever. But the c- the, the characterization test, basically what I would say is like increase, um, encases my app in concrete and says, "Okay, any change in this app is now gonna be really brittle." But again, I don't really care'cause that, that, that's a lot easier to work around now. Um, but it also gives me the confidence to say like, "Hey, if I upgrade JSON, you know, a major version, I know that all the outputs are still exactly bit, byte for byte the same, because I have golden masters for every single JSON output on this website." So if you're in that situation, I would just encourage you to characterization testing, because I think-- So this is new to me, so I'm trying it. I'm not gonna say like I know for sure it works, but certainly the math around characterization testing has changed and, uh, I'd encourage people in situations like that to try it.
Ryan:Well, and, and I can back that up because we just went from Rails 7.2 to 8.1. Before I did that jump, you know, I had s- I'd been working over the years to improve my test coverage, but I also was at a point where I was like, all right, I still have a manual QA script that's like 40 pages long that covers the entire app that when I do a major upgrade like that, I like to run through it. Well, I just took that script and I basically told Claude to go write these characterization tests, and it did. I, it, it took me about, I don't know, probably a three-hour Claude session to, to get through it and get everything green, and I gave it a quick once-over on the GitHub review and then shipped it. But yeah, like it gives you a lot of confidence. So, you know, it, it works, right? It's a step. It's a stepping stone. Um, but yeah. All right. So, you know, I think we're, we're just about out of time. Do you wanna… I, I know you have a product that is coming out, uh, and, and by the time this hopefully rolls out, uh, in a couple weeks, do you wanna give a quick tease on that?
Nate:Yes. Um, if you too are tired of understanding what your app does, um, I have made something so that you can understand it less. Uh, S-speedshop.co is my website, and, um, by the time this goes live, I should have, um, Thread Pilot online. So Thread Pilot is automatic thread pool tuning for Sidekiq and Puma. It's something that I've wanted for 10 years. Um, I have constantly had people come to me and say like, "Well, how many threads should I be running with Puma and Sidekiq?" And unfortunately, the answer is very complicated, and the answer is very, um, dependent. So really the, the a- the a- the correct answer is different for every app, and the process for coming up with the answer is very complicated because it requires a deep understanding of the global VM lock. and what we did is basically took that whole process and packaged it up into a product which is now possible basically due to changes in the Ruby API, um, the Ruby C API that were shipped in Ruby 3.4. Uh, no, earlier than that, 3.3 actually. Um, so if you have Ruby 3.3, um, and up, uh, basically or, and use Puma or Sidekiq, um, we can automatically, uh, tune your thread pool to the correct size. And what that means is it's also doing this at runtime. It's not doing it once. This is also, even if you do have a good number for this today, the correct number can change over time. So in Sidekiq, if you have a bunch of jobs which are very high IO, you want a high thread count to be able to burn through those as quickly as possible. And then if you have a bunch of high CPU jobs coming right after that, you're gonna want a low thread count'cause otherwise each of those jobs is gonna be competing for the GVL, and they're gonna take a really long time. this, this trade-off between how long it takes to run the work and h-how much throughput you're gonna do with the work, um, is what this gem automatically sort of shoots down the middle w- for you for. and, uh, we've been working on it for over a year now, and, uh, it's now in production at several of our clients and, and basically depends on your workload, but, you know, getting the correct thread number can be, um, a 10 to 50 to 75% increase in how much throughput you can serve, uh, gi-given the same amount of, of resources. So,
Ryan:So it's a real-time configuration that changes the, the counts as stuff comes onto the queue?
Nate:Yep. Uh, or, uh, as stuff, as work is processed, um, requests in Puma. Um,
Ryan:Right.
Nate:yep
Ryan:Does that remove the… I, I feel like the way that I've solved that in the past is just I have dedicated queues for the type of work it is. Does that sort of eliminate that need?
Nate:well,
Ryan:could it? Yeah.
Nate:if you're wrong? What if somebody puts something else on that queue tomorrow?
Ryan:Well, yes.
Nate:you know, so
Ryan:If I'm wrong, it was me, because I'm a single developer. But yeah,
Brendan:or what if you leave it on the default queue by accident?
Nate:And, and queues, queues also are not free, right? Like every s- add-- queue you add in Sidekiq makes it more expensive to, uh, check and wait for all the work on the queues. So, um, like you can't have an infinite number of queues either
Brendan:Yeah, it's a good point
Ryan:Cool. Um, I'm looking forward to checking that out. I think that would be helpful for me, so
Brendan:Yeah, agreed. That sounds cool. Well, it's been great having you, Nate. Uh, is there anywhere else you wanna point people to before we jump off?
Nate:Uh, that's it. Speedshop.co. We got a newsletter. Um, it's got links to all my other stuff on there, so check it out
Brendan:Awesome.
Ryan:Thanks for joining us
Nate:All right. Thanks guys
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
Code with Jason
Jason Swett
Standing in the Fire
Very Good Software
Remote Ruby
Chris Oliver, Andrew Mason, David Hill
IndieRails
Jess Brown & Jeremy Smith
Startups For the Rest of Us
Rob Walling
Vibing with AI Code
Brendan Buckingham