Rails Business
Brendan Buckingham and Ryan Frisch talk about developing with Ruby on Rails and how to leverage it to build a business.
Rails Business
Ernesto Tagwerker
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Ernesto Tagwerker (FastRuby.io / OmbuLabs) returns to talk through how AI is reshaping both sides of his business — speeding up Rails upgrades while simultaneously eating into revenue, and opening a new lane in AI strategy consulting for non-technical businesses. The group digs into the messiness of AI-generated tests, mutation testing, and how to know when your tests are actually testing anything.
01:00 FastRuby's Open Source Claude Skill for Rails Upgrades
03:30 How AI Has Eroded (and Grown) FastRuby's Revenue
07:00 Engineering as Marketing — Why They Open Sourced the Playbook
10:00 The Tooling Isn't the Problem — Engineering Culture Is
12:30 Using Claude to Write Contract Tests and Improve Coverage
15:30 The Problem with AI-Generated Tests
17:00 Mutation Testing Explained
19:00 Classicist vs. Mockist — Where Claude Gets It Wrong
23:00 Playwright for Acceptance Testing in the Development Loop
26:00 Who Watches the Watcher?
28:00 Staff Augmentation and AI Strategy at OmbuLabs
31:00 Working with Non-Technical Businesses in Philadelphia
33:30 Engagement Sizes — AI Discovery vs. Rails Upgrade Projects
35:30 CRM Summaries and One-on-One Prep with AI
38:30 Observability and Guardrails — How Do You Know It's Working?
43:00 LLM as Judge
45:00 Event Scraping — AI Makes Scrapers More Maintainable
50:00 Ryan's Universal Scraper — Using AI to Heal, Not Replace
LINKS
- OmbuLabs
Hey, how's everyone doing today?
Brendan:Good. How are you?
Ernesto:Good
Ryan:Uh, we've got O- Ernesto back, um, from FastRuby and OmbuLabs. I think this is the, the second time you, you've joined us. Uh, so welcome back to the, to the pod
Ernesto:Yeah, I'm excited to be here. I really enjoyed the last episodes on AI and wanted to join the conversation and see what you've been up to
Ryan:Yeah, absolutely. Well, I'll, I'll, I'll lob you an easy one to start. You know, pre-call you said you wanted to talk about how AI is transforming business, and I think we're all thinking the same thing. So how is it currently transforming your business?
Ernesto:Yeah. Yeah, it's definitely transforming both of my businesses. And just for anyone who's, um, interested in FastRuby.io, we do a lot of, uh, tech debt remediation and Ruby on Rails upgrades, that's a product I service by OmbuLabs. And OmbuLabs is more about AI and basically helping small businesses figure out their AI strategy, and also, um, build custom AI solutions for them. So think chatbots on top of pr- proprietary data and that sort of stuff. So there's definitely been, like, an interesting between AI and Rails upgrades, and, uh, I'd love to talk about that.
Ryan:All right. Well, let's, let's talk about it. H- are, are you using this, uh, internally mostly? Um, you know, I think you have a, a skill that you've, uh, published. Am I correct on that?
Ernesto:Yeah. Yeah. So, you know, like everybody and their mother, uh, around December, we started using Claude Code a lot, and we saw a ton of potential there. So, uh, I started building an open source Claude skill to take you from Rails 2.3 all the way to Rails 8.1 following the FastRuby methodology. And usually that means we like to dual boot a Rails application with, you know, Rails 7.2 and Rails 8.0 and run the test suite with the two different versions. we found that Claude initially would be against dual, dual booting. And we thought it was kind of funny and also a little bit frustrating'cause we work with really big code bases, and this one client project has been around for more than 15 years, and of course it has a ton of code. So when Claude was like, "No, this, uh, adding dual booting would add too many if-elses to the code base," we were frustrated and we're like, "From experience, we know that adding the if-elses, uh, depending on the Rails version helps us debug weird edge cases, we really want you to do it." And at the time it was like, well, to tweak Claude and to make it do what you want, Claude skills were the way to go. So that's why we came w- up with this idea, and then it turned into three different Claude skills to do three different things for Rails upgrades, and we've been using it and it's been helping us speed up upgrades for a ton of our clients.
Ryan:How has that affected, you know, your business, right? Has it-- I, I know last time we chatted, uh, you know, I think we talked about, you know, how do you charge for it if it's saving you a ton of time, right? Has, has it eroded your revenue? Uh, how have you dealt with, with that? I, I-- or has it just been a complete non-issue?
Ernesto:No, it's definitely eroded part of our revenue because, um, you know, we charge, um, by, you know, using time and materials. So if we are using less humans or we need less humans to do the upgrade, then we are basically, uh, s- splitting our revenue with AI. So let's say back in the day, maybe last year, we would've needed two humans to do an upgrade. Now we can probably do a similar upgrade with one human and one, uh, agent working with that human. I think the part that is super important for this is like it has to be combined with experience. So we've seen, you know, real, real gains in productivity, but then at the same time, it means that we need to sell more projects because we have-- We're still, like, a pretty small company. We're about 14 in the company, and we can do of the work with robots, it's great for, for the client 'cause they don't have to pay that much per week. But for us, it means that we need to scale our marketing, our biz dev efforts to find more, uh, leads and more companies to work with and drive them to the top of the funnel. So definitely shifted and, and impacted our business in a weird negative way.
Brendan:And just to follow onto that too, like I, I'm just thinking about because you open sourced this or made skills essentially the same thing, um, you know, th- this allows others to either use it themselves obviously, or, or like other competitor, uh, you know, uh, clients or, or, uh, game things the right word, but like OmbuLabs competitors or FastRuby competitors, you know, other freelancers even. Like, are you able to even gauge how much of that is eating into your business too?
Ernesto:I don't have a good way to gauge that right now. Um, I don't even have a good way to say how many people are using the skills. Um, you know, I go to the Philadelphia Ruby meetup and I hear from people that used it, or I go to, to the Artificial Ruby meetup in New York and they tell me,"Oh, I used your skill and it worked." And it's awesome to, to hear that, but I don't have any sort of like telemetry in there to say, "Hey, report back to us," saying like, "Someone used this skill to upgrade from X to Y." Um, and it's also like a touchy subject, right? Like nobody wants anything that looks like spyware in their Claude skill, right? So definitely open to your ideas on, on how to track that sort of stuff. Um, but back to the original question, yes, we understand like open sourcing and sharing our playbook with the community can definitely like drive business away or it can basically empower people to use our skill to do the upgrade themselves. Um, but the way I see it is, um, if they're gonna use Claude to upgrade their Rails application, I would like them to use our skill and do it in a way that's more controlled and more organized and is based on real experience, than using Claude without that, the skill. Um, and it also goes back to our roots, you know, like we exist because there's open source. Like if there wasn't Rails, we wouldn't have a business and Rails is open source. And for many years we've thought about the best kind of marketing is to do engineering as marketing. So open source has been like key for us. Like anything we use, uh, we like to open source it as much as possible. And this one is like very core to our playbook so I understand like it b- can be seen as like cannibalizing our own market. But Claude is like good enough to do a lot of this. Uh, and a lot of the techniques that we share in our blog, um, I actually wish Claude would've used it more that I didn't have to come up with the skill. But any, any of these things that are inside the skill or like 90% of the things in the skill are things that we have already published over the years,
Brendan:Sure
Ernesto:like packaged in a nice way that it can be like plugged into Claude and, and used
Ryan:A-and
Brendan:Yeah, that makes sense
Ryan:as a customer of Ernesto's, right? Like, could I use Claude? Sure. But I'm paying him 'cause I don't wanna deal with it, right?
Brendan:Right
Ryan:uh, and I don't, I don't-- You know, just to push back on, uh, the, the whole idea a little bit is like, I, I don't think that really changed from pre-AI to post-AI. You know, maybe the tool's, uh, a little bit better and it the audience of people that could do it themselves. Um, but that doesn't really change the desire to do it. Um, and as we all know, there are, you know, as good as Claude is, they-- you still hit edge cases that it's like I'm, you know… I don't know about you and, and maybe have some experience with this in your skill, but the thing that I've found Claude, uh, least effective at, at the moment, and it may purely be that I'm horrible at prompting it, is, is monkey patching, right? When I have to monkey patch a gem to get it to work with the upgrade, it just, it almost never works for me. Um, you know, I have to dive into the code and figure it out. Uh, um, so, you know, maybe that's feedback, maybe you have an input. Um, but at the end of the day, I don't really think it's probably affecting your business all that much. And if anything, it's probably giving you, uh, the, the marketing bump, um, at least ideally, right?
Ernesto:Yeah. I think like one of our top articles right now is the one announcing the Claude skill to upgrade Rails. So, um, open sourcing and, you know, sharing our recipes definitely gives us stuff to talk about and sh-share in social media, and people find us that way. Um, and the, the last thing I wanted to add about that is that people don't upgrade because, not because they don't have the tooling or the tools to do it, like anybody can spend enough time and do the upgrade themselves. so now that we built a skill and you have Claude plus the skill, it's like, okay, you have better tooling to do it, and it can get you there faster. But then at the end of the day, it's more about, uh, your engineering culture. Is this something that you prioritize like every month? Like, do you invest like a few hours every month some of this technical debt? And if you can plug in Claude and have it give you like small pull requests here and there, um, that's great, but you still need humans re-reviewing the changes that make it to production, right? Or at least I hope everybody does that. I, I'm not… I don't think we're there yet, where it's like AI is gonna do the pull request, do the QA, ship it to production, and everything is be-- every-everything's gonna be kosher. Maybe we'll get there, but at this moment in time, I think like we still need humans making sure that things continue to work, um, because all the applications that we work with have good code coverage and good test suites, but they don't cover 100% of the, the code base. So there's definitely some human work that needs to happen, and that's, that's where we come in. We can the skill ourselves for our customer projects, and then we can help more people upgrade faster. But, um, yeah, it was, it was never about the tooling. I think it was more about like prioritizing this type of work, and a lot of our clients, um, are running like Rails 4.2 or Rails 3.2 in production, and it's because they, they haven't made it into, uh, a consistent and intentional change in the engineering culture to upgrade a little bit at a time gradually.
Brendan:Yeah, that makes sense. Uh, I guess I'm also curious, like has having the skill out there, and you, you, you know, I think maybe you kinda already mentioned it with the fact that the post about the skill was, I think you said your highest performing post or something. But have you seen people actually come to you that have tried the skill and maybe got partway, and then they're stuck for whatever reason, and Claude can't help them, and they actually come to you 'cause they were using your skill, so getting some benefit as well?
Ernesto:Yeah. Um, we haven't seen that yet, but we have seen some pull requests come where maybe there's an edge case where the skill is not good enough for them. the one that I can think of is a pull request that, um, makes this, the skill work for applications without a test suite. And that's one of the strong opinions we have in this skill is like it will use CI and, you know, if, if, if you don't have a test suite, then that work doesn't really make a ton of sense. So someone submitted a pull request to do a quick smoke test where it's just basically calling Rails runner, you know, puts whatever I'm working and, um, yeah, that is like a quick smoke test and it's like a quick variation that someone is trying to, to use the skill without a test suite. And I like that. I like that, you know, people are-- have used it and they found like, okay, this particular control or this p- particular condition that FastRuby added is not gonna work for me 'cause I don't have a test suite. And then you know, these teams might think about adding a test suite, uh, because it is kinda like a best practice. Um, but I don't know. This is something that we've recently launched and it hasn't really generated a lot of leads, it might have improved our, our brand awareness a little bit. And in-- there's still like a lot of people that know, don't know anything about FastRuby maybe they haven't tried to upgrade yet. Uh, but now if they're using Claude and they're searching for a skill to upgrade Rails, they'll find our skill and hopefully, hopefully they'll give it a try.
Brendan:Yeah, that makes sense. Um, I'm also kinda wondering, and we can kinda maybe start moving into other ways that you guys are using AI, but one question that just came up in my head as we were talking is to get someone-- like a client's test suite in a better position, are you able to… Are you finding that you're able to leverage Claude to effectively do that? Like, uh, is that something you're even doing? Um, like what do you think about that?
Ernesto:Yeah, no, that's a good question. Uh, we're not using it that much, um, but we have used it to write some contract tests for microservices, um, where you basically use Claude to feed it, uh, real production access logs. Um, ideally, what you wanna do if you do something like that is to filter any PII or any sort of like sensitive data from access logs, and then you can feed it into Claude, and then you can feed like the source code and maybe like the patterns that you've used, uh, when you write tests that it helps you write more contract tests for like a microservice. Um, but that's still something that I would love to explore more, uh, not just for contract testing for microservices, but also for like real, you know, QA teams that still have to like manually go and test certain things, because I think it could add a ton of value. With one caveat, though. I don't know about you guys, but every time I see tests written by Claude, they're probably like the worst code I've seen that's AI-generated. So I think, of course, you know, garbage in, garbage out, right? Maybe I'm feeding it the wrong prompts. Uh, maybe I'm not giving it, uh, explicit enough instructions. But I think there's definitely like a ton to explore on that field where it's like, if you could use, uh, staging or production data to write better tests, that would be awesome. And then at the same time, I think you can combine it with mutation testing to check whether the tests are good or not. Uh, so we, we have done that sort of work to help clients like determine whether a test suite is or bad or, you know, mu-mutation testing. I don't know if everybody knows what it means. Do you guys know what mutation testing is?
Ryan:That was the first time I've heard it
Brendan:I can't put a good definition, so go ahead and, like, enlighten us all
Ernesto:Okay, so I'm not a mutation testing expert, but this is the way I understand it, how it works. It basically changes part of your code, and then it runs the tests, right? So if you have a test that's testing a particular method, then it will go and it'll, like, change lines of code within that test, uh, within that method, and it won't change anything in the test code. And it'll run the test, and it'll say, "Oh, I changed this method, um, definition, and the test still passes." So then it's like, what? That's kinda weird, right? You have a test that's expecting something to happen with this particular method, but then you went and you removed, like, a few lines from the, the method, and it still works. So what can happen in that scenario is that maybe the test is stubbing or mocking too many things, creating this, like, crazy, unreal world, and then it calls the method in this mocked object, and then it's like, okay, cool. It's returning what I expected it to do. And, um, I think mutation testing can be combined with, like, AI-generated tests to kinda see, like, okay, this tests are maybe not that good. Like, yes, they are exercising every line in this particular method, but if I change the method, they, it still passes, so you need to, like, improve the test.
Ryan:All right.
Ernesto:Does that make sense?
Ryan:yeah, I
Brendan:makes sense. Yeah
Ryan:I mean, is that basically the red-green refactor loop, right? You have to see the test fail before it passes and then iterate, right? So like if you change it, then you have to see both sides of it, right? I mean, it's the same general idea or d- is there a fundamental concept that's different?
Ernesto:It, it is connected to that concept, um, but you can think about it as in, hey, you are changing a feature in your code base, right? And you're confident 'cause you have a test in that for that particular feature. So then you go and you change the behavior for that particular feature, and then you expect the test to fail. But then you run the test suite and the test still passes, and you're like, "Why does it still pass? I just changed the core behavior of this feature, but I can still see the test passing." Um, so it kinda addresses that particular problem.
Ryan:And then you, you said that, you know, the, the comment of you haven't seen Claude write, you know, good test code. Is that like because of You know, is it testing the wrong behavior? Is the code itself, like, just poorly constructed? Is it a, you know, side effect of too many mocks and stubs? A combination of all three? Uh, I was about to tell you, you, you, you know, we're, I'm going through the, what, 7.2 to 8.0 upgrade and, you know, we started at 4.2 with you, right? So we've gone a long way over the years. And when I first started, I had zero test coverage, right? Now I'm probably at about 50%. Now, I, I don't know, at least 20% of that is Claude, right? Um, I know where the gaps are, and I have a QA script that every time we go through an upgrade, like, here are all the places that I have to, know, click through manually. And I figured as far as Claude's come, right, that this next time I do it, I'm gonna try to write the smoke test with Claude, right? So maybe this is a good experiment, and we'll see how bad it is, right? Um,
Ernesto:Mm-hmm. Mm-hmm.
Ryan:so I throw that out there just as a thought for, for later. But back to the question, is it that is actually makes it bad, in your opinion?
Ernesto:Yeah, and it might be just a matter of taste. Um, you know, when you're doing TDD, uh, there are two like you're either a mockist or a classicist. Uh, so the classicist is usually the one that to build the objects in the way that they would be built, like using like real application usage. And then the mockist is more creating mock objects and stubbing methods and creating this world that just makes it faster to test like the behavior of a method. so what I've found is that the tests that I that Claude writes are heavily mocked and stubbed, and think that's where I, I get like a funny feeling of like, maybe this isn't such a great test. And I was just having a conversation with a friend recently, and he was like,"Have you ever thought about combinat- c- combining, uh, mutation testing with the tests that are generated by Claude?" I hadn't, but I thought that was a, a really good idea, so I, I can't wait to, to explore it
Ryan:Yeah, so
Brendan:is interesting. Sorry, real quick to just jump in. You know, it's like, uh, the inverse in some ways of testing your code anyways, but it's like there's always been in this, in my mind when you write the AI code, it's like how do you make sure that Claude or whatever you're u- whatever LLM you're using can know that it's done, right? And one answer is writing tests, but if we're now questioning the tests, which I've also seen too, by the way, like I didn't really get a chance to mention it, but, but yeah. Generally, it's kinda like, yeah, the, I don't know that I actually trust the tests. So now it feels like the mutation thing or m- mutation testing is kind of like we gotta test the tests now, which, which is cyclical though at least, so it's not like you're like going off and writing like a third thing that's now gotta be tested again too. But yeah, it's kind of an interesting take. I, I kinda like it 'cause where I see it going is, you know, like an agentic loop that could potentially do all three, and then it could be a closed loop that comes back to you ev- eventually and says, "Hey, I think I'm done. Now you confirm for me," kind of thing.
Ryan:Well, so, uh, I'll, I'll add onto that, right, is f-first, I, I had never really thought about this classist versus mockist universe, right? For me, I feel like I wanna be somewhere in the middle, right? Like from a… You know, writing the test, I wanna test the behavior, not the method, right? And in order to do that, in most cases, I have to mock or stub some stuff, right? But where that line is, that's the h- that's where the human experience, I think, has to come in to be like, "No, no, no, this test is like fundamentally, you know, uh, uh, built around that method. Like, you can't mock or stub that," right?"But this other thing is completely irrelevant, so you should," right? And I think that's probably for me I've seen Claude get tripped up, right? It goes too far one way or the other. Um, but with the, um… I forgot where I, my other points, but you know, you talked about like… Oh In my MD file, I always, you know, one of the rules is you have to, you know, you have to go through the red-green loop, right? Every time you write a test. So like, you have to write the test, then you have to intentionally break it, and it has to fail, and then you have to fix it, it has to pass again, right? And that's helped a little bit. Usually, when I go into a test I see a test that I don't like, it is almost always because they're te- you know, it, it became like a, "I'm testing this method," or, you know, the classic, "I'm testing Rails functionality," right? Like, I don't need you to test the validation, right, directly, right? That like, Rails has tests for that, right? I need to test the behavior that's, that results from it. Um, you know, so I feel like it's been a ongoing battle, but I, I think for me, it's getting better. And, you know, the last thing I wanna throw in there is I've experimented lately with telling Claude to use Playwright to test the interface as if you were a user, right? It's like you know, spin up, you know, the server, go visit the website, click the button, expect X to happen, right? So that's essentially a browser test, and it-- I wouldn't necessarily want that in my test suite because then the test suite time, you know, the, it takes too long to run. You know, I'm gonna do critical pass only, but as part of the development loop, it feels like that might be part of the equation, right? Where you can write these acceptance tests without having to write, you know, the mutation test to test the tests of concept that you just brought up. So any- just another thing to throw in the, in, in the pond.
Ernesto:Yeah. No, I, I love the idea of, like, who tests the tests or who watches the watcher, right?
Ryan:Yeah
Ernesto:Um, one, one other anti-pattern or weird tests that I've seen, like, Claude generate is, like, behavior or a test that basically tests the behavior of Rails, and it's like, well, that's kind of noise. Like, I- Rails already has tests. Like, you don't need to test that a callback gets called or that the middleware does something, you know? Um, so yeah, I mean, I, I could talk about this for hours, but I think, like, going back to the original question of, like, uh, writing tests with AI, I think there is definitely, like, a ton of potential there, and I need to explore more and do more research on better, better prompts and also, like, more context or, like, better examples. It's like use the existing of tests and, like, the existing patterns to write better tests for us, and maybe that's something I can, yeah
Ryan:I-- So i-in my app, I do like one thing that is like bucks the Rails convention, which is, uh, uh, and this was from Jason Swett, where he order-- you know, he puts the domain in the file path before the type of test, where usually it's like, you know, you do model and then test file name. I'll do something like, uh, let's say like event importer/models, event importer/ you know, controller, right? So like that way I can run it as a group, which I really like 'cause it's, it, it feels more aligned, but because it's not like the way that it's taught or done traditionally, like it-- I have to correct Claude pretty much every time I run it, even though it's in like every file. It's usually in the prompt and I still have to be like, "No, no, you put this in the wrong place." And it's so frustrating where it's like, "Oh yeah, you've told me that. It's in my memory. It's in the Claude file. It's in the pattern of the app. I should have just done that." I'm like, "Yes, you should have." Um, anyway, uh, you, you mentioned, you know, in, in the pre-call that you were sort of doing stuff, uh, you know, with staff augmentation and helping, uh, you know, businesses set up AI, uh, you know, processes, I guess. Um, I don't know, that wasn't the word you used, but everyone knows what I mean. And I think those are two separate things, but do you wanna a little bit about each of them and, and why, from a business perspective, you went into that, you know, those two areas?
Ernesto:Yeah. Yeah, sure. Um, so I think it's all connected to what I was saying earlier, where we need fewer humans to do upgrade work for our clients. So that means, like, we have a couple folks on our bench now. we were thinking, "Okay, how can we connect with more clients and, and more companies and do work that's beyond Rails upgrade work?" Um, and I kept having this conversation with some of our clients that were asking me whether we did feature work or whether we would focus on their product roadmap, and I was like, "Yeah, of course." Like, and I thought it was kind of funny that lot of the clients that hire us do an upgrade, then don't think of us as, oh, you could just be another Rails engineer adding value to the product roadmap. So I thought, "Okay, maybe we can just package this and announce it," because it seems like, uh, if I keep hearing this question, it means, like, it's not clear that we do more than upgrade work. So, um, yeah, that's, that's how it came to be. It was like, we need to find more work, and maybe the upgrade work is not hot right now." So we thought about, okay, what is, like, the uniques or the, the value proposition that we come up… can come up with to, um, reach more customers? then they will reach out, and they will work with us on product work, and they'll know that they're working with, like, an engineer by, you know, that's h- has been working at FastRuby for a while. So we do keep, like, things like tech debt in mind at all times, and we understand, like, the trade-offs of picking one library or another if they're using, like, an old version of Rails. Um, and in terms of AI, uh, that's kinda like the other side of our business. Like, we work with small businesses in the Philadelphia area that are non-technical. So they're not, like, software as a service. They're more like, you know, investment firms or they're running, know, uh- medical practices and stuff like that, and they know that they need to use AI, and then they just don't wanna start using AI without some expert coming in and saying like, "Hey, these are the type of s- type of tools that you should be using, and this is the type of infrastructure that you should have in place to make sure you're not leaking data into, uh, the wrong ChatGPT account," and stuff like that. but yeah, sometimes that AI strategy service that we offer turns into, "Hey, we can build like a custom AI solution for you." It can be like a chatbot, it can be like a prediction model, uh, it can be like some sort of like LLM, uh, solution that will help them be more efficient in one of like their more time-consuming processes. Um, yeah, that's, that's kinda the type of work that we do at OmbuLabs, and it's definitely like growing. Uh, so like I said, AI has been like positive and negative at the same time. It kinda like has decreased our business for the Ruby on Rails upgrade projects, but it has also increased our business on the AI strategy and custom AI solutions, um, part.
Brendan:Yeah, that's interesting 'cause we, we just had Colleen Schnettler on a- and Joe Masilotti, and Colleen is working mostly as an AI consultant too, and it sounds like similarly to what you're doing, um, where it's, like, a lot of automations and just, like, it's not Rails work. You know, it's… I- it can be anything really, which is interesting. I think there's a lot of opportunity in that space. So you mentioned that you're working with, like, local Philadelphia businesses, so that's all you're doing at the moment? It's just kinda, like, the local thing and… Or that's just what's worked? Or I, I would imagine you'd be open to, to working anywhere, but just kinda wanted to clarify that.
Ernesto:Uh, yeah, we're actually open to working with other companies, like in other cities. Like we had a-- we have a client in Boston as well. But I think in terms of, uh, SEO and, you know, GEO, somehow we started ranking very well for like Philadelphia AI consulting. I think we just started getting, uh, random requests from like small businesses in the Philadelphia area because I think that they're searching for like AI consulting in Philadelphia, and we show up at the top of the list. but yeah, we're, we're open to working with companies all over the US. Um, it just happened that keep getting like a stream of like companies from the Philly area.
Brendan:Yeah, that's interesting. How-- I mean, you don't maybe wanna answer this, but I'm just kinda curious, like, how those clients compare to, like, on, on an income, income basis. You know, are they-- Do you have to get a lot of them to kinda make up for what you would expect for, like, a Rails contract? Or is it, you know, are you able to do pretty well with, with the clients you're getting and the, the fees you're able to charge?
Ernesto:Yeah, I think it compares, um… Let's see. We do charge less or we, we don't charge less, like our rates are very similar between like AI and Rails upgrade work.
Brendan:Okay
Ernesto:but this, the engagements or the size of the engagements for the AI work are usually like smaller than a Rails upgrade.
Brendan:Sure
Ernesto:a Rails upgrade project, we're usually talking about, know, $80,000 to half a million dollars depending on the type of upgrade project we have. for AI discovery, we're talking more about like 12 to $20,000 to go in, do a discovery, talk to stakeholders, review the use cases and all that. Um, so we definitely need more AI discoveries to make up for a Rails upgrade project.
Brendan:Yeah. Got it
Ryan:What kind of-- Can you give us an example of AI project that you've worked on, right? Like, I, I think the thing that I struggle with is going from the abstract is like, oh, that you can automate sales, you can automate support, you can automate, you know, productivity, personal productivity, and like inbox management, right? Which are like, it all makes sense at a high level, but then I like lack the ability or, or maybe just the, uh, know, the time, uh, to, to understand the details and what that stack, uh, might look like, right? Like, how do you… If, uh, so I guess there's two questions. One is like, you know, what is a example process that you might implement for somebody? And then how do you go about sort of breaking it down into the pieces and stitching it all together, right? Um, and I-- obviously that second part depends on what you pick as your example for the first one.
Ernesto:Yeah, no, in terms of like automating some processes, like usually, um, we've done it for our own processes. So like we have experience like, uh, basically connecting the dots between like the, all the content that we publish. We're publishing about like two blogs, two blog posts a week per, uh, website. So at the same time, we do our own social media. So basically, there's like a Claude, uh, of sub-agents that basically goes to all of our content and it says,"Oh, okay, you just updated the, uh, Heroku Ruby compatibility table, so I'm gonna, you know, make a draft in Buffer for you to go and like publish it." So we definitely have a ton of like tools that do that. They're not pu- posting for us. Like we're very like, uh, conscious about having a human in the middle for anything that's public-facing. So it does kinda like set it up kinda like tee-ball, right? It kinda brings the ball, puts it on the tee, and then you just have to like swing your bat, and then it just goes out, um, through social media. Uh, but for some of our clients, usually that means, um, helping their staff be more efficient. Um, we believe, you know, we don't wanna work with companies that just wanna reduce staff. They ju- we don't wanna build something that's like gonna replace like their entire uh, team. We want to work with companies that are looking to like enhance and improve like their processes so that their current humans can do more work with, uh, less time, right? let's say for an investment firm, we'll go in and we'll look at their systems and, uh, CRM, right? for the next call they have with one of their clients, the, the LLM, what can do, what it can do is basically go through the CRM, go through the notes, and then give them like a quick summary of like, "Hey, last time you talked to Ryan, he was interested in investing more in clean energy, uh, funds," you know? so when the human gets on a call with the client, they can see that summary, and they don't have to do the legwork of going through the notes and being like, "Oh, right, this is what I was talking to them about." And it might be like it might save them 10 to 15 minutes, but if you multiply that for, by the number of clients that they have, um, it can be a ton of time, and it can make them look better, and they can have more calls with their clients to keep adding value with the same number of humans.
Ryan:I, I saw an example recently sort of in that same space where it would, you know, take a, you know, discovery call, for example, recording and essentially grade it based on your playbook and then, you know, turn that around to the sales rep and start the coaching. You know, what, what would normally be one-on-one coaching, give sort of that immediate coaching feedback of like, all right, know, you got a one to two, you know, uh, you know, one out of five of getting, you know, trying to get the next meeting scheduled before you get off a call and, you know, a four out of five in identifying the problem, know, a three out of five on XYZ. And then it goes back and it gives them that grade and it's like, all right, on your next call, you need to remember to do these two things. And that was really appealing to me because you can sort of codify your business process and what normally would've… You know, it doesn't replace the need for a one-on-one meeting, but it certainly speeds up the feedback loop, So that when you get to a one-on-one meeting, you're really honed in on like, all right, well, I-- we saw here's the feedback you got and you were able to improve it. are some more tips to get you to the next level That said, I don't know how to build that, right? It's like, all right, so we have Zoom. Um, you know, I'm sure I can figure out how to get the transcript and, you know, I have a playbook Right. I mean, to me it feels like the next step is just giving it to an LLM and say, "Take the transcript, compare it to the playbook and, you know, rank it on these five things." But I, I think I have a lack of trust in the LLM's ability to actually do that. Any opinion? you know, is that how you would do it? You know, do you, do you have those same or reservations? And if not, why?
Ernesto:Um, yeah, no, I have, I have my s- the same concerns that you have. Um, usually for something that's so, uh, private or, or may have like sensitive information, what I would probably try to do is like spin up some sort of like open source model and maybe not use like OpenAI or Anthropic, um, which is something we, we've done for some of our clients that wanna use some of these services because maybe they don't trust them or maybe because, uh, they just wanna host everything in their infrastructure. I haven't done it yet, but it is like a really good, um, use case for, for my one-on-ones. Like now I want that, you know?
Ryan:Right.
Ernesto:Now I want a robot that basically goes through all the things that my direct report did in the last month, and it kinda like bubbles up things that could be like interesting topics or questions to ask. Um, because yeah, I have one-on-ones every month, and I always feel like I'm playing catch up to come up with like good questions for my one-on-one.
Ryan:We
Brendan:Uh, and for what it's worth, I, I do that with my one-on-ones, but like I'm using Slack primarily, you know, and that, that's basically it. I mean, we don't do a lot of things through email. We don't have like calls that I'm recording. But even just like having the, our Slack interactions and being able to-- And even like get, I guess, pull requests and things like that. It's like you can get a kind of a simple version through that for, you know, basically by just saying, "Claude," you know, "This is what I would like to do," and it's just the MCP set up. But yeah, it, it, it took me a long time too, Ryan, to like really wrap my head around how do I piece all these things together? And a lot of it for me has just been like, I, I ask Claude, like, "How would you do it? How should we do it?" I mean, I think I mentioned it on the, the last time we, we recorded where like I was stuck trying to get like N8n things set up all the time 'cause I'm like,"Oh yeah, I gotta pull from this one and that one and put them all together." And it, it turned out it was just like Claude can do it because it's got MCP or it can even, it can even write the API calls, and I don't need another harness. Um, but yeah, just wanted to throw that in there 'cause that, that's something I've been doing as well.
Ryan:Yeah, I, I, I think maybe my hesitation is like, how do you know that it's working the way you want it to work, right? Like, what are the-- how do you set up the guardrails? Like, how do you put visibility and observability in place? You know, and th- and this goes beyond like internal processes. Like, I wanna, you know, build tools for our customers within our platform, right? But I gotta know, I gotta have some way of knowing, you know, of, of analyzing the result that they're getting to make sure that it's, you know, up to, you know, the quality standard that we want, and that it's not complete garbage, or that they're not abusing it in some other way that, you know, I don't want them to be abusing it in. and I feel like that is a, something that not a lot of people are talking… I mean, people are talking about, like, the need for it, but not the solution. Um, and maybe that's just because no one's figured it out yet in a that enough people got excited about. Um-
Brendan:to be clear on that part about like for your customers, are you thinking, you know, introducing a chatbot or s- that sort of an interface into your app as the quality thing? Or are you thinking about just like
Ryan:Well
Brendan:things that use AI, but maybe it's not a, a chat?
Ryan:Well, the, the feature, you know, think about the tools that Basecamp just put out, right? The, the notion that, you know, uh, a chatbot or AI LLM, right, should be able to interface with your app and do all the same things a human could do. Well, we're a marketing platform. We can do a lot of the normal marketing things that we've been talking about, uh, on the call, right? So like the vision is someday to get to Basecamp level of, you know, API, CLI, right? But,
Brendan:Yeah.
Ryan:you know, a, a, a concrete example, you know, I think we've talked about this in the past o- on the podcast is, you know, something to the effect of I have a, um, you know, in our space, local space, a lot of people do event flyers, right? You know, they build it on Canva, and that's not a great, way to put something online, right? You actually want it to be real data with text on the page so that, you know, uh, you know, Google and other AI at this point can pick it up as SEO, right? So like the example would be, well, let's drop in, have them upload the image, extract the data, and enter it into the calendar, right? Huge time saver for the people that need to do it. But how do you make sure that that extraction is doing the right thing and that it meets the, you know, the required fields and that it's mapping the right way, right? Like that, that's probably the simplest example I can come up with. Obviously, there's much more complex ones that I'm thinking about. But like if I can't solve the simple one, or at least the-- You know, I, I-- The best I've been able to come up with thus far is that, um, you know, obviously there's hu- there's human in the loop, uh, you know, e- especially at the beginning, and then there's sort of like regression testing, right? So like once you think it's working, then you measure if there's sort of a measurable change, and if there is, then you come back and look at it, right? But still sort of feels like spam, uh, management, right? Which is always a pain when you-- once you reach a certain size. And I haven't solved, you know… That, that's gotten a little bit better with AI, but not much, right? AI can break it just as much as it can help you. Um-
Ernesto:And you, you could have like some sort of like LLM as judge solution too. Like, uh, like you said, you could use AI to like extract data. and then sure, you could have a human in the loop for certain things, but you could also have another, a separate LLM judging whether the data that it extracted is accurate or not. but yeah, it's, it's definitely tricky. Um, the other part that you could do is, uh, have some evals to check whether model is extracting the right type of data. But, um, when you-- when I think about evals, I think about like, sort of like tests for code. Um, but yeah, it, it is tricky and we, have a couple of like custom AI solutions we've built for clients that struggle with a very similar, uh, problem. Like we basically built like an AI solution to do scraping for this par- particular client. And mean, we run into like the usual problems of like scraping, you know, bot control and, you know, rate limiting. That sort of stuff is like really hard unless we talk to like the particular partners this customer is, is trying to scrape. and they have like a legit business model. They basically are a tourist website for the city of Philadelphia, so they wanna have like a list of a ton of events, right?
Ryan:Mm-hmm.
Ernesto:but yeah, we, we also, uh, run into issues related to like the quality of the extraction of the event data. you would think that websites and, and applications for, uh, venues would be like very friendly to bots and like Google and all that, but there are still a lot of like websites that just publish like events in an image.
Ryan:Mm-hmm
Ernesto:And is really, it, it, well, it's not hard to extract data, but it is, uh, a problem that we face where it's like the thing that it does, it should be like checked and we should make sure like it doesn't start drifting, uh, because it started com-
Ryan:the same tool.
Ernesto:Yeah.
Ryan:Um, which is
Ernesto:We should talk about this as offline, you know.
Ryan:Um, the, yeah, I mean, it's-- So like here, here's the thing, right? And again, this is a topic that's come up on several episodes. It's like, where, you know, uh, especially with like event scraping, like where do you actually need AI?
Ernesto:Mm-hmm.
Ryan:Like, it, it-- I, I think there's
Ernesto:Well, actually, I think the, the case for scraping with AI is super interesting because, um, you don't really care about the HTML structure anymore. You know, we all built scrapers in the past, you know, find this ID and get this data from that thing. You don't need to do that anymore. It's just like, give me the text from this page, and then basically use a model to go through that te- uh, content and say like, "Okay, these are the events that I found using that particular text content." But I think it just makes the scraper more maintainable because you don't really care that much if the HTML structure changed as long as they keep the, the text content in there
Ryan:All right. So, so h-here's an interesting-- This is how I'm approaching it, And, and this is largely because I hit every event that I wanna import into our system, and I fed it through L- like I, I'd run out of money pretty quickly, right? We're, we're processing hundreds of thousands of events, right? And I can't afford to AI extract every one of them, right? Not at the prices that we're charging. You know, maybe if our business model were different, that's fine, right? So this is-- We set up, and actually this is a project that, uh, Brendan and I have been working on together. Um, but we're, we're building a, a universal scraper, is what I'm calling it, and you give it the parameters of the, you know, classes or IDs for-- that it needs to scrape the sites. Like traditional scraper, nothing crazy. It's just dynamically built. And then we do, you know, we're gonna, once that is pulling the content and it's sort of human verified and working, then like I said, as long as that contract is still working until someone changes the HTML, right, it's fine. And then once it does, it reports back and says, "Oh, there's something wrong." Like something changed. It's not importing anymore. There was an error. Then I feed it to AI and say, "Find," you know, "Find me the new classes." Right? So I u- I use-- I'm using the AI to heal what would otherwise be just a normal scraper, and that allows me to use AI when I need it, but not run up the cost so much that we go out of business. Um, anyway, I don't know if that… excited about it because I think it's cool.
Ernesto:Yeah
Ryan:But, know, if I could just give everything to AI, that would be easier, It's
Ernesto:Yeah, and we're, we're working on an article for the OmbuLabs blog that is all about this. So, um, yeah, I'll, I'll share it with you as soon as it's live. Um, I am not the head of AI at Ombu, so I would love to connect you to my head of AI so you can go even further into detail on, yeah, scraping with AI 'cause, uh, I know only, like, the high-level solution that we built, but I don't-- I can't really tell you exactly, like, the, the little details here and there
Ryan:Yeah, and, and frankly, I was just sharing'cause I'm excited about the idea. Wasn't really, uh, know, expecting to dive into it at all. But, um, all right, what, what, uh… Is there anything else you wanna share with us before we, we start to wrap up?
Ernesto:Uh, no, I think, uh, it's been a pleasure as always. I feel like this hour just went by so quickly, so thank you so much for, for your questions and, uh, yeah, I would love to, um, about anything related to like AI or tech debt. So if anybody has any questions after this episode, they're-- you can find me on LinkedIn or Bluesky, and I'd be happy to, to answer any questions
Ryan:We appreciate you coming back on. It's always a pleasure to, to chat, and I feel like we could do this for another few hours, but it's Friday at 5 o'clock. I think we're all, we're all toast for, for this session, and we'll, we'll do it again soon
Brendan:Yep. Thanks, Ernesto
Ernesto:Thank you
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
Code with Jason
Jason Swett
Standing in the Fire
Very Good Software
Remote Ruby
Chris Oliver, Andrew Mason, David Hill
IndieRails
Jess Brown & Jeremy Smith
Startups For the Rest of Us
Rob Walling
Vibing with AI Code
Brendan Buckingham