Ryan: 0:37

All right. Welcome back to the Rails AI Podcast with Ryan and Brendan. Um, we're going to… Oh, wait, no, this is Rails Business. Just kidding. Um, say that a little bit facetiously'cause we're gonna talk about some more AI stuff today, and, uh, I think we're gonna start with skills. So what do you got?

Brendan: 0:56

Yeah, I've been hearing a lot and reading a lot over the months here about how Anthropic's skills, and I mean, that's kinda like… I- if you watch a lot of the YouTube videos that they've got or read articles about them specifically, and, I, I imagine OpenAI's the same way. I, I don't know. But, they talk a lot about, like, everything they do is based on skills. you know, there's platforms or, or, like, plug-ins or whatever you wanna call it, that have just groups of skills, like superpowers that people use all the time. Um, tons more out there. And actually, after we recorded with Jesse last week, you know, he was talking about Cursor, I just asked some more questions about that, and he basically sent me something that's a plug-in with a bunch of skills for Cursor, too. Probably could be used anywhere, but it, it, kinda like a theme where… A- and even I've been talking to other people as well, like, that they are writing their own skills. And so I, I just wanna do maybe a check-in and see, like, what each of us are doing with skills, if we're finding them useful, if we even care. I started having one for code review specifically. And that one I did tailor to, you know, Ruby and Rails, and the things that my application does, because I wanted to make sure it was looking for specific things that, uh, you know, just like c- common issues we would see, like migrations with things missing a- and stuff like that that's Rail specific. Um, you know, that- that's a re- review thing. And so the coding side, I- I, I guess I've started with, like, big skills, right? Where it's just, like, all-encompassing, even that code review thing. The, the thing that Anthropic at least points out in a lot of the things and the, the stuff that I've seen with them you want your skills to be more, uh, y- you know, incremental, like small, that you can plug and play. So you could have, say, a code review skill that does, like, one part of the code review, you could have a, uh, TDD skill maybe that is broken down into multiple skills. Uh, so more modular basically. Um, I haven't quite gotten there myself, but I, I like the concept because, know, their, their argument is partially just if you write one skill that's like basically like a huge script, right? And it goes through and it does stuff, and then you wanna modify it, or you wanna take like part of that and use it for something else, you know, it's just like object-oriented code.

Ryan: 3:29

I was just gonna say that that sounds very familiar, doesn't it?

Brendan: 3:32

Yeah, right. Exactly. But for whatever reason, like that's not where my head went when I started writing skills,

Ryan: 3:38

Well, let's dive into that, right?'Cause I'm having the same feeling as you're talking, right? But it's like You know, it's, it's how do you give the LLM the resources I'll say it needs, right, to do a task. Um And, and I think the thing that is hard for me to wrap my head around is simply this idea that you make skills available, but you have to trust that they will choose to use it. Right? And that, that's the part that I always get tripped up on. But, you know, if I went back, it's like, all right, when we first started, it was like, all right, let's have a Claude or agent's MD file, and it has like all my rules for my project

Brendan: 4:30

Yeah

Ryan: 4:31

Well, that is okay if it's a small file, right? But as, uh, you know, I think we all experience as that file gets larger, you know, the LLM just chooses what it wants to use or not use, which is sort of the same problem. You know, so then we were like,"All right, well, now you need like agents for each piece of it," right? You need a har- you know, a harness to orchestrate all the different sub-agents,

Brendan: 4:57

Yep

Ryan: 4:57

right? That, that was sort of the next step. And then it was like, all right, well, maybe that was too much. Maybe we just need skills and like one main agent that can choose different models based on the, the, the skill that it's using, right? And all that to say is like, I feel like that at a very high level has been sort of the iteration we've gone through recently. And all that to say is like, I still don't know if we're done, right? Is skills like, you know, the, the thing that we settle on? I mean, it sounds appealing. I like the idea of it, right? But I'm not convinced we're done iterating and figuring this out. You know, I think the interesting thing about skills is it opens up, it's like, all right, if, if you wanna build a skill and, you know, tune it, I guess, and like sell it as a package or open source it, you know, either way, um, you know, that's appealing to people, right?'Cause it, it's, it's the, it's the ecosystem we've always known and under a different name, right? It's a gem. It's a, a, you

Brendan: 5:59

Yeah, right.

Ryan: 6:01

right? It's a package. It's whatever you wanna call it. Um- The hardest thing for me is just I don't quite grasp where the lines are yet, right? Like you said, like, okay, a TDE skill, a design skill, a code review skill. Those all sound appealing, and I want to have, like, all of those things available, right? But I still don't know how to enforce it, so I'm, I'm sort of waiting for someone else to figure it out. Um, thoughts on that?

Brendan: 6:35

Yeah, I, I, I think like Superpowers for one, it, you know, it, it's set up in a way that? is basically like an orchestrator you know, you, you kinda run your whole process through it, if I remember right. It's been a while since I've actually used that one. Um, there was another one too, I'm forgetting the name. But the point being is that from my experience at least, and some, you know, listeners might have different experiences, and maybe I'm not quite on base, but what I've found is it's almost like you kick off one skill starts the process of, say, building a feature, and because you started with that skill, it, it, you know, has all the information to then do, okay, I'm building a feature or I'm doing this or that. And so once we get to the… It'll ask you questions, and once you get to a step of, okay, we've planned it, let's just say, go ahead and build it, then it's essentially invoking skill to do all that, and that skill then knows to trigger agents or other skills or whatever it needs. So it is much more orchestrated than probably the skills that you and I would make out of the box, where we're not necessarily baking into the skill specifically. You know, it, it-- Let's just say I created my features skill, which I don't have, I probably wouldn't have thought about, oh, yeah, and then you tell, know, or you, you tell the agents to use the next skill and the next skill. That's not how I was building things in the past anyways, and that's my understanding of how at least some of them work

Ryan: 8:11

So how did, like, if you were to sit down and build this system then, right, would you have to explicitly say, you know, "Here are the skills that this, you know, top-level skill might need"? And if that's the case, like, isn't that, you know, it's, it's, that's like the opposite of OOP, right? Um, you know, it's procedural to some extent, and now you're designing this complex skill system, right? At that point, what's the value of splitting the skills up if they can't be composable?

Brendan: 8:48

Well, it's reusability though, 'cause, like, if I wanted to… I'm trying to think of a good example. So, like, TDD's not the best example because obviously you're- that's a pattern and you want a bunch of things that go into it. But, l- let's make it simple. Let's just say, like, blog post writing or something like that, I don't know. Where maybe one skill is, like, doing research, and one skill is actually writing, and one skill is, like, editing, stuff like that. maybe you want that research skill to be more universal and you don't wanna just use it for writing a blog post. You might wanna use it for re- researching a new feature, something like that. Um, maybe not the best example, but I guess that's the way I, I think about that modularity is that… And they had examples in one of the videos I watched, it's been, like, a couple months now, on the, the Anthropic stuff, and their, their example was better about, like, why you'd want the reusability. So I will link to that so that we can get a little more context for people. But at a high level, that's kind of where head is at with, with why you would want the reusability, and it's not just procedural, and it's not just one, big script or big skill that does everything, and you don't need the, to be able to do it or, or pull pieces out if needed

Ryan: 10:04

All right. So then what's like, you know, we talked to, um, was it Jesse last week, um, about Grok, right? And Grok's approach, as far as I can tell without having, uh, gone too deep yet on it, is like agent-based, right? But agent as in like a agent for a specific role versus a agent for a specific skill, right? So like where do you think that, you know, uh, comes into play, right? Is there still this need for, you know, agents that have a role, that have a, a set of skills that they can use? Or is like, does that break down for you?

Brendan: 10:49

That's a good question. Um, I mean, I, I think they're, they're not, you know, mutually exclusive at all. Like, you, you would probably have… And again, I haven't used Grok Bot, and Grok Bot is very similar to OpenClaw or Hermes, and I haven't used any of those either. But just they're all AI, they're all LLMs, and at the end of the day, I would guess that probably using skills themselves anyway, so you probably have your marketing agent, your support agent that, you know, Jesse was talking about, things like that. I would imagine that tho- you know, you tailor those in a way that they probably have skills or some, some sort of tuning that makes sure those agents know what to do. and I've experienced this a little bit with my code review stuff. So th- this was something new to me recently, where I, I created a skill, like I already mentioned, and I've actually got multiple skills, but they're not… Th- these multiple skills are not modular in the way we've been talking about. Like, one of them is the initial review, one of them is, okay, I wanna re-review, and it looks at things differently. But the, the thing I wanted to bring up is that my, um, my skill is able to take in multiple pull requests at a time. I didn't realize until recently is that if I do that, it spins up other agents, which I knew. But those… Th- there was, like, a disconnect where the agent that, it gets spun up is basically being told by the orchestrator agent to do, but not using the skill itself. And that might just be something with the design of mine or, like, how I, I let Claude build this versus what GroqBot or those, some, some of those other things are using. I would imagine those are more rigid about it.'Cause it came back at one point, and this has been long enough I don't remember the specific example, but there was something about one of the reviews that happened when I did multiple came back with a question that was like,"Well, you usually catch, catch this. Why didn't you catch it?" And then it was, came back and it's like, oh yeah, well that's because I only gave it, like, the information it needed. I didn't give it everything that the skill had. Which was, like, a weird why would we do it that way if we've got a skill and I want you using it kind of realization.

Ryan: 13:10

Yeah. I mean, I, it, uh, you know, I, I would assume, and I don't know this for sure, right? But the agent, I think with Grok Bot has a persistent memory, right? And always has a, uh, you know, quote-unquote,"computer" to, to boot, right? But what it does is probably not so different than just kicking off the, you know, the first skill that then calls the other skills, I would imagine. Um,

Brendan: 13:39

think, yeah

Ryan: 13:40

it's, it's been-- I've had that browser opened since, uh, to a tab with all that info that we talked about last week since the call, and I still haven't gotten to it. But I'm hoping one of these nights I will actually get the opportunity to spin one up and see if I can make something fun happen

Brendan: 13:57

Yep

Ryan: 13:59

Um, all right. Anything else on skills that you wanted to touch on?

Brendan: 14:05

Uh, not me, but I guess was there any-- Are you using them much at all? I guess that'd be my only other question

Ryan: 14:11

No, but not because I don't want to, just 'cause I haven't had time to sit down and write any. Um

Brendan: 14:18

Sure

Ryan: 14:21

Yeah, this is like I, I'm torn here, right? Like, on, on one hand I see the potential for the business. Um, you know, on the other hand, I've got, you know, a list of competing priorities that are als-also important, right? And finding the time to, you know, at this point experiment with LLMs and agents and skills i-is difficult, uh, to find. Um, but, and we've talked about this in the past, it's like, uh, if you don't adopt it, you're gonna get left behind, is my opinion in general. I, I don't know what the timeframe is, but, you know, the, the sooner you start, probably the better, you know, w-with the, the one caveat is it's changing so fast and people are changing their opinions and thoughts and how it all works so quickly that you probably waste a lot too. So I'm sort of in the, it's on my list, I wanna do it. In the meantime, if other people help me figure it out, um, I'm happy to consume their content, uh, in the short term and, you know, continue to formulate a thought that might work for us. But it's-- The biggest thing is, is it's not necessarily the skills, right? Like I, I feel like you can put a skill together pretty quickly.

Brendan: 15:40

Yeah

Ryan: 15:41

Um, it's the systems around it that need to be in place in order to make the skill effective, right? So, you know, I think everyone seems to downplay that as like, "Oh, just, you know, pick a support center with an MCP." Well, okay, great, right? But we have, you know, 15 years' worth of data in the one that we have. Like, if they don't have an MCP, I'm not switching right now, right?

Brendan: 16:07

exactly

Ryan: 16:08

a-and even if they did, right, that's only one piece of it. Like, I need to connect into our app to get, you know, product usage data to really make it, you know, fully, uh, useful, I think, in that example, and we don't have that yet. So I need to build the… You know, not only do I need to build the API and then potentially an MCP or CLI or something to connect the dots, but then I have to actually design the API system. I have to decide what the endpoints are that are, are useful and, you know, those are all big projects on their own, right? Certainly accelerated by using AI to do it, but it's not like I can whip it together in a couple hours, right, on a production app that is used by, you know, tens of thousands of businesses. I can't just throw it out there without, you know, regard for the consequences. Um,

Brendan: 17:02

exactly

Ryan: 17:03

So it's like, all right… A-and, you know, on top of it, I have a, I'll call it a healthy distrust of AI. Not that I think it's going to, you know, uh, uh, be evil by any means, but like if, if, if I haven't thought through the guards then I'm not comfortable putting it out there, right? And something like support tickets, I'm not quite at the point where I'm like, "All right, I'm ready for it to just answer everything," right? Without, you know, oversight. And until I have systems in place to give me that oversight, I'm less comfortable with the idea. Um, but it's like, uh, you know, there's this also this other notion of like, it has to be good responses too, and that, you know, in, in those types of examples, whether using it for sales or support, anything customer-facing or, you know, prospect-facing. If I wouldn't send it myself, I don't want AI sending it,

Brendan: 17:59

right?

Ryan: 17:59

So, you know, again, guards, you know, figuring out how it all works. And, and I will say we are, we are investing in that part of it, right? So we, we have one AI feature that's under active development, and, you know, the thing that we were, we were talking about this week was like, all right, how do you evaluate AI responses, right? So now we're thinking about, right, what's this-- what, what does that system have to look like in order to have confidence, right? So this is like a, this-- we, we've talked about this in the past, right? The universal scraper for calendars. Um, you know, it's a lower risk area, right?'Cause you can have human oversight, uh, as part of the process in the short term. Um, but, you know, from the ground up, we're thinking about, right, how do we make this automatic? How do we evaluate every response

Brendan: 18:48

Yep

Ryan: 18:49

Um, so yeah, we're working on that, thinking through that. Um, but I, I wanna do some more of this stuff I even thought about today, I've been battling spam since about 3 o'clock this morning, which

Brendan: 19:04

Oh, geez

Ryan: 19:05

is annoying. You know, we, we've over the years developed, uh, quite a complex, uh, I guess it's an algorithm at this point, to identify potential spam accounts. And I'm running out of ideas, right? Like, you know, I've-- LLMs have obviously been a huge help in, you know, expanding that system. It's great at if I can give it a chunk of data, it usually can find like,"Oh, here's a gap, here's a gap." But it, it feels never-ending, right? So like there's stuff that we can do on our end, and there's stuff that we can do, you know, at the, at the edge with Cloudflare and u-using their network and expertise to block things. And to the best of my knowledge, we've done everything that I can think of, and we still had, you know, 2,000 new spam accounts this morning that were all just like, you know, any drug name by now online, right? Like every single one of them.

Brendan: 20:03

Yep

Ryan: 20:04

and in this particular case, the-- You know what, what, what tends to happen is like, it's like, all right, the algorithm blocks a certain pattern, and then the spammers figure out what the pattern is, and they find a new one, right? So there's this constant chase of like, all right, I blocked it, it worked for a while. They figured it out. Now I gotta block that one, and it's just never-ending. So I'm like, "All right, what else can I do here?" And of course, AI is like the first thing that pops into my head. Um, do, do you have to deal with a lot of spam or not really something you have to worry about too much?

Brendan: 20:37

I mean, we definitely get it, yeah. We, we've got some systems in place that work pretty well, like if you create a new link we only allow certain links that we know about to even get through, so like Amazon, Facebook, you know,

Ryan: 20:49

Right

Brendan: 20:49

big ones. We, we don't even let, like, link shorteners come through because people just use those for spam and stuff a lot. So we definitely get some, but I mean, it's not, we're not getting like 2,000 new accounts and stuff like that. And as far as the accounts go, I guess like we don't… M- maybe we get more than I realize, but it doesn't cause a problem like it would for you, so

Ryan: 21:12

Well, let, let me-- I'll, I'll expand on what I said before, right? It's like, you know, so we have a marketing platform that has, you know, websites and profile information for businesses. So I don't actually care if people create spam user accounts, right? It's annoying, it, it bloats the database, but it doesn't actually cause any harm.

Brendan: 21:33

Yep

Ryan: 21:33

then they go and create a business, right? Now that business gets a profile, it shows up on websites. That's not great, especially if it's drug and paraphernalia related. Um, and then of course there's like, You know, interestingly, like someone will add like, you know, a branch of Bank of America, right? And that gets flagged by Amazon because it's like, "Oh, you're spoofing, you know, Bank of America. This is a phishing scam." We're like, "Well, no, it's just their address on a map." I said, "Don't know what to tell you, but fix your system." Right? So like obviously everyone has the same problem. It's hard to get it right. But I am certainly tired of, of chasing it, because the chase requires programming and, you know, making updates to the way it's done. And I was like, "All right. Well, could an LLM be better if I just said, 'Hey, here's the payload. Give me a yes, no and a confidence level of is this spam?'" Right? But don't give it any further instruction. And it's like, all right, that's intriguing if it would work, but like at what cost, right? Let's bring it back to Rails Business, right?

Brendan: 22:45

Yep

Ryan: 22:45

how much would that cost me every day if we're getting, you know, thousands of checks? Um, maybe nothing. Uh, I don't know. Um, I'd have to quantify that before I decide whether it's worthwhile. But

Brendan: 23:01

Yeah

Ryan: 23:02

I don't know. Do you have any other thoughts on how to fight spam?

Brendan: 23:05

Yeah, I don't think about it a whole lot. Um

Ryan: 23:08

Yeah, I'll put you on the spot then, yeah.

Brendan: 23:10

Yeah.

Ryan: 23:11

Or, or we can, or we can move on.

Brendan: 23:14

Fair enough.

Ryan: 23:14

Yeah.

Brendan: 23:15

It, it does feel like something where AI probably should be able to do something, but I don't know what that is, I guess, specifically. Like, yes, it can catch patterns, you know, it can pattern match way better than we can, for one.

Ryan: 23:29

Right

Brendan: 23:30

You've already talked about that. Um, what does that mean? Like, h- how… It, it's almost like you need a… This is similar to other things that AI can do, right? Where it can, even review code or something. Like, what is the confidence level in that, though?

Ryan: 23:45

Yeah

Brendan: 23:46

much do you trust putting that code out it's just AI writing it and then maybe reviewing it? Like, is that enough, or… You know, I, I actually was reading the, the, um, blog post from Intercom that Nate Berkopec mentioned a couple episodes ago.'Cause I, I had listened to the episode, but I hadn't actually read the blog post, and the reason I'm bringing it up is because they talk about how in-depth their code review is with AI, and that it's basically, you know, back to skills. I don't know if it's specifically skills or not, but it sounds like they've got at least, maybe it's agents, I suppose, but like, you know, eight or nine different types of agents that are reviewing all the different things

Ryan: 24:30

Угу

Brendan: 24:32

would never be able to do, and that was their point.

Ryan: 24:34

Right

Brendan: 24:35

And I guess that's kinda where my head goes with the spam stuff is, like, could you break it down maybe into different agents so it's not just one AI that

Ryan: 24:46

Right

Brendan: 24:46

at it and pattern matches? Like, do you have one that's an expert at, I don't, I don't even know what all the pieces would be, but, like, different parts of spam to at least increase the confidence. And maybe there's, like, a, um, agent, which is also another thing that people use,

Ryan: 25:03

Yeah

Brendan: 25:03

it's like, "Okay, great, these guys have analyzed it." The judge comes in and says yes or no or whatever. I don't, I don't know. That, that's, I guess, where my head goes,

Ryan: 25:11

Yeah. I mean, there's, there's obviously different levels of spam, right? There's the obvious stuff, right? Uh, we don't have too much trouble with the obvious stuff unless I, you know, there's a programming error of some sort. Um, it's the, it's the middle area that's the most difficult, right? So you get a lot of like, uh, you know, roo- roofing, plumbing, lawyer-like spam that's like, you know, find legal experts in Florida, in Mass, in Maine, in Illinois, in California, and clearly it's spam,

Brendan: 25:48

Yep

Ryan: 25:49

right? But it, it's actually hard to classify that, and I'm obviously using very, uh, simple ideas, but it gets more complex where, uh, even as a human, I look at it and I'm like, "Well, that could be a legit business or it could be garbage," right? Like, and, and like you have to research it. Um, and that's like the level I'm at now is like I've got, you know, I've got a volume problem. I've got like 60,000 plus that fall into that category that need human review, and that human review is slow. Um, right. So then it comes back to, you know, business practice, right? It's like, all right You know, what is the, what is the risk of having those show up? All right, well, it's not ideal, but it's not drug-related, right? So it's not as critical as the, you know, by, you know, whatever drug it is.

Brendan: 26:46

Yep

Ryan: 26:47

but it's not great. It's not a great data experience for, you know, those that pull in information in that, you know, zip code. Um, yeah, so I, I don't know. I have to-- I'm, I'm searching for, for answers, and LLMs are giving me a lot of suggestions, but I'm vetting them at this point, uh, more than anything else. I'm just, I'm tired of it. I'm tired of dealing with it. I don't know why people do this

Brendan: 27:15

money I guess. But um, I, I, one thought that I had, do you, and this doesn't necessarily block it, 'cause people do it to us too, but like, do you require a credit card, things like that, that at least make it slightly harder? And, and what I'm thinking about there is like, could you like pre-auth the credit card if you require it? And, you know, even just like if it fails a pre-auth, maybe that's a layer of, you know, not trusting an LLM, but trusting Stripe to say, Yeah this doesn't look legit." Like, "Let's flag it right here."

Ryan: 27:51

Yeah, I mean, obviously that would be ideal if we had that kind of workflow, but we have a free forever plan. Like we're a freemium model, so like pre-authing a credit card is like in direct competition with that whole point.

Brendan: 28:04

Yep

Ryan: 28:06

you know, so right now, if you create a business You automatically get a profile on localable.com, right? Which I think is like why we're getting targeted with so much spam, right?'Cause they're trying to build backlinks or, you know, whatever they're trying to do there We have decided, partially because of spam and partially because of, you know, business reasons, that we're gonna get rid of those profiles. So I'm hoping that that will s- at least deter people from putting in any effort, right?'Cause it, at that point, it does nothing for them.

Brendan: 28:42

Yeah,

Ryan: 28:43

Right.

Brendan: 28:43

think that would at least knock down the amount you get for sure

Ryan: 28:47

yeah, I mean, and, and since this is a Rail- Rails business podcast, right, the business side of the equation is, you know, our software is meant to help communities build a resource for their community, and we want, you know, the business listing, so to speak, to show up on their site. And the way that happens is by opt-ins, like they have to be added to the site versus just showing up. And the profiles were starting to, you know, even though we've like no index, no followed, you know, all the profiles, they're still somehow competing, you know, in, in search rankings for legitimate, you know… Like I, I don't want our local profile, because they have so many businesses and, and things on it, to outrank any community, right? And the, the reason we did that originally where we needed a place, you know, for the content to live, like have a c- canonical domain, and we've solved that in a different way now, right? So this has sort of been, you know, multi years in the making of, of doing it, but now it's like, all right, I, I really need to just get these profiles gone and just turn them off and be like, "You're, you're done. You're not getting… There's nothing that you're getting out of this at this point." Um, you know, so hopefully that is a deterrent. And then to your other point is with the credit card is, uh, you know, if you're a paid account, it's sort of, it's a pretty strong signal that you're legit, that not too many spammers are putting down a credit card to, to, you know, build, build a link, uh, link farm. Um, you know, so we obviously take that into account. I won't say how. But, um-

Brendan: 30:23

Yeah, that's fair. D- do you guys, just real quick question more on the business side of it, like if I signed up for an account right now and I put my credit card on file, do you bill me for like the monthly fee right up front, like when I'm…

Ryan: 30:36

Uh, y-you actually can't,

Brendan: 30:39

Okay.

Ryan: 30:39

put your credit card in. Um, so our, our whole go-to-market strategy ha- is essentially, like, let's work with what we call anchors, so chambers of commerce, main street associations, local magazines and newspapers, uh, you know, the tourism department at the city. Uh, and let's give them the tools they need to build a resource online for their community and organize events and, you know, uh, host job boards and, and all that kind of stuff. And our go-to-market strategy is then to get, through that relationship, people to upgrade to use Locable for themselves. Now there's a whole bunch of positioning that goes into that, which I'll, I'll skip over for now, right? But it is essentially an enterprise sale for the anchor, right? So we send a proposal, they pay, we turn it on.

Brendan: 31:36

Right

Ryan: 31:36

And right now it is essentially just free for all the businesses, but the, the long-term plan is to have a sort of in-app, product-led, you know, onboarding. So right now there's actually no way to-- Like, we don't have self-service upgrade, right? It's like, if you're an anchor, you pay us, and we turn it on, and everyone else sort of gets into the system, and then we'll layer in the other stuff later. So I don't know where you're going with that question, but that's, that's the reality of our situation is there's no upgrade path to use.

Brendan: 32:06

Yeah. No, all I was going, getting at was just the fact that if you, back to the credit card thing, like if you were putting it in and you were charging them right away, like you wouldn't even, maybe you wouldn't even need that pre-auth. But you're not doing that right now, so

Ryan: 32:19

Yeah, and, and we never will'cause the whole, whole point is the more people that use the tool to contribute events to the community or jobs to the community or posts to the community, the better the community is overall, and we're never gonna take that away from… You know, so there's, there's gonna be a free set of tools no matter what we do, and, you know, the, you know, the game plan is that we're providing so much value for free that you'll wanna upgrade to use the, the paid stuff. And, you know, if we can get, somewhere between 2 and 10 paying customers in every community we're on, we're having a pretty good, uh, we have a pretty good business, and that's sort of the end, end hope.

Brendan: 32:55

Yep. Makes sense. Yeah. Well, I, mean, it sounds like you got some potential to maybe cut down on some of the spam, but still, still gonna be a challenge either way

Ryan: 33:05

Yeah. I, I, I think I gotta prioritize getting, you know, I have a branch open that, you know, is deprecating that feature, and maybe I just need to get it over the finish line. That'd be useful, um, if it made a difference. Uh, in the meantime, I gotta figure out how to… I need to analyze these 60,000 or so businesses that haven't, had human review. Um, and clearly I don't have time to sit and go through all 60,000. So there's definitely gonna be some LLM analysis and see what other patterns we can find to help identify

Brendan: 33:38

Yep. Makes sense.

Ryan: 33:41

Was there any-- Did we talk about anything else pre-show that we wanted to cover? Spam, skills, AI,

Brendan: 33:48

No.

Ryan: 33:49

business

Brendan: 33:50

the, the only other thing that I've thought of right now is, is, uh, just the backlog and getting through that, but that could be a whole nother show. We don't have to do that right now

Ryan: 33:59

What, what do you mean? Just the, the normal, like how do we reduce the number of pull requests?

Brendan: 34:04

the…

Ryan: 34:04

Yeah.

Brendan: 34:05

how do you keep up with all that stuff? And we don't have answers necessarily. I think that, that blog post I mentioned a few minutes ago the thing that, that, Well, not the thing that solves it, but it's like, you know, a, a example of s- a company who has solved it, but, you know, that's probably its own conversation really, so

Ryan: 34:23

Yeah, well, uh, a few things to think about and pick it up next time. But it's like, it's like anything else, um The You know, part of the problem is as developers, we need to be disciplined about finishing the thing we start. I th- I think, uh, for me in particular, that's a problem, um, that I'm aware of, that I'm trying to be better about, right? Is like don't… AI makes it so easy to start the next thing, but it's the last 10% that's hard every time, right? You can get 90% there easy, right? In an, in a hour or less sometimes of like prompting, you know, from your phone while you're doing something else. It's the final 10% getting it over the finish line that is problematic, and that, that problem has always been there, right? We're just letting ourselves, to some extent, go nuts and create all these PRs of unfinished work, right? So I think part of the problem is just having a culture of discipline. Um, that said, as the team size grows, there's other concerns, right? So like I'm, I'm a solo, so like that is controllable by me. If you have even a handful of developers all doing the same thing and it requires, you know, your procedure is every PR has to be reviewed by somebody else, now you're introducing a bottleneck. You know, so the question then becomes, is that bottleneck still valid? Are there other opportunities? Like you just said, can you, you know, have eight agents review it and get, you know, another 9% there? So now it's like instead of 10% of your time, it's 1% of review. And then the other piece of it is like, and, and this is a, a, a problem with AI, um, as well is like I think we've, we've fallen into bad habits of like moonshotting an entire feature at, at one shot, right? Like, you know, when you read about Intercom and Shopify and how they're doing AI, like the average PR is less than 200 lines,

Brendan: 36:28

right?

Ryan: 36:29

right? Well, when the average PR is less than 200 lines, code review is not that hard. Like you can, you can figure out 200 lines, and if you, if you can't, the AI can explain it to you quickly so that you can understand it and move on. Um,

Brendan: 36:43

Yep.

Ryan: 36:43

right, so

Brendan: 36:44

And it's the incremental stuff that we've talked about in the past that, you know, Pragmatic Programmers was talking about, So

Ryan: 36:50

So I don't think the problem is, you know, necessarily just like AI is creating more volume. I think it's AI is creating more volume because we let it, right?

Brendan: 37:01

Mm-hmm

Ryan: 37:02

we've forgotten all the normal programming things that we've been preaching for 20 years that are still really important And even if we had like, you know, an agent of eight, uh, or a, a group of eight agents to do PR review and it was good at it, it would be better if it was still following all the best practices that we always had, right? So,

Brendan: 37:22

Yep.

Ryan: 37:23

you know, those things aren't mutually exclusive, and I think we're gonna find that, you know, once the, the glow wears off, and I don't know how long that's gonna take, but we're gonna realize it's like nothing has changed that much. This-- AI is a tool. It's up to the person wielding that tool to decide how to use it And until we decide that like, hey, we should use it the same way we always would have used it with, you know, always would have programmed without it, I think that'll solve like 90% of the problems, honestly. Um, but that requires like layering AI on top of it requires systems. So we're back to like what we were talking about before. It's like you gotta have the skills, you gotta understand how the harness works, you gotta have CI, you know, you gotta have all the things set up so that it can work autonomously. You, you have to have guardrails. So like that, that's sort of how I think about it right now. And in, in the meantime, trying to be better about not opening 500 PRs a day. Um, I'm obviously exaggerating, but that's sometimes how it feels. You know, making sure that I finish work. And the other thing is like I'm-- I do a lot of, you know, my new workflow with AI is like if I am gonna like take on a big feature or a big refactor or whatever, you know, the first session is like the output is a plan doc broken down by phases, you know, sequenced, and I will commit that right away, right? Like I don't even need to review that, right? Like after, you know, not like code review, like I'll read through it, but like once I feel like the plan is at a sufficient level of detail, I just commit it, and then I can be like, you know, that will lend itself to doing these smaller commits. I'm like, all right, you know, find the next thing that can be done in, you know, less than 200 lines, right? Maybe that's the skill. Um, you know, maybe it's like, all right, you know, you need a skill for data migration, right? When, you know, when there is a backward compatible issue, like how do we approach that? Like, and the, you know, it-- that's sort of like taking known existing patterns of how to migrate data and encapsulating that in a skill so that when you need it, it can be called. But to me, that all comes back to orchestration and systems again, right? And that I don't have a handle on. So until, until there's like a, a real example, uh, out there that is, you know, more than, you know, sort of hand-wavy, which, you know, I can get excited about the hand-wavy stuff, but it's still hard to implement without the detail. Um, but that's also, I think, you know, when you talk about Intercom and, and, um, Shopify, right? Like they're great at sharing stuff, but it's never the mi- minutiae, right? The, the, you know, whether they admit it or not, that's a competitive advantage for them. And, you know, they're sharing as a strategy, not, uh, as a… Well, I don't wanna speak for them. I d- I actually don't know that, that that's true. But like it feels like it's, uh, it can be a strategy and, uh, open source, uh, you know, give back at the same time. But th- that line probably isn't clear all the time.

Brendan: 40:22

Yep. Yeah, no, I, I think that's all relevant. And I mean, that, that was gonna be my point too, overall, is just, like, yes, you can get there, but They also have I don't know how big Intercom specifically, but they're, they're big, right? They're both very

Ryan: 40:36

They have a whole team doing the orchestration part. Like, that's not fair.

Brendan: 40:40

Yeah. Right,

Ryan: 40:41

Us little guys

Brendan: 40:43

they've got way more pull request history to even, like, train their skills or whatever on to, like, patterns to look for. So a lot there. I mean, you And you're right, y- you definitely have to be able to build the system at some point, and if it's just one person or even a small team like mine, like, y- that's not your top priority. And, and then even for them and, and us as well, if it … Like I'm, I've, I've already seen my CI bill go up from extra things. And, like, they even mentioned it in, in either the article or the pul- or the podcast, I forget. But, you know, s- the CI cost became a problem that of course they got to and they fixed or whatever. But, you know, there's always something down the line, and so

Ryan: 41:29

Yeah. I mean, it's like anything else. I think, you know, the adoption of AI is going to be incremental over time, just as if you were building a product, 'cause frankly, you are, right?

Brendan: 41:39

Yeah

Ryan: 41:40

and, you know, I, I think the point maybe of all of this is not, like, drop everything and, you know, try to build a system. It's like do a little bit when it's needed. Um, you know, the challenge for me is like it, it feels like the minimum amount is still a lot right now. So, um, once I get over that hump or, you know, force myself over that hump, I imagine I'll just start like everything else. It's like, oh, this could be a little bit better. Let's spend 20 minutes making it a little bit better. But if I do that every day, you know, after six months I'll be like, oh, I probably have a pretty good system in place.

Brendan: 42:17

Yep, totally.

Ryan: 42:19

Yeah. All right. I think that's a good place to call it.

Brendan: 42:21

Yeah, I think so. Cool.

Ryan: 42:23

right. See you next time

Brendan: 42:25

All right, see ya