Rails Business
Brendan Buckingham and Ryan Frisch talk about developing with Ruby on Rails and how to leverage it to build a business.
Rails Business
Breaking Down Work into Manageable Chunks
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this discussion, Ryan and Brendan delve into the challenges and strategies around managing Stripe invoice rate limits and the broader topic of breaking down work into manageable chunks. They highlight the importance of breaking down large tasks into smaller, more manageable pieces for better estimation, easier reviews, and improved project flow. They also share their thoughts on estimation and the Shape Up methodology, emphasizing the benefits of delivering constant incremental improvements.
00:00 Introduction to Stripe Invoices Rate Limiting
00:15 Manual vs. Automatic Invoice Processing
01:13 Rate Limiting and Its Implications
02:22 Different Approaches to Invoicing
05:02 Stripe API Changes and Challenges
08:46 Breaking Down Work into Small Chunks
18:57 Project Management and Estimation Techniques
23:33 Conclusion and Final Thoughts
Links:
So , Stripe invoices getting rate limited. Let me see if I can articulate this. And this is my current understanding based on the conversations I had earlier. So subject to change as my understanding gets more clear. But when you, with invoices, you can create an invoice. So you can make a call to the create endpoint, then you can finalize an invoice by hitting the finalize endpoint, and then you can send an invoice by hitting the Send endpoint Now. That is like the manual way of creating, finalizing and sending an invoice to get it to send when you want. The other way to do it is when you create the invoice, you can select auto. I believe the attribute on the API is auto advanced to true and it will automatically finalize and send the invoice, a predetermined amount of time later based on a setting in the dashboard. So you said by default it's one hour? Or you can create the invoice with auto advance set to fall so that it always stays in a draft until you're ready to finalize it. And then you can finalize it with auto advance True at that point, and it will send, basically, you just can't call send, right? If you hit the Send API endpoint, you are subject to the rate limit. If you don't, you're not, which to me is silly because. I can finalize it and then have a setting that says, send it one second later. What are you really accomplishing in that one second for Stripe? Yes. Their response was bad actors abuse the send endpoint to I guess get past what I assume is some sort of sophisticated system that tries to defend against it. Which is great because. We've always just manually sent it when it was ready so that we know that it went out and that we could get the web hook back and, process it and display it to the user. That's not necessarily a bad thing, that there's a delay. Yeah. It's a second fine. If it's an hour, that's fine too. It's just gonna require us to re-architect how we handle it, which to some extent I want to do anyway. After having run it for a couple years, like I know where the pain points are and, I have some ideas on how we can clean that up. We're gonna do a sprint on that probably in a month or so and see where we end up.
Brendan:Yeah. Yeah. I know you and I have talked about invoicing a little bit as well. And what I realized as you were talking there is that I must be hitting a slightly different rate limit because what I was remembering or just like thinking through like the difference between your system and ours is that we actually don't put our invoices into Stripe. We basically have the invoice in our own system and then we call charge to Stripe to just charge. We do get. The rate limit, but the rate limit isn't what you're talking about. And that's what prompted me to ask the question of how do you get Stripe to call you? I don't have any of that set up in my app.
Ryan:Yeah. Yeah, let me make sure I understand what you mean by call you. You mean have it, you're not talking about a web hook, you're talking about just like it sends the invoice on your behalf. But I don't fully understand that question. From my perspective, the way
Brendan:we do it is like all the invoicing is internal to our app. So we build an invoice, we send the email out, and all of that is done by us. That right is separate from Stripe. All we're really using Stripe for is to charge the credit card. So we build that invoice internally, you know, calculates all the different numbers, and then we run every single one of those I send I, have a. A bunch of Sidekiq workers that run in parallel.
Ryan:Sure.
Brendan:So I've got quite a few that are hitting all charge. What do you
Ryan:mean by call you when you said that? What, what were, what I guess was going through, I guess maybe I just,
Brendan:yeah, maybe the way I'm saying that isn't quite right because I was, I. Trying to figure out what you meant about the way they rate limit, if you call the send versus them charging your invoice for you or creating it. I guess maybe that's a little confusion on my part still. Yeah.
Ryan:My best understanding is I. If I hit the send API endpoint that, that would be like me calling the API endpoint, right? So like I am dictating when it sends, versus the other way around is I have no control over when it sends. All I can do is finalize the invoice and then stripe based on whatever settings I have set up in my account besides when it's gonna send the invoice. And the difference is, it's their workers, you know their queue, right? right. So, yep. In that scenario, not that I wasn't doing this already, but you basically just have to listen for the web hook that, hey, the invoice was sent, and then you can update your database accordingly to show that it's sent versus I Gotcha.
Brendan:Yeah. Getting
Ryan:ready to be sent or whatever status you wanna call it, but with the charge endpoint. Man we could go deep on this one, but the, the stripe API has changed a lot. And I don't even think charge is like the most current,, right? So they have charge, I think is equivalent to. Maybe it still is charged to actually like do the final charge of the card, but you have to create a payment intent now, right? Which is like the intent to charge the card. And then I don't know off the top of my head what the endpoint is to actually finalize the payment intent. It may be the same sort of. Workflow where you create the intent to pay. Then you can manually charge it or you can allow them to auto advance the payment intent to create a charge. That would be the equivalent, I think, but I'm not as familiar with that part of the API.
Brendan:Yeah you're probably right in the fact that like what I'm calling and what you're thinking is the new one is. Something different because I know we're on an older version of Stripe. We looked at upgrading it a while ago, and it took not surprisingly, a long time and we never quite finished it. We still have never had a good reason to, although I'm sure there's something we're missing. But
Ryan:Yeah, yeah, that to me that, go ahead. I was gonna say that to me the reason is at some point they're gonna deprecate the old. Oh, totally. Like that. That's the reason. I'm right there with you. I think I'm running three different versions of Stripe, right? Depending on when I built a feature, right? Some of it uses the old, like the original checkout flow where you embedded the JavaScript, on your page and you handle it as if it looks natively. That was the original. And then somewhere along the line we introduced the payment intent stuff, which is the newer stuff. And somewhere in between. We use the, hosted checkout, right? Which is very similar to the original checkout, but we, you, you get redirected to a Stripe hosted page to actually, put your card information in and then you get. Redirected back. And I know since I implemented that, and we always tend to implement these things like two weeks before they make some big announcement changing something that seems to be our luck. But now they have the native sort of version of the hosted checkout page, which you can embed into your site and make it look like it's yours, even though it's stripes. So we're somewhere in between all that mess in terms of, the different ways you integrate, but frankly, it's hard to sometimes keep up with the changes when you're a smaller company. I think Stripe does a really good job of not forcing you to do it right. Like they Yep. Support you for probably a longer time than anyone else that I've observed. So that's always nice. So there's no worry that we're doing something. Bad or wrong, it's just there's a new way now. Yep.
Brendan:Yeah, totally. And I think that was one of our big holdups when we went to do it is that a we bid off way more than we should have all at once. But part of that I think had to do with the fact that they did change their interface. So now there was like that payment intent. I remember that, although we don't currently use it. It was like a, basically a different model, you have to build out your rails model differently than you, at least that we were or do. And I think that slowed things down. I seem to remember some JavaScript stuff too, but I don't recall enough of that to remember what it was. But it's the kind of thing though that's been in the back of my mind for quite a while. Okay, when are we actually gonna loop back around to update Stripe?'cause I, to your point, it. Probably will be deprecated at some point.
Ryan:Yeah. This, that's a good segue into what we thought we were gonna talk about today, which is, why it's important to break down work into small chunks. And, but before we dive into that, there's this, anytime you're making a big change, like to something critical like a stripe integration where the, I'll call it the. Domain model, whether it's on Stripe side or on your app side, changes which require, meaning that they require some sort of rethinking of how reflect that in your database perhaps. Yep. How do you go about even. Rolling. That kind of change out, right? When there's a, it doesn't have to be Stripe, but that's the one we're talking about. But just the idea that I know I have to change what my database table looks like.
Brendan:Yeah.
Ryan:In order to support the upgrades that I want to make, how do you break that down into small chunks or small stories or whatever you wanna call 'em, and do it in a way that's not, that's backward compatible, that you're not gonna. Break everything as you go through this process.'cause a lot of the changes that I feel like I would need to make to update the stripe API is just like rethinking how we store that data on our side, which means there's gonna be a database migration, or at least a data migration if the table itself doesn't change, right? I gotta move, an invoice ID from one model to another or, when you're working with. Money and, particularly other people's money not something you want to screw up. So may maybe we'll put that in the back as back of our minds as an example. And let me start out why do you think it's important to break down work into small chunks?
Brendan:Yeah. One of the biggest things I have running a team especially is, the bigger the prs are. Then I can't review as many and the team can't review as many, so it just slows down all of the cadence of shipping things. And granted, you still have to do the same amount of work whether you've got small prs or big prs, but I. If you're coming to anyone with, 30 file changes in your final PR instead of maybe one PR that's got, I dunno, two or three. And, they increment that way. So they're much smaller. They build on top of each other. And that, that's the biggest thing in my mind, from a manager's perspective is that, but it's also like just thinking as a, developer too, it would be easier to write that code and get that functioning, just that small bit functioning and also get some feedback right away. It's like I, if I'm writing a code for you and then I'm like, Hey Ryan, can you review this just so that I can make sure I'm on the right track? That's another advantage I see with doing it.
Ryan:Obviously we've been working on a project together and we've been trying to. Break these stories down into manageable chunks based on the definition that you just gave. I feel like we're failing in that in that attempt. And that's not to say that we're not making progress on the project, right? The project is moving ahead. Like we've had actually good communication and feedback as we've worked together. You know that, that stuff is all good. Perhaps the missing thing is we still have a big PR that's gonna have to be reviewed. Even if I'm, I've been reviewing it as we go, and I can see the incremental pieces. But when you get to the end and it's like, all right, this is, final push to production. Yep. Doesn't matter how many individual. Checkpoints I reviewed along the way.'cause I don't know necessarily what's changed in between each one. Part. Now I gotta go again and, if I'm gonna be prudent, I gotta say, all right, I can't just look at the one thing that, the last two commits you made on that pr I gotta say, all right, I gotta look at this whole thing as a group again, and I'm not sure that actually, accomplished what we set out. To accomplish.
Brendan:I agree. I've run into this as well with my team. A lot of what'll happen is, we'll try and break it down and probably not even into small enough chunks, but either way we'll do something like, say let's just take an example of say like a homepage redesign where you've got maybe five or six sections of that page and they don't necessarily all have to be done at the same, or you don't wanna do them all at the same time probably, but. You wanna push the whole thing out together. So one thing you know you could do is you have a separate PR for each of those sections, but depending on the pace of the developer writing the code and the pace of the person reviewing it. If you're not keeping up and actually incrementally, testing and essentially checking off that first section or the second section or whatever, then you run into that problem you just described where. Maybe you check in, but like you don't really know what's changed and maybe something changed and so you get to the end. And I think the difference there is in traditional, like incremental development, as I understand it at least. And I'll note that the pragmatic programmers book has the same kind of concept, but they call it tracer bullet development. And the reason I'm bringing it up is'cause their definition is basically make it a small increment of. Not like a front end versus a backend kind of thing, but make sure that it's a whole piece. I guess in the case that I the example I always had before, there's not like a, I guess a good way to describe that, maybe like the CSV upload is a better way to put it. Like you could have the CSV upload. The first piece is just, you can upload a file. It doesn't matter what that file does, but you can just upload the file so it's front to back. And everything's there. And then the next piece would be maybe you add in validations for one of the columns and then you can increment that way. And if you did it that way, ideally, you could review it, check off and actually merge in the code. So we've essentially said at that point, file upload works. We're done with that. We don't need to really review that part again. Okay. Validations on whatever company name works now. Again, we both, as the developer and the, reviewer, they, you need to be able to keep up with the pace, otherwise the developer might just keep going and then it's at what point do you just review the whole thing again?
Ryan:Yeah. And I think we, just for those listening, the project that we're working on is a CSV importer, right? I don't think we actually said that. So just to make sure everyone's on the same page. I think we. Did that fairly well with the first story, which was create the upload right front to back. Yep. Now? And we did deploy that to production and it's behind a feature flag so that nobody can see it. To me though this one is actually a little bit easier than say, even though we had an importer before, we're just creating we're putting it in a new place in the app. So there's new routes, new controllers, new views. Essentially greenfield development, even though we're starting from a known, past point. And that makes it a little bit easier to do that, right? Because I can just deploy it behind a single feature flag and no one's gonna know the difference. Or that gets trickier is let's say these endpoints already existed and we had to start implementing the new one on top of it. I guess you can still use a feature flag, you just have to do it within, the same controller, the same view. It gets a little bit trickier I think.'cause you have to then worry about not breaking the existing functionality while you're creating the new functionality. Anyway, I don't know if I had a point there just talking through it. We got to the second step, so we had upload a file and in our case we started on S3. So it was there for future reference. If I remember correctly, step two was then take that file and import the data.
Brendan:Yep. Create like basically to workforce, create
Ryan:a right. And we, I think we did okay with that one too, right? I think where we fell apart was maybe this last one, right? Like we, yep. It got big. There are two types of imports.
Brendan:Let's talk about that. I would agree. We've, it got big and I probably didn't know enough to push back to say, Hey, why don't we do this? But knowing what I know now, I see opportunities where we could have said, okay, we're doing company import. And there's, I don't even, the count maybe more than I do, but there's 30 or 40 columns, something like that. We could have, we don't necessarily have to say every single column's one pr. That'd be overkill. Yeah. But maybe we take really easy ones. Let's just say like name and address and the things that like don't really need any massaging on the backend. Like that could be, we could've done the
Ryan:core company record and not all the associations, for example.
Brendan:Exactly right. Yeah. And then that's what I was gonna get at you could then do , one of the associations, and then the next one and so on. And we still could do that.
Ryan:the purpose of doing that is to make the reviewer, you know, the, the flow better, right? For the developer and the reviewer. So it doesn't feel like it is big amount of work. Probably more importantly, when you're reviewing it, not a as big of cognitive load that you have to load up all at once to understand all the moving parts. Yep. So that, that makes total sense. So maybe we think about that. It's, I feel like we're, we have contacts and we have companies, so two separate importers that we're working on, and I. For the most part, the contacts one is done, so we should probably just get that outta the way and then decide how we want to move on the company one. But we can, let's talk about that separately. All right. What other, are there any other reasons why you wanna do. Small chunks for you. Like I, I've heard a bunch in the past and I don't wanna like cloud the judgment. I don't agree with them all. But what else do you, is there any other big ones for you?
Brendan:The two I already listed are the big ones for sure. I'd be curious what you've heard and what you either agree with or don't agree with.
Ryan:One argument I hear frequently is the idea that it's easier to estimate. Smaller stories now. I personally don't like estimation. I think we're all really bad at it, and it's a, I understand the desire for it, the execution. I've never seen an execution that I like on it. So I just, I don't estimate. I'm more of a, in the shape up methodology where you have a set amount of time to work on something and the scope changes versus the amount of time takes Yeah. Your
Brendan:appetite right. Is what they call it.
Ryan:I don't remember the exact stuff, but it's, I read the six years, so
Brendan:I'm familiar with it. Yeah. Yeah. I think the way. Instead of the estimate, it's the appetite. What are you willing to work on for the timeframe? Yeah. Which I actually, I like that idea as well. And we don't estimate either. I agree with you there that like estimation is really difficult to get right and, unless the business side is really pushing towards that sort of thing. It I think it puts extra pressure and even extra meetings that don't really need to be there. To get things done. Yeah.
Ryan:Look I'm sure there's at a certain scale of company, following that and using that as a tool probably makes sense, right? But for smaller company, even medium sized companies, I think, if you want to have like perfect estimates, right? Like you, you basically say, if you do six week sprints throughout the year. And you say, all right, there's only so many six week periods throughout the year, and you wanna put some buffer on either side for other work, or, break or whatever you wanna do. You can basically decide at the beginning of the year generally what you wanna work on in each of those six week sprints, right? And then you get the group of things that you're considering for that particular feature, or epic or whatever you want to call it. And then you say, all right, what can I get done in six weeks? And you say, all right, this is what I think I can get done in six weeks. And then you just start working, right? And you follow this. You break it down into small stories. You try to do incremental development, you write tests, you do it as you go, and as you progress, you just constantly look at the feedback and be like, all right, I am not getting. As much done as I thought. So I'm gonna have to change the scope, right? I'm not gonna extend the time that I have to work on this thing. I'm gonna cut features or scope back until I can fit it into that six week period, or vice versa. It's hey, this is going really well. I've gotten a lot further. I can now pull more stuff into this epic or sprint or what, again, I'm mixing words here which is probably not a great idea, but if you follow that methodology. You're always gonna deliver on your timelines 'cause there's no way you can't. So if you wanna be, the challenge with that is you gotta be in agreement on what the priorities are of that particular feature. And that is like constant feedback with management, with the product leader, depending on your organization, just depends on how you guys are set up. But to me that's much more appealing to say, look, we're gonna, we're gonna push something, at the end of the six weeks and it's gonna be. Incrementally better than what we had before, because I'm slowly pushing stuff as I go and putting it into production. It's almost once you get into that mind flow, I find it hard to think of things in any other way.
Brendan:Yeah,
Ryan:I have not, I've been developing for, I've lost track. It's been almost 10 years in Rails and another 10 years and other stuff, but I still, this is the. The best system I've come up with to date. And, I'm constantly trying to, find better ways to manage expectations, manage, product development so I just, I find it all fascinating.
Brendan:Yeah, and I think I would agree with a lot of what you say there. I think the one thing I'll add is whichever system you're doing, whether it's a two week sprint or six week, shape up you, whatever. I forget what they call 'em, not sprints. I know they hate sprints. But either way the, my point is there's still that planning that needs to be done ahead of time. Otherwise, you're not gonna hit, you're not gonna know what to work on for one. And you gotta have things scoped down, at least in the shape up methodology, since that's what you prefer. And I would agree I like the idea of it. I've just never actually worked, really worked in it. There still is the concept of, okay, you come to the table with your. Plan or at least your story. But you gotta make sure that story fits into six weeks. And if it's not going to, then you cut out the things that do, and you can do it while you're working on it too, which is probably more what you do because you're a small company. You don't need to be doing all that upfront work, but you have to have something. Yeah. I
Ryan:would say I, I have a variation of Shape Up, like I've borrowed the pieces that work for our company and specifically it's the thing that I like is I can say, all right, from February 24th to April 4th, which is a six week, six work week period. I'm gonna work on this invoice problem and re-architecting how we do membership. Right now, the goal is to address a few common, customer complaints about the way our system works and to fix this rate limit thing which is gonna require a little bit of rethinking of how I, integrate with the API. And then there's a list of other things that have been requested by customers or our team or whatever, right? But I know my top priority item. Yeah, I know. You know the next group of things, and I know here's some wishlist things, and I know I'm probably not gonna get to the wishlist. I know I'm probably not gonna get to that middle group, but I know I'm gonna finish this first thing that is the most important. And if I can sprinkle in a few things here and there on top of that, everyone's gonna be pretty happy because it's come April 4th, there's gonna be an improvement. And actually that's not even really true. It's come, day one after I start this work, there's gonna be improvement if I'm, pushing incrementally throughout the day and throughout the week,
Brendan:right?
Ryan:Yep. Anyway, maybe that's a. Good place to cut it off.
People on this episode
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
Code with Jason
Jason Swett
Standing in the Fire
Very Good Software
Remote Ruby
Chris Oliver, Andrew Mason, David Hill
IndieRails
Jess Brown & Jeremy Smith
Startups For the Rest of Us
Rob Walling
Vibing with AI Code
Brendan Buckingham