Ryan: 0:10

Today we're joined by Chris Gaffney, the owner of Dead Man's Snitch. Welcome.

Chris: 0:16

Hi. How you guys doing today?

Ryan: 0:17

I'm not too bad. How about yourself?

Chris: 0:19

Huh? Doing good.

Ryan: 0:21

Good. Um, well, why don't you tell us a little bit, uh, just about Dead Man's Snitch, what it is, why you tried to build it. If you wanna mix in a little bit of your origin story, you can do that too. Um, but. Uh, I kick it over to you to, to let us know, uh, what it, what, what's going on.

Chris: 0:41

Okay. Um, so Dead Man's Snitch, um, we're this like niche monitoring product specifically around, um, CR jobs and like service heartbeats, that kind of thing. Uh, the, the general gist is like you. Tell us when things are working and when they happen and we kick in and say, let you know when, um, they fail to check in with us or when, when stuff stop stops running. uh, we've been around since, oh man, 2012 I think is when dead message got started. Um, it's a little bit of a, not rocky road, but just, um, so it got started in 2012, but, um. We, it was started by a man named Randy Schmidt, who at the time was, um, doing a bunch of little projects. Um, I don't know if any of you remember like Nerd, merit Badge or, um, lose It or Lose It was another of his, his little projects. Um, and he got started on this at one point, reached out to, um, collective Idea, which is, um, rails consultancy, um, that. was working for at the time and asked us if we wanted to buy it. Um, came around where it was like, yep, numbers worked. We bought it. And then it became this, like one of the products of a consultancy, which, you know, every consultancy wants to have like products. It's an easy, it's an easy sell. Like, Hey, we've got all these talented developers and designers. Can we build something that, you know. Is decoupled from just billing hourly. And um, came to a point where I was like, Hey, if we're not investing in this thing, then it's never going to continue to grow. And through kind of luck, kind of, uh, um, just the business at the time was able to spend some time on it, get it up to profitability and then. Um, yeah, as of pretty much the beginning of the, this year we rolled it out of collective idea, um, and I'm now the primary owner of it with, uh, with Dan, who's also the, um, the principal of that collective idea. So yeah, that's, that's kind of, that's kind of our history. E.

Ryan: 3:01

Yeah. That's very interesting. So, uh, a couple things that, you know, stood out to me is, one, it was sort of a long journey, right? Um, you mentioned, you know, you got it to profitability. Uh, I don't know exactly what that timeframe was more than a year ago, but was that in the last, like five years sort of timeframe?

Chris: 3:22

Yeah, I'd say I've been full-time. For six

Ryan: 3:25

Okay.

Chris: 3:26

give or take. Um, in that time I also have had kids. So, uh, it's, it's, uh, times and dates are very, very fuzzy these days. Um, but yeah, it definitely took four to five years before we were able to really be profitable, um, and sustain like one, one full-time engineer.

Ryan: 3:49

Okay, and, and now are you doing this full time? You're not doing anything with the agency anymore, so this is your primary focus going forward? Yeah.

Chris: 3:57

Yep. Yep. Yeah, I'm, I'm full

Ryan: 3:59

That's awesome. Um, well, congratulations on, on hitting that milestone. That's a, that's a big one for a lot of us that, you know, start with a side project or even in, in your case, a project through another company and then it, you know, grows. So that's pretty cool. Um, you know. Were you part of the, it sounds like you probably weren't part of this decision of why it was built originally. Um, but I, I feel like you have to have an opinion on why it is still a useful product. Now, how do you feel?

Chris: 4:29

Yeah. Yeah. So necessarily involved like Randy. I, my understanding is at the time, like, um, you know, going back more than a decade now, uh. Is was that like Heroku scheduler, um, is sort of a best effort scheduling system where they, they don't make any guarantees about, um, the jobs actually running. So especially when there's like API fluctuations or issues, um, jobs just won't be executed. And so kind of my understanding is that Randy built it around that after realizing that like certain. Scheduled tasks weren't happening at the time. Um, and you know, really a lot of those problems haven't gone away. you know, even with like Kubernetes scheduling the, it's, uh, or, you know, external chron chron tasks are just like servers that, you know, can just kind of disappear in the cloud. Um, there's still kind of this need for knowing. Whether or not these, these jobs are running. Um, and also like a, a, a big, another big use case for us is like, um, alerting pipelines where somebody will set up like a Prometheus, uh, an alert manager install, like in their network. Um, but there hasn't traditionally been a good way to know like, Hey, when my alert manager actually needs to send out emails or send out, uh, web hooks to alert the team that. Something is going on. How do I know that, that, um, that those connections are still up and, and running. And so it's, it's been very consi, uh, frequent to like set up a heartbeat to us to, to be able to know when like those, those pipelines fall over.

Brendan: 6:17

And what platforms are you on? So you mentioned Heroku Scheduler, and so obviously Heroku is one of the platforms, but is, is that the only one? What? What are, what do you support?

Chris: 6:26

so we pretty much support like any, any service, any platform. Uh, the actual API for your using us is, is really simple in that you set up a snitch, which is a monitor. You tell us how often it's going to check in. Um, and then your CR job you are, you add like an HTTP call. it succeeded or after it's run. And that, that's pretty much it. Like we do have an add-on in like the Heroku marketplace. and that's kind of where we got started. Well, we, we, we were open to pretty much everybody and then built the Heroku add-on, like I think a couple months after they announced like third party add-on support kind of deal. So, yeah, I mean we're, yeah,

Ryan: 7:17

in the case.

Chris: 7:17

time ago for a lot of this stuff.

Ryan: 7:20

Yeah, that makes sense. Um, in the case of, uh, you know, something I wanna monitor that may not have a set schedule, right. Like, like a job that, you know, maybe, you know, could vary anywhere from, you know, every 15 minutes to once a month, depending on, you know, 'cause it's triggered based on some action. Right. How would you. I mean, is it worth monitoring something like that? Or is this more really geared towards things that are more consistent? Um, or how would you approach that thought? Because I like, I'm, I'm looking at your homepage and I'm like, all right, new snitch, I got these intervals. And it, you know, it makes sense, right? Like if you don't hear, you know, from if, if it's supposed to fire every 15 minutes, you don't hear from it for 30, you got a problem. Right? Like, and that's, that makes total sense. But a lot of the things, a lot of the background scheduling things aren't. They might, well, I guess as I'm thinking about it by the term schedule, right, it sort of implies that you have to have something. But I might have a job that runs once an hour or once every 24 hours, and it may or may not actually do anything depending on the state, but it would still succeed or not. Right? Whether it takes any action. I guess I've sort of answered my own question, but anything to add?

Chris: 8:38

you, no, I I mean, you would still schedule, you would, sorry. You would still monitor that job, even if it doesn't pull anything from a queue or process any, any data. Um, 'cause you still wanna know, like, Hey, is my, is my scheduler process running? Um, I mean, I've had cases where. I update, you know, you do an app, get update on Debian or whatever you're, whatever, and like, it, it, it updates the cron daemon, but then doesn't start it for some reason. Um, and so you come back later and are like, wait a minute, none of none of these have actually run because I shut my scheduler down an

Ryan: 9:15

Yep.

Chris: 9:15

So. Yeah.

Ryan: 9:16

Yeah, that makes sense.

Chris: 9:18

yeah. And the, the ad hoc jobs are harder because like, you don't. Actually know if it, when it comes in, you can't know when it's, when it's failed in those cases.'cause you're just like, well it could, we could go two months without ever receiving this event that we then need to kick off with the stuff on.

Ryan: 9:37

Yeah. And you know, as I I was asking the question, I realized that those are really two different use cases. And I don't know if I'd actually need to monitor the ad hoc one, right? Because it's being triggered by an action. And, you know, my test suite should theoretically, uh, you know, handle whether that actually happens or not. Um, but anyway, uh, so. You mentioned getting into the Heroku app store fairly quickly after they announced it. Um, you know, what was that experience like or what is it currently like, uh, to, to manage that integration? How do you feel about it?

Chris: 10:19

Um, man, Heroku, mixed feelings about Heroku, like it's still, uh, a great platform. Um, and at the time it was really exciting. They were still getting a lot of investment. I think I, I think this was post Salesforce acquisition. Um, but before, my understanding, um, they. Stopped getting future investment and like, um, releases were paused, headcounts were paused, that kind of thing. Um, and it was fairly easy at the time. I, I think, um, also came out in where like every marketplace has a 30%. Revenue cut. Um, you know, we're still paying that today for customers who signed up in 2018, 2019. Um, whether Heroku has continued to that 30% or not. Um, 'cause especially like when we first got into the marketplace, I think they were doing more to highlight certain, like add-ons that has, has. Gone by the wayside or been filtered out, um, for some reason potentially to like, not show favoritism or, um,

Ryan: 11:42

Whatever motivation they have, right.

Chris: 11:45

whatever motivation they have, but they're, you know, they've, they've clearly cut back on that, like the Adams are gonna market themselves. We're just gonna make it easy to install them. Um, but the actual process of building an ADD o is pretty straightforward.

Ryan: 11:59

Interesting. Would you do it again if you were starting over today?

Chris: 12:04

Um, with the state of Heroku as it is today, or the state of Heroku as it was the dream, the dream of Heroku as it was 10 years

Ryan: 12:14

Well, I, I was thinking today, but I'll, I'll caveat it with, you know, I, I've heard the same sort of, you know, rumors that they're not really, or at least they weren't really putting a ton of money into it. However, I, I think at, you know, from my observation from afar lately, that seems to have changed, right? They're

Chris: 12:33

Yeah.

Ryan: 12:33

their stack, they're reinvesting in the platform. You know, they finally turned on http two,

Chris: 12:39

Mm-hmm.

Ryan: 12:39

Which was a big thing that I was waiting on. Uh. All our apps are on Heroku. So like from a user perspective, you know, I'm still, I still love Heroku. It makes my life a lot easier. Um, but I, I sort of ask the question because what I've seen, you know, I, I don't have a ton of add-ons that I use with my apps, but I have a handful and outside the ones that are like, owned by Heroku, right? Like red as Postgres. In general, I've sort of adopted the idea that, uh, I will go directly to the provider and just set up an account and just cut Heroku out of the middle. Um, you know, sometimes that is because the provider has reached out to me and say, Hey, I'll give you a 30% off if you just, uh, come over and sign up because, you know, they're cutting Heroku out of the middle. And, you know, sometimes I just find that. The feature set doesn't, uh, keep up, uh, on the Heroku side. Um, you know, so the, the heart of the question was, do you still find it worth it now to do it? And if you were to start today with the current state of Heroku, would you even bother? Right. Is it a useful, I mean, it's a marketing channel, right? So it's like,

Chris: 13:49

Yeah. Um, that,

Ryan: 13:53

or, or you don't know. That's that's an acceptable answer.

Chris: 13:56

Yeah. No, uh uh, I will say, like, I agree with you that Heroku has been back to investing. Um, I talked to, I think, um. Head of marketing, um, or like platform marketing, um, a while ago, and it was, was clear that like Heroku was kind of spun into Salesforce at one point, and now my understanding is that their, their, their own organization

Ryan: 14:22

Hmm.

Chris: 14:23

um, with a lot, a lot more resources under that. So rather than trying to share resources with the parent org, they're like. Able to say this is what we need. Um, so yeah, I'm, I'm hopeful for, for them to continue and grow, get back to growing. Um, but as far as like a marketing channel, I don't, knowing what I know today, I don't know if I would've gone, would go through the, the process because I, the 30% does hurt. Um. But again, I don't think that they're doing, especially as a relatively small player, um, they're doing their part to continue to like market us, um, and, and make ourselves visible. I don't how much people are going through their, um, their catalog and

Ryan: 15:17

Yeah.

Chris: 15:17

what's this like, this looks interesting. I should install this for my app. Like, we have to still sell teams on. Their ability to use us. And the actual integration piece isn't that terrible, where it's like we're able to continue integrate. It's sort of like an OAuth thing

Ryan: 15:35

Sure.

Chris: 15:36

they just come into our system and look like a user. Um, there's a couple technical bits to have them appear separately or like, like limit. The ability to do like bill management, that kind of

Ryan: 15:49

Yep.

Chris: 15:49

Um, but yeah, I, I, I will, I, I don't know that the small percentage of our customer base that is coming that is, is on Heroku would've been worth the effort, even though it wasn't a ton of effort,

Ryan: 16:08

Yeah, I mean.

Brendan: 16:09

speaking of marketing, I, I was just gonna say, speaking of marketing channels, I mean, you, you kind of just answered the question. It's not a huge marketing channel,

Chris: 16:16

Mm-hmm.

Brendan: 16:16

what, what is something that you do? That, you know, you find that works? How are you finding customers?

Chris: 16:22

I very much fall into that. I'm a developer first, uh, with a more of a backend focus. Um, and marketing has always been hard for, for me to do. Um, but we're also targeting developers who are. Traditionally very hard to market to.

Brendan: 16:41

Mm-hmm.

Chris: 16:43

uh, I'll say that, I mean the, the most things we try and do, or I try and do is, um, just kinda like engineering is marketing. Um, all of like, uh, I maintain the Heroku jemalloc buildpack for example,

Brendan: 16:56

Okay.

Chris: 16:57

On the Build Pack where people are coming to us from Heroku, we, you know, we include a note about how, like all of this is possible because of Dead Man snitch. Um, and traditionally our best avenues have been developers and operations teams telling other folks about our service and our offering and the work that we've done to facilitate that the most has been like, try and give really good support. Um. But I, I do feel like that it's been harder folks to recommend us over the last couple years, and I don't know if that's just like people aren't talking as much or there's wider competition. Um,

Brendan: 17:40

Kind, kind of makes

Chris: 17:41

yeah.

Brendan: 17:41

about just like, you know, in-person events have, to come back,

Chris: 17:46

Mm-hmm.

Brendan: 17:47

of gone away with c. Other things, and I wonder how much that impacted that as well.

Chris: 17:52

Yeah. Yeah. I'm not sure.

Brendan: 17:55

I'm, I'm sure it's hard to measure, but it just kind of got me thinking. Um, but yeah, I mean, I, I've definitely, I don't use your service, although I, I'm thinking about how we probably could use your service.

Chris: 18:06

should talk.

Brendan: 18:07

yeah, we, we should talk afterwards, but, uh. Exactly. Yeah. And I mean, honestly, coming on the podcast, you know, that's, that's another way to to, to talk about it, obviously. Um, but what I was thinking there, uh, just had to do, and of course I just lost my train of thought a little bit. we, we were talking about, uh, marketing and in-person events and, uh, oh, I was just gonna say that I. I've definitely seen Dead Man snitch. You know, I don't know why I didn't choose it or, or whatnot at this point, but, and I don't recall where I'd found it. So it, are around, I guess, you know, it's not like you're completely hidden. Um, yeah, it, it's always interesting to kind of just think about. Like you said, you're a developer first. Marketing's not your primary thing. I think that's a pretty common thing for developers. myself included, not that I have my own company, but the thought of always like trying to push something on my own is like, eh, I don't know. Although the podcast has helped me with that, honestly. Like just doing this,

Chris: 19:08

Yeah.

Brendan: 19:09

out, talking to people does help. So, I feel like I'm rambling now. I don't really have a good question.

Ryan: 19:16

All right. Well, let, let's say this, you know, you, you've, you know, your journey has been a long one, right? It seems like you're on a, a good path now, you know, you're doing it full time. Where do you wanna, where do you want it to go from here, right? Like, what are your, your plans, your hopes, your dreams? Uh, or is it, you know, you're just sorta comfortable cruising where you're at.

Chris: 19:39

Yeah. Um, so I have three small kids at home, uh, between six and one. And, uh, my of dream for d for Deadman, such DMS, um, is just like, like, not like there's no exit plan here. There's no sale plan, there's no like. I want this to be a billion dollar, uh, a year business. Um, I mean that, that'd be really cool. We're not there yet. Um, But like me personally, like I want to be able to do an interest in engineering and I want to be able to help like these operations teams that are using us, um, to, to get, I don't know, like I, I just want to kind of have like kind these teams, like I'm not. some big business that's like growth oriented and focused on like the bottom line and sales people and sales pipelines. And, um, there's, yeah, it's kind of a rambly answer, but, um, I've been thinking a lot about like the, like a small business, like this is very much like the extension of the owners themselves and, um, like. I want the work that I'm doing to be meaningful to other people, and I wanna be able to expand that, that more, and reach more teams. Um, and I think we're doing a good, good job there. There's always, there's always more to be done. Um, but, uh, if it's, if it like a, it's a case where I want like other teams to be, to want to work with us as well because they get. Not value just from the, the software, but like the interactions we have and, and to support this, this kind of small team. So sort of a weird like, answer, not answer, um, but it's, uh, it's definitely more of a like, let's keep this business and provide a good, you know, good life for my family, for, uh, hopefully another, you know, engineer or marketing person at some point once we've got the, the revenue for, um, you know, a second full-time hire, that kind of thing.

Ryan: 21:53

I think it's a,

Brendan: 21:54

thing that kind of jumped out. Go

Ryan: 21:55

I was gonna say, I think it's a, you know, maybe a, a trap question, right.

Chris: 22:01

Hmm.

Ryan: 22:02

There's nothing wrong with that answer. Right? You know, you can have a very, I call it a lifestyle business, right? Like you can have a very nice, you can make a very nice living, you know, have a good career alignment, something you love doing. You know, you're supporting your family. You don't have to grow it to, you know, anything. And I think that's a perfectly.

Chris: 22:24

Mm-hmm.

Ryan: 22:25

Fine answer, right? Like, there's nothing wrong with it. So, um, you know, I, I got the sense that you're like, well, I don't really have an answer to that question, but I think you do. I think you have a good answer to that question.

Chris: 22:37

Yeah, it's, it's, um, if I had, uh, rather than going full-time in this, like taking a full-time engineering job somewhere else, like. We would have more money in the bank today than, than by running DMS. Uh, but, um, it's al also allowed me the freedom to like go pick up my kids after this. Um, and and like spend more time with family, which is really priceless.

Brendan: 23:07

Yeah, it's important stuff for sure. And I forget what the exact quote was, but I just heard a quote the other day from a different podcast. Um, and it was basically along the lines of if it had to do with, you know, taking a job versus owning your own business and how. The, I think the, the point of the quote was basically like, how to make a lot of money. That's not necessarily what we're getting at here, they were just kind of saying that like if you, if you have a steady paycheck, you're like not gonna reach out and try and do anything to, to make that extra money. like, you still have the opportunity, not that you necessarily want to get there. Obviously, like you said, it'd be great if you did, but you know, you've got a little more control too, which is, you know, both good and bad. But,

Chris: 23:51

Yeah, and a lot more paperwork.

Brendan: 23:53

Yeah, I'm sure. The thing I was gonna mention earlier, after you answered Ryan's question, you know, I think there's something, especially in this world of AI and all the potential new products that could be spun up, are gonna be faceless and nobody's gonna care. Like to have someone you know, like you behind a product and be the, not necessarily the main focus, but like to be able to get answers from that person.'cause. If that's the other part of the AI stuff to me is that anybody sure they could spin up an app, but like the three of us know how to run an app already

Chris: 24:27

Mm-hmm.

Brendan: 24:27

a thing. So I would come to you, Chris, and trust your answers more than I'm gonna trust somebody who just vibe coded an app and now they've got dead man snitch, clone or whatever it is that doesn't even know how to answer the technical questions anymore. So that's at least where I value things as well. Um, for what it's worth.

Chris: 24:47

Yeah, I, I think back to a couple years ago having a do customer support request where they had some very technical questions about cron and, um, doing a very unique schedule. Um, and I was able to answer them and like, well, I, I forget the, the specifics, but it, it was one of those like. On the weekdays, uh, or the first, second of the month kind of deal. And that's not something that they would've, you know, they might've been able to ask AI today. Um, but knowing that that comes from like not just, um, oh, I read this on the internet, but hey, I've actually like, through the imp an implementation of cron myself and like looked at the source code and, um, you're getting the support for me for free rather than. like having to pay like consultant rates kind of deal.

Ryan: 25:40

Yeah. So I, I think that, um, maybe is a good segue into my next question, which is like, what has been the biggest sort of technical challenge, uh, you know, with building. This app or you know, if there is no technical challenge, what's the most interesting part for you? Uh, if that's a better question,

Chris: 26:00

Yeah. Oh man. Technical challenge is just like, as a monitoring service, people expect us to be up like a hundred percent of the time. And, uh, there's constraints there with, you know,

Ryan: 26:15

I.

Chris: 26:16

and my partner, like I'm the primary ops person. Um, and so making sure that if I am. Gone for a weekend. You know, there are systems and tooling in place that, uh, you know, when if a downtime occurs that uh, ideally things will kind of route around the problem. Um. or be backed up in a way that like, once the problem gets fixed, we're able to consistently replay like the history. Um, and so there's a, there's just been a lot of effort there and, uh, frankly like deep technical problem solving to be able to say, how do we have as much uptime as we possibly can? so yeah, I, I think that that's kind of like the biggest technical thing.'cause like, it's not something that you see. Going to the application, working through it, like it's all behind the

Ryan: 27:14

Sure.

Chris: 27:15

um, technical choices and, uh, a lot of complexity that, you know, I have to keep in my head. Um, or yeah.

Ryan: 27:24

How so, you know, this is just outta curiosity and, and share what you're willing to or not willing to. Right. But the, you know, what sort of things do you reach for when you're trying to, you know, quote unquote have a hundred percent, uh, uptime? I mean, is this like, you know, uh. Distribution proxies, Redis, combination of all things like, you know, what does that stack even look like? That that's a problem that I have not had to deal with yet. Um,

Chris: 27:59

Yeah, so just kind of give, like the main web application is all Rails. Um, but then we have some pieces in Go. Um, I like managed services because I know for like our database, which is a with crunchy bridge, that there's somebody on call 24 hours a day. But not only that, that like when we have our, um, you know, primary, secondary. Uh, cluster setup that they've tested that, that failover process, they know, you know, they know that it's going to work. And I, and they will attest to that. Um, 'cause that makes me feel better if, like, if there is a database issue, um, with the cluster itself, that their tooling will work. Um, but then we also, like, use Kafka. Um, from Heroku, which has been, which has been good, which has been fine. Um, and that as a technology is more resilient to fail to, to failures. Um, it's also fits kind of our model where, you know, there's always like an ordering of events that kind of go onto the Kafka like topic partition pair, that we are able to process. So like. are kind of the key things. And then the rest is just like trying to keep each component simple and minimal. Um, like we've got a service that does our main like state machine for what does the history look like, do we alert? And that was originally just like Postgres and um, Redis. Um, but then last year we're like, Nope, like against Redis. We still use it significantly in other parts of the app, but it's like, okay, if I can minimize. The dependencies on this particular key piece, it's going to just be Postgres and then Kafka for the, you know, inflows and outflows of, of data. Um, so that, that's all been really helpful. But yeah, another thing, it's, it's just how do we actually structure our data so that a lot of it is a timeline and, you know, events can happen at different times and we can take key pieces like that state machine out of the rotation. Do maintenance on it without affecting, you know, the histories is too much, um, or at all, ideally.

Ryan: 30:22

So do you have a lot of, uh. Outside of the actual alert, do you do any reporting, you know, sort of, uh, like time series reporting? Like do you run in, like, I imagine you have a large amount of data going through your pipeline at any given time. Right. And that challenge alone is, is one that is, uh, hard to deal with.

Chris: 30:46

Yeah, I mean, it's mostly just, you know, is this thing, ha this thing run, has this thing checked in or not? Um, we do have, we don't, not reporting so much, but you can see like, Hey, what's the recent history of, of a snitch? And yeah, we do keep. Data, um, like two plus years. Um, because we've had people come to us for like compliance reasons, just like, Hey, can I verify that this task ran in January of, of 2024, for example. Um, so that's all available to us, but we've. So far decided not to expose

Ryan: 31:27

Right.

Chris: 31:27

because it just adds kind of complexity on a, on a day-to-day basis. Most people are coming in like, Hey, what's, what's the history of, of this job in the last like 24 hours, 48 hours? That kind of thing.

Ryan: 31:38

So with, with saving data that long, I mean, are you running into sort of storage and database size issues and how are you dealing with that?

Chris: 31:51

Uh, a little bit, um. And that's one where we've done some things that were probably not the, I, I wouldn't advise most people to do. Um, so one of the initial versions, we kept every single check-in in a database. And we're doing these crazy queries to say, Hey, has, have we received an, you know, an event for this particular snitch within this time window? Um, that did not scale particularly

Ryan: 32:22

Mm-hmm.

Chris: 32:23

Um, but the kind of the new newer iterations of it are actually, it's like. There's a, there's a table that represents a, a time range of expected check-ins, um, expected runs for a job. Um, and then that actually is stores an array of proto buff, uh, packed data. So that, like for those multiple years of data, we're, we're keeping, you know, everything in a manageable size. So it's a lot smaller than if we kept like every single event, um, in its entirety.

Ryan: 32:59

I I'm asking for personal, uh, uh, development reasons.'cause I'm running into that problem with our analytics, right? Our, we have like two tables that are like, make up 96% of our database and they grow quickly. So it's like, all right, what is our retention policy gonna be? You know, if it has to be a certain amount of time, like how do I make the. You know, do I have to start partitioning? Do I need to sort of offload that historical data? Um, you know, are there, you know, do I need to do sharding to make the reads more efficient? Right? All of those things are in play, but I'm not ready, quite ready to reach for it, right? Right now, the, the easier solution is let's just bump up the Heroku database to the next level, and that solves, you know, buys me time to figure out what. You know, the, the more permanent solution will be.

Chris: 33:53

Mm-hmm.

Ryan: 33:53

Um, but also anytime I do that, I, I, I cringe a little bit because it's, you know, Heroku is great, but it's not cheap. Um, and as you get to bigger sized databases, that number goes up fairly quickly. Uh, so it's like, all

Chris: 34:07

it also makes transitioning to a new provider that much

Ryan: 34:10

right.

Chris: 34:12

when you're, when you're dealing with so much more data. Data.

Ryan: 34:14

So, you know, I, I ask those questions because I'm, you know, constantly thinking about, all right, what is the next thing that I'm gonna need to handle? And that is, uh, frankly, just an area of expertise that I don't have yet. Um, right. I'm, I'm learning, I'm trying to gather information, and at some point I will actually have to solve the problem, and I'm sure I'll learn from that. But, uh, in the meantime, what are other people doing? Is, is always a question I ask.

Chris: 34:41

Well, and, and one of the constraints with us that like gives us a few options is that after a like job is done. should have completed. Um, and we've alerted that it's missing. Like that data never changes. So we keep it in the database right now, um, but could easily spit that off to like an S3 bucket, for example, like some other long-term storage's, like a, you know, a file. Um. But just haven't gotten to a point where we, we need to do that either.'cause like then that adds a bunch of complexity for like, what happens when somebody goes back and looks at something from three years ago.'cause that can actually happen in our system right now. Um, now we're pulling data from S3 and, yeah.

Ryan: 35:28

Yeah, it, it, it adds complexity, right? So you don't wanna reach for it too soon,

Chris: 35:32

Mm-hmm.

Ryan: 35:33

right? You do it when you need to do it, but no sooner.

Chris: 35:35

Yeah. Yeah. And I, I feel like there's a, a lot of things, um, I mean, I said, we've been around for 13 years and I'm revisiting a system this week that, like I wrote initially five, four or five years ago, it's like, Hey, it's been working great until it just hits this inflection point with even like, um. Um, data size or like changes in operations where it's like, okay, now I actually need to solve this problem a lot deeper to guarantee that, you know, the, the specific instances like web hook ordering, just to make sure that they're going out in the correct series. Otherwise end users or customers, own systems can get outta sync with, uh, you know, based on the ordering of, of when they receive those. And it's just like, it worked great, but.

Ryan: 36:30

It works until it doesn't.

Chris: 36:32

It works until it doesn't. Yeah. Yep.

Ryan: 36:34

Yeah.

Brendan: 36:35

Which in my experience is just typical scaling, right? Like you don't have to over optimize at the beginning, but

Chris: 36:41

Yeah.

Brendan: 36:42

hit a point if you're lucky, you hit a point where you do need to optimize now because you've got the traffic and the customers and stuff, so,

Chris: 36:47

Hmm. Well, and if we'd done this work four years ago or five years ago, it would look very different than it does today.

Brendan: 36:57

sure.

Chris: 36:58

um. Probably we'd still be doing the same, same kind of re architecture that is happening today. So it's just like you don't have to solve every problem

Brendan: 37:08

Right.

Chris: 37:09

its completeness to be able to have a wor, you know, working application that people depend on. I

Brendan: 37:15

Mm-hmm. Yep. Exactly. Um. One other question I had really quick that I just thought, you know, we're, we're the Rails Business Podcast. We haven't talked a whole lot about Rails and that's

Chris: 37:26

wait.

Brendan: 37:26

fine, I, I, you know, you're a small shop and not shop, but you're a small company. Like

Chris: 37:32

Mm-hmm.

Brendan: 37:32

what, what are the things that you think without Rails, or maybe a better way to just say, it's like, what, what does Rails give you that's helping you, you know, run the business successfully?

Chris: 37:41

And. I feel, I feel like I'm much more productive in rails than in like the go pieces of our, our architecture. Um, I mean that's also like, I've been working with Ruby and Rails for 20 years now, since I think my first version of Rails was like zero 12 something, which I think was 2005. but yeah, it, it's a comfort zone, but like, um. I just am so much more productive with like, Active Record. Um, the tooling around like, like, like the Stripe, API gems are great that they, they provide them and. Working with their API is sometimes harder to do than I wish it was. Um, but just like I can, I can get in and I can build a prototype and I can test it like in a console, for example, before going through the, the full process of building it. Whereas like go, I very much have to have like an idea. Of what I'm trying to build and just those iteration cycles just feel so much slower, um, that it just takes everything longer to, to come to a final form.

Brendan: 38:59

Yeah, that makes sense. Yeah, it's, it's kind of what I expected just based on our

Chris: 39:02

Hmm.

Brendan: 39:03

conversation is, is what you said there. So, um, I guess is, is there anything else that you wanted to, to cover, you know, before we jump off or?

Chris: 39:12

Man. Um, I don't think so. Not necessarily. Um, and we haven't, we talked a bit of technical, we fell into that trap, but

Brendan: 39:19

Huh. That's all right.

Ryan: 39:21

Some technical is good. I, I, I like that part. Just, um, you know, from a practical perspective, I want to try to fill, uh, a gap in the podcast market with talking a little bit more of the business side, which, you know, I think is missing. So, but, uh,

Chris: 39:36

Mm-hmm.

Ryan: 39:36

thank you for joining us. Uh, where can people find you?

Chris: 39:40

Um, GitHub, uh, I, I am a, I'm a bit of a, uh, social media. Uh, hermit as it were. Um, gave it up years ago and have not really missed out on

Ryan: 39:54

Good for you.

Chris: 39:55

So yeah. Um, gaffneyc on GitHub. Otherwise, um, I guess check out Dead Man's Niche and if you have any questions about it, like reach out to support and you're most likely to get me, so,

Brendan: 40:08

Cool.

Chris: 40:10

yeah.

Brendan: 40:11

lot.

Chris: 40:11

Yeah. Thanks guys.

Brendan: 40:13

Yep.