Mint Shutting Down, Project Catchup

CJ: Welcome to build and learn.

My name is CJ.

Colin: And I'm Colin, and we are back
again this week to talk about what

we're working on, what we're learning.

And there's a little bit of
a kind of upheaval in the

finance app space this week.

So we're going to chat about
that a little bit as well.

CJ: Yeah, I've been, I have been a
mint user for a super, super long time.

But recently too, I've been frustrated
that some of the things don't connect

to it and I haven't been able to like
manually manage those and so in my

head I'm already sort of doing math and

Colin: So yeah, what's the, the
news is that Mint is getting shut

down by Intuit January 1st, 2024.

CJ: that's so wild.

I feel like so many people use
that for not, I mean, I don't know.

The budgeting in it
honestly isn't that good.

Colin: it's not,

CJ: But, like, the, just tracking kind
of your overall financial health, like

it gives you your credit score and it
gives you an attempt at a net worth,

like, estimate based on Zillow minus your
loans or whatever for real estate, and

I know it does like a pretty good job of
tracking credit card balances sometimes,

so, uh, yeah, I don't know, do you
have, yeah, like, It sounds like you've

been using this for a long time too.

Colin: yeah, I've been a user.

I mean, I think from the very early
days, I pulled up some of the dates

just cause I was like, okay, how
long ago have I been using this for?

And, um, I think I probably,
have been using it since 2006.

So it launched in 2006.

And there's some fun things that
I want to dig into here, but

how long have you been using it?

Do you use any other tools?

CJ: It's the one I've
been using the longest.

I have also tried Personal Capital.

And I have also tried for budgeting,
we, right now we're using YNAB.

But yeah, we've tried a bunch of stuff.

Also just like Google, billions
of Google spreadsheets.

But, um,

Colin: Always a spreadsheet, right?

Yeah, I, I use Mint.

I use Credit Karma and I think
I actually had wondered because

Intuit bought both of them, right?

Credit Karma got bought.

More recently, and I was wondering why
they would buy two of these things.

And they are pushing folks towards credit
karma, which is not a budgeting app.

But you know, it kind of like
watches your credit score.

It watches if you own any, if you
own a house, if you own a car, if

you have loans, like it kind of gives
you a sense of your that high level

picture of your financial health.

But it'll be interesting to see if Intuit
pushes some sort of like credit karma

budgeting app or something in the future.

But yeah, I think it's a little
bit different too in that Mint made

money by recommending credit cards to
people which is very similar to like

NerdWallet or, or some of those sites.

CJ: Yeah, I feel like they, didn't they
also like recently do something with taxes

too, like you could, you could do your
taxes through Mint or something, or is

that just like an on ramp to TurboTax, or?

Colin: I think it's yeah, they, like,
you can have your single Intuit login

would log you into TurboTax, log you
into Mint, any of those products, and

I think be interesting to see, because
I think there's something about how the

IRS is rolling out, uh, the ability to do
taxes without TurboTax in certain states,

states without income tax, so Nevada,
Texas, things like that would be good,

CJ: there's gonna be like, yeah, those
new self file, like the IRS is gonna

provide a free website where you can go
and file your own taxes in the same way

that you would through a paid service.

Yeah, I...

Was I mean, as soon as I read about
this announcement, I was like, Oh, I

wonder if that was related that now
they're like the IRS is trying to offer.

I don't know.

It's so wild to me that the
IRS doesn't already have that.

Colin: yeah, well, and the
argument there is that Intuit has

been lobbying against the IRS.

they're being choices
for a very long time.

So yeah, I'm, this is an interesting
timing for me cause I, I'm

more of a passive user of Mint.

I actually just started using co
pilot money, which I really like.

It's very, it feels like the
linear version of all these tools.

Very like sexy looking, but also just like
has your investments, has your like debt.

And asset ratios and everything in it,
you can see your stocks, you can see your

portfolios, you can see your budgeting.

And I've never really had an
actual budgeting practice of going

through and categorizing things.

So I'm trying that out in copilot.

But like you, I have personal capital.

I never really log into it.

I have lunch money, which I really
wanted to support because it's built

by a single solo developer and it's an
awesome tool, but it really only does

the budgeting piece and I kind of want
that like bird's eye view of everything.

So

CJ: yeah.

And you in the past have had like talked
about building your own thing, right?

Is

Colin: yeah, and I think that's, that's
where it gets into like the detail,

like it's so hard to do that stuff.

Like we saw maybe.

co ran into this too.

And some of the tools that they're
like plaid and modern treasury, it's

just like looking at transactions.

especially in the world of like Venmo
and bank transfers and all this stuff.

It's like really hard to
auto categorize and do it.

You kind of have to do it by hand.

And the really interesting story.

I just want to throw out some things here.

We can chat about them before we talk
about what we're working on is that Yodlee

was the tech that ran And they, there's
a story that we can find the link to

where Yodlee was just like a contract,
you know, they had a contract with Mint.

They made like 4 million or so on that
contract before they sold to, into it.

And then there was this
owner of a domain mint.

com that they sold to Mint for equity.

And so the owner of the Mint domain
made more money than Yodlee did

on that acquisition, just because
I think it was actually like a, a

cap, like a VC that owns the mint.

com domain.

And so they took Series A stock
in exchange for the domain.

anD that turned into either a stock
and into it or actual dollars.

I'm not sure which, but and just
looking at these numbers, you know,

we talk a lot about Bootstrapping
and kind of indie co founding and

things like that on, on the show.

But they had, I guess, quote unquote,
only raised 30 million, but like at

the time when they were acquired,
there were only 35 employees.

And I think when you look at
startups raising that much

money today, you see them having
typically a lot more employees.

They're raising way more money,
which makes it really hard to get

acquired in today's market, just
because you don't have a lot of.

Acquirers who can buy things.

And it's like, once you get into
even handing like a first time

founder, 25 million, it's like, how
do you know how to spend that money?

It's crazy.

CJ: Yeah, it also is interesting to
think about, like, how Mint really didn't

have a business model, if I can remember
correctly, like, I feel like in the

beginning, I was like, how is this free?

And how, like, feeling like, oh,
they've got to figure it out eventually.

And then, After I want to say it
was like after years of using it

or something, then they started
showing like all the credit card ads

and trying to get you to buy stuff.

Maybe, maybe I just didn't notice
that when I first started using it.

And maybe that was like part of their
thing all along, but I was like, when

is this going to not be free anymore?

I just, and then the fact that
they were able to raise money.

I mean, yeah, I don't know.

It's kind of surprising,

Colin: Yeah.

I mean, they had about 140 million
valuation right before they

were acquired for 170 million.

So they, they went up in value with
that acquisition and the CEO worked it

into it for a little bit and then pieced
out and it looks like he's doing some

AI stuff these days, which is cool.

So he's, I haven't seen his name.

In years and so it's just interesting to
kind of go down that rabbit hole of like

end of an era for sure it's cool to see

CJ: Yeah, interesting.

Yeah, I guess I'm curious what
the listeners are using to

track personal finances, and
if you have apps or whatever.

I, I would love to try a bunch out if
Mint is, since Mint is disappearing, so.

Yeah, let us know.

Colin: definitely and I think we'll
have to do that money episode we

keep teasing at some point where we
get a little bit more into the tools

and tricks and things that we do.

I think that'd be fun.

CJ: Yeah.

One of the things that surprised me
about Mint was that I could connect

my like, Stripe stock account, which
is run through Solidus, I think it's

Shareworks, Solium, Solium, which is a
Morgan Stanley thing, so maybe that's

not too, too surprising, but that you
could see, The value of your private

stock in mint, which was kind of cool.

It's like being able to see as that
changes and as the valuation changes.

And

Colin: Yeah, a little bit less
so these days, like everything is

just pure red in the stock market.

CJ: yeah, but it gives
you a good idea of like.

You know, how am I tracking towards,
you know financial independence

or, you know, how much margin
do I really have in my life?

Or should I really be like buckling down
and getting, as Dave Ramsey would say,

gazelle intense about my budgeting and
like, you know, rat like, I don't know,

eating rice every night or something,
trying to thin out the, the lifestyle.

But,

Colin: yeah, well, in the similar
vein to that, an update from last

episode is that I did not drop a
bunch of money on a school bus.

So I was outbid and I can't remember.

I think I like.

Emotionally, I just had to
set a limit and walk away.

And so I put four grand down as
my limit and walked away and they

sold for four grand, but not to me.

So I think that means that somebody had
a four grand limit put in before me.

And it's okay.

Cause I would have had to figure out
how to go get the bus and drive it back.

Right now.

I would probably be doing that right now.

I'm happy to not be doing that.

So I will keep an eye out
for another school bus.

We'll see if, if I'm still interested
when another bus comes up, then I feel

like it's going to be another bit.

Cause you know, it's a good, it's
a good test of like, is this just

a fixation of mine right now?

Or is it something that
I really want to do?

And I did have to figure out like,
In another, yet another spreadsheet

do I, where do I store it?

Where do I work on it?

How and when do I find the
time to do all that stuff?

So I have some more
time to figure that out.

Can make a better informed decision, but

CJ: Nice.

Yeah.

Speaking of utility
vehicles, I've had this idea.

This is sort of like a, I don't know.

If this started when I was burnt out
when I was at Stripe or like when this

onset happened But I have this weird
desire to start like a leaf A leaf

blowing company that like comes to your
house And we'll just I don't want to

like do the leaf blowing into the pile I
just want to come over and like use the

vacuum thing that like sucks up all the
leaves into the truck and then drives

away And so it'd be like an uber like
some sort of like uber for a leaf pickup

Colin: I feel like this is a,
the next craft work service.

CJ: Yeah, it could be.

It definitely is like a
Kraftwerk add on for sure.

Colin: Mike, Mike's already
got you thinking about the

future, future house things.

CJ: yeah, I mean, I, the It crossed my
mind that a school bus would be a good

solution for this because you can get
like a towable leaf vacuum for like

two grand and then you just like open
up the back window and you have it all

blow in there and Then you have some
system for I don't know compacting and

Colin: You just make like a really, or
like if you've seen a wood shops, they

have the really big dust collectors.

Right.

And so you just fill up the
entire school bus with leaves.

CJ: Yes, exactly.

Yeah.

And

Colin: Very brandable.

CJ: yeah, like, Oh, what's
the, what is the by product?

How could we sell like the leaves
somehow as, you know, maybe somehow

reprocess them into pellets that then
go into pellet stoves or reprocess

them into some sort of like mulch.

Yeah.

Compost.

Yeah.

Some other things.

So

Colin: I'd be a stinky bus,

CJ: Right.

Those like feed the bus or those, yeah.

Like

Colin: biodiesel for the bus

CJ: Yeah.

Yeah.

So anyways, nerding out about whatever.

Colin: Anything we can do to get
away from a computer these days.

I think,

CJ: Yep.

Yep.

And also it, I only
think about it in fall.

Like right now we are literally making
several dump runs per week with the truck

full of leaves because I don't know.

In New England there's,
there are a lot of leaves.

Colin: definitely.

All right.

Well, there's, there's
the school bus update.

We'll get back into it.

Maybe you'll have a school bus one day.

We'll just have a couple of schoolies.

But what are you what are you working
on in front of the computer this

CJ: Yeah, we got a couple things going on.

At Kraftwerk, we're still building
out this comm center for customer

customer facing comms, and I've
been integrating with Twilio Flex.

Which is like this drop in UI that
lets you handle incoming phone calls.

You can make outgoing phone
calls with a little dial pad.

You can do incoming texts, outgoing
texts, and then also like a web

chat widget all through their UI.

And when I was testing it out and sort
of validating this idea, I was like,

Oh yeah, this is going to work great.

And then we don't have to build all
these different things ourselves.

And at least for the desktop
experience where agents can like

mark themselves as available and then
they can have these inbound support

or sales requests routed to them.

I thought this is a pretty sweet
way to avoid a bunch of dev work.

And so I went to go implement it for
realsies and realized that in the dev

experience, you can easily log into
the console as an admin just by like

offing into the Twilio admin interface.

But in order to log in as like agents
in real life, I wanted to have a

single sign on experience so they
didn't have to like go to Twilio

and create yet another account.

And so I started learning
SAML, which is like.

This SSO protocol that you can implement.

It's what, you know, if you are not
logging in with OAuth, but you're logging

in with SSO, meaning you just open
some corporate website that is probably

somehow tangentially related to your big
corporate company and it automatically

logs you in, you don't have to put in
anything like not even your like Google.

Like.

Email and password, then that's
probably happening through SAML.

It turns out there are a couple of
different libraries in Rails for

setting up your Rails application
as either a SAML identity provider

or a SAML service provider.

And so in this case, Twilio
is the service provider.

So we're implementing like the
identity provider side of that,

which is, it's interesting.

There's like a lot of new off concepts
that I'm trying to pick up and learn.

And all these different things
about XML attestations and how you

sign this thing or sign that thing.

And there's instead of one URL, there's
like 10 different or more like five

Colin: Sounds like
OAuth to, to nth degree.

CJ: Yeah, it's sort of like OAuth.

If it's like, I don't know, I've
been trying to think of it as

like OAuth and OpenID and JWT

Colin: Well, and you have
like a directory usually too.

So it's like with OAuth, it's like, oh,
anyone with a Twitter account can OAuth.

But now it's like, no, if you're not
in this directory or LDAP or whatever

it is, you're not allowed to, to OAuth.

CJ: exactly.

And then it's also like,
what roles do they have?

And like, what are all these other
like, really particular details

that the service provider wants.

So on the identity provider side,
you're supposed to like, try to be

as flexible as you need to be to
work with all the service providers.

By implementing like attributes
that they all require.

So maybe some of them need an
email and other ones need a user

ID and other ones need you know,
their title at the company or the

department they work in or whatever.

So yeah, kind of digging through
that, but that's been interesting

to learn about for sure.

So.

Colin: Yeah, we use, I think a lot
of startups and companies use Okta.

I don't know if Okta is like, is
it technically SAML under the hood?

It's probably a whole bunch of
different flavors of different things.

CJ: Yeah, I think Okta, like Okta and
Auth0 and a lot of these folks provide,

like they are both IDP and service
provider and like an OAuth platform

Colin: They're a bridge from
OAuth to the providers for SAML.

Yeah.

CJ: yeah.

So Okta, they do have like tons of
documentation though for testing and

learning about SAML, which is pretty cool.

So, uh, anyways, I didn't find
any content on YouTube about this.

So this is like, as soon as I
figure this shit out, there's

going to be some content because

Colin: for you.

Yeah.

CJ: using, yeah.

So.

Working on that and yeah, what,
what, what's going on with

the, the conference room thing?

How's the conference room
booking software going for

Colin: Yeah.

It's going good.

I started, I just had like a lightning
bolt the other night in my brain where

I was like, I just need to work on this.

And so I'm in that accountability group
and we've got two more meetings, I think.

So I got.

A lot of the Google Calendar stuff
working in React and Node, and I used

Tailwind to do a little UI for it
and I'm not doing, I'm doing like the

iPad screen right now, I've just like,
this room's either green or red, it's

available or not available, and then
you can tap on a button that will,

Anonymous, like if you tap on it, it'll
just book it for whoever tapped on it.

There's no sense of who
logged in and did it.

We don't really need to know that.

But we want to have the ability
for like you to log in and book

a room for tomorrow or for me to
log in and book one for next week.

And I'm starting to like, look
at this node and react app.

And I'm like, I do not want to
do all this stuff from scratch.

And so I think I'm going
to move to jumpstart.

Move it to rails because I don't
want to build all that stuff.

And then I think I was going to use
react native, but I think I'd rather

use turbo native and just play around
with that to be able to handle the iPad.

And then eventually like Android or
whatever, if someone really wanted

that, we have the iPads already.

So it would just be,
you know, an iPad app.

I was planning on just doing
it as a website for now too.

Like it doesn't need to
be fully native though.

The Robin one has some cool hooks
where it knows when our hours are

and on the weekends and stuff,
it actually dims the iPads.

And so it hooks into the actual stuff.

So I'm not sure if TurboNative
gives us that, or if we have to

like create a function in, in, you
know, Swift or Objective C that

TurboNative can call to do that stuff.

Those are obviously like scope creep.

To hell for sure.

Like I don't need to have
auto dimming iPads for this.

So my goal for my presentation, my show
and tell will be just that the iPad

app turns from green to red when people
have, when the room is actually booked.

And there's like a little bit of
admin, like you need to be able

to create a few conference rooms
and assign iPads to those rooms.

And then that way each
iPad knows which one it is.

So that it knows like,
okay, this is downstairs.

This is the calendar that's associated.

So you need to have the OAuth,
like an admin OAuth in and

give access to the calendars.

And so you have this room,
this calendar, this iPad.

CJ: Mm hmm.

Sweet.

Colin: yeah, that's kind of
where we're at right now.

And we get the, we get to
have the obligatory Chris

Oliver shout out that way.

We get to mention Jumpstart.

CJ: Yeah.

Yep.

Totally.

Yeah, I jumpstart, I don't know, I've
started leaning on it a ton for things.

So it, yeah, I, I'm, I'm loving it.

Colin: If I stick the node way, it's like,
okay, do I want to go figure out which ORM

I'm going to use and build out my models?

And no, I just want.

To have user logins and all of that.

CJ: right.

Yeah, I think the closest I've seen to
something that was productive and kind

of like got you started on the JavaScript
side was remix, like the remix experience

where they, and they have these things
called remix stacks where I think

Kent C Dodds maybe has like a repo or
something, or maybe it's on the remix

repo where you can say like, Oh, I want
to create a new remix app from this stack.

And it will have like auth
already built for you.

And it will tell you this, you're
going to use Prisma and you're going

to use fly to deploy and you're
going to use whatever, and then

you kind of just follow that path.

But even with that sort of kickstart,
I still didn't feel like it was as

productive as you can be with Rails.

And so

Colin: It's tricky because if, if all
that stuff's been done for you, you need

to figure out how they did it for you.

Right.

And in rails, it's like, okay, I
know these patterns and I know which

drawers to look in and JavaScript.

You're like, Oh, that's
an opinion way to do this.

And I guess I'm going to now
have this opinion if I don't want

to swim upstream against all of
the things that they recommend.

So.

CJ: Right.

And it's right now, I don't think remix
is like the number one most popular thing.

And so who knows if that will end up
being supported long term and, or is next

JS going to just continue to steamroll
everything that happens over there.

And

Colin: Right.

Well, and I don't really get to write
Ruby at my current job, so it'll

be good to get back into that world
just to stay connected to Rails and

stay connected to Ruby because I am,
I'm eyeing RailsConf for next year.

Just as a, probably as an attendee, maybe
I'll think of some talks to submit, but

I'm already, I think two for two on the
last two RailsConfs for doing a talk.

So,

CJ: nice, nice.

Yeah.

Well, yeah, I don't know.

It just, it also just, yeah, will
make you happier if you get to write

Colin: yeah, totally.

How's the story feel going?

CJ: Oh yeah.

So I don't

Colin: know if we, I don't
know if we used the name last

CJ: Yeah, we didn't
use the name last time.

So story fuel is a little
side project I'm hacking on.

The idea in the beginning was, Oh,
I want to give I'm going to author

a story, a fiction story, and I want
to give it just like a little bit of

a, you know, genre audience, short
description, and then have it spit out

like a really good starting point for me.

And then use that to go do stuff.

And what it's sort of morphed into is that
I made it so that you can give it those

things and it uses a bunch of open AI
endpoints and it will write a story and

you can sort of, uh, guide it a little
bit, but it will write a story for you.

And then I'm using 11 labs.

So I'll convert the story into audio
and then I'm using FFmpeg to like

combine and concatenate all these
audios into like one long audio.

And then I use Descript and images from
Dali to cut like a little video together.

So it's like it's kind of
like a little audio book with.

The captions kind of flashing at you the
descript way and I I'm going to start

publishing all of the episodes on, uh,
on like a podcast story fuel podcast.

So I think the direction it's
heading is that you'll be able

to dear listener head over to.

Some website eventually and suggest a
story and then it may or may not get

picked up by story fuel and converted into
something that you can listen to or watch

Colin: of a cool interactive way, right?

If you're subscribed to StoryFuel
podcast and you submitted, like you

have this, you know, creative community
of submitting stories and then like

yours gets turned into a story and
shows up on the list and that's cool.

Are you going to have a three, three
books per day limit like Amazon?

CJ: So I, what I found was that it
doesn't generate enough content for it

to be like an audio book or like a, an
actual book that you could like, yeah.

So you still would need to
like heavily, heavily edit it.

Right now it, it comes out to like
between 20 and 30 minutes of audio.

Which is like perfect
for a podcast, right?

Like you want a podcast
that's kind of bite sized.

And so what I've been doing lately
is sort of asking it to tell me

stories that I'm interested in.

So I'm like, tell me about a dystopian
future where AI is taking over and

like some hacker has to save the
world or whatever, and then it spits

out a bunch of stuff that is, I
think is like kind of interesting.

And so Yeah, it also like
will develop the characters.

So like one by one, it'll go
through and add character traits

and detail and all this stuff.

And then I use those as input into
the Dolly output in generation.

So I'm like, Oh my gosh, like
that's totally this character.

You know, that's Nikki right there.

Like, look at that.

That's a wild

Colin: What'd you think of 11 labs,

CJ: Oh my gosh, it's really good.

I've been, so 11 labs.

Takes text and converts
it into audio for you.

Now, there are a lot of text to
audio tools that you can use, but

what 11 labs has surprised me with
is that depending on the context.

Of the, of the text, it will use different
emphasis and it's not the same every

time and it's definitely like kind of,
I don't know if stable diffusion, it's

like some sort of thing that they're
trying to figure out, like, how would

this reader read this sentence in
this context, not just like, how would

they say this specific word or phrase?

And so it comes out a lot less
robotic, I would say, and they

have in the same way that.

If you've played around with any of
like the GPT playground or models, how

you can control the temperature, which
makes it like more or less creative or

whatever, or like kind of more or less
off the beaten path, you can control this

like stability and some other parameters
that go into the 11 labs prompt, and

then it'll come out like more or less
sort of emphasis on its own sort of

accents and syllables and things yeah.

So, yeah, it's been pretty sick.

It's also been really fun to
learn ffmpeg, like more of ffmpeg.

It's such a, that is such a
powerful little piece of software.

It's just mind blowing
what you can do with it.

So my next little task that I have
in mind for that is like, take the

images that were generated from dolly.

And use FFmpeg to cut a video instead
of having to do it manually in Descript.

sO the idea is like, oh, it's going
to generate the story arc and all of

the chapters of the story and all the
content for the chapters and all the

characters and all the places and then
it'll generate images for the characters

and the places and it'll keep track of
their memories and then it'll, you know,

generate this audio and then ultimately
cut it all together and splice it

into something that I want to watch.

And the pipe dream and what I've been
telling people is that like, in the

future, I don't think people are going
to be able to sell like generated content

that is like, I'm not gonna generate
like a book and sell it on Amazon.

Instead, you'll build a platform where
someone comes and says, I want to hear

a story or see a movie about X, and
that is going to be generated for you.

Like in real time that and it's going
to be one for one, you experience it

one time and then it's gone forever.

And it's right around the corner.

And the fact that I'm just little old me
hacking around, like a couple of tools

on the weekend and able to put together
enough that like, You can get pretty

good fidelity or like pretty interesting
stories just out of whatever open AI.

So

Colin: Yeah, there's, there's this big
wave of, I think Cory Doctorow's book,

and he's making waves right now on
the inshittification of the internet.

And what I'm really curious to see is
whether or not, like, I'm not sure if you,

have you played around with asking it to
do stories that are based on a theme of

an existing story or anything like that?

CJ: I, I haven't just because I
want to be able to like publish

them and I don't want it to.

Colin: Be like a new hope.

CJ: Yeah, exactly.

My kids are my kids.

On the other hand, when we're playing
around with it, they're like, do a one

that's yeah, do one that's based on on
Star Wars or like make the character

based on Han Solo or do one about Pokemon.

I'm like, no, like, I don't
want to take anyone's IP.

Like, I want it to be just some generic
storyline that will be interesting.

And I want new characters that I've
never seen before and whatever.

But,

Colin: But the, the thing, the thing
there is when you're talking to it

about making stories, like, it is
basing it on what it knows about.

Right.

And so over time, do we get this
general, like, reversion to the mean

story, right, where it's like, it's
now being trained on other AI stories,

and it's garbage in garbage out.

And I think that this idea
of prompt engineering or

storytelling is pretty powerful.

Because if you can Okay.

Have an idea for a story and then use,
like when you think of like the Iron

Man movies where he's kind of like
riffing and having Jarvis pull in things.

But he's still very much in the cockpit
and very much pulling it all together.

And it's pretty powerful because I
think there's still gonna be that

storytelling element and still having
that un or we end up with a bunch of

things that end up being the same.

So we'll see how that pans out.

CJ: totally.

So I reached out to Sam Bafa from that
like works in the Reno Collective about

his choose your own adventure platform.

I don't know, I haven't talked
to him in years, or and I

don't have any updates on that.

But I was like, Oh, another way that this
could go is, as you're experiencing the

story, whether it's in you're reading
the written word, or you're listening,

It would be sick if like you get to some
point and you're like, actually, I don't

like how this is going, just mix it up.

And like from that point
forward, the story is different.

You know, like it can just like pick a
different conflict or pick a different

Colin: Or choose your
own adventure, right?

CJ: Exactly.

It is literally

Colin: you a

CJ: own adventure.

Yes, yes.

And that's a, that's like totally another
thing too is like when you, the way that

it works right now is you give it this,
you give it those like little pieces or

whatever, and then it generates a story
arc and some characters, and you have an

opportunity to like edit the characters
or edit the story arc at that point.

And then you say, okay, now generate
like a list of chapters that

might happen within the story arc.

And so then you can edit those
chapters a little bit if you want.

And then you say, generate, now generate
the content for each of those chapters.

And then you, if you want to do like
the painstaking work of going in and

editing all the content, you can do that.

But yeah, so one, one thing is like,
do a choose your own adventure version.

The other thing was I, I thought about
doing like a rap genius, like a rap

genius style thing where like, In order
to edit it, you sort of select a chunk

of the text that you want to change and
then give it a prompt that's like, make

this more dialogue or, you know, you
know, make this an alliteration or change

this into like a joke or something.

I don't know.

Colin: Yeah, did you, did you ever watch
the Black Mirror Bandersnatch movie?

It's,

CJ: no, I haven't.

Colin: it's a, it's a, it's not
AI that I know of, but it lets you

choose options as you're watching it.

So like the, the show, you're watching
the show or the movie and it stops and

asks you like which one you would do.

CJ: okay.

Okay.

Colin: is different based on it,
but it's, it's not AI generated.

And I think there's like a certain
number of possible outcomes

and story branches and stuff.

So it's like playing a video game
or like you know, Knights of the

Old Republic style type thing.

CJ: Yeah, exactly.

And in fact, like when I was thinking
about doing the Choose Your Own Adventure

version of this, My first thought was
to use Dragon Ruby and like make it an

interactive thing where you're kind of
like, I don't know, maybe you become

the protagonist and you have your little
avatar that was generated by Dolly and

you're just standing in a dark room
type situation and it's reading off the

text and then you can be like, no, I
don't want to, or like, you know, more

like, or kind of like a one player D&
D situation where you're kind of like,

Oh, I want to, I want this to happen.

And you can type out, like, I want this to
happen and then boom, the story changes.

But what's interesting is like, I want
to keep all of those characters memories.

So I'm storing like a bunch of
contexts in different tables in

this giant Postgres database.

And the idea is that I can
use sort of use embeddings.

With each of the memories.

So then I can do like a vector
search on all the memories and

then embed the relative memories
into the context when generating

like the next section of the story.

So I don't know.

Colin: Wow.

CJ: It's fun.

It's like super, super nerdy.

Like,

Colin: I was going to say, I need to
sit down and learn more about embeddings

and vectors and all this stuff.

It's, that's pretty cool.

CJ: it's fun.

It's super fun.

And the tools are getting, I don't
know, I feel like For a long time.

I tried really hard.

I was like, what is it?

All this machine learning stuff.

I really want to learn
it and I don't get it.

And a lot of the content I found
was like how to implement and

train your own models and whatever.

And like, it was way over my head and
it still is like all the mathy, like

how the models actually work just.

Escaped me.

And to this day, if I watch anything
about training or, you know, VLLM or

like the, the new high speed stuff that
makes it faster or whatever, I'm still

just like, how do people figure this out?

I don't get it.

But when I'm using it and applying it
like in actual applications, I'm like,

holy smokes, like this is, this is fun.

Like, it's pretty cool.

So I don't know.

It's

Colin: sounds like something
you're working on and learning.

It's, it's a

CJ: yes.

Yeah.

Colin: we're always learning.

CJ: Yeah.

Always learning.

Always learning.

Speaking of learning.

Yeah.

, you're doing some deep stuff
over there with is that audio?

Mm-Hmm.

Colin: it doesn't feel too
deep when it doesn't work.

But kind of shifting
back to my actual job.

We've been trying to review our PRs.

We have like our, our docs are public and
so we take PRs to the docs and I'm trying

to close out some PRs from 2022 so that
we can get a clean slate of 2023 PRs.

We're down to like less than
40 PRs open, so we're going

to try to get through those.

But we, one of them was
around this voice connections.

And so if you've used Discord,
you know, you can And those bots

can easily send texts like texts.

We can test all day long.

I can look at it and be like,
yeah, that coach would work.

But when it comes to a bot joining, so
a bot can join a voice channel and you

can do all kinds of stuff with that.

You can have it play a sound, you can
have it do all sorts of things like that.

You can think of like a lot of the zoom
apps and bots and Slack apps and things

like that out there is that you can have.

You know a soundboard for instance,
which is also a feature of discord

now, but before we had that, you
could do it as a bot or an app.

And so I couldn't figure out whether or
not I could even merge the PR without

sitting down and actually doing this.

And it's pretty easy to do if you use.

Like one of the third party libraries,
because a lot of the libraries have built

around all the basic tenets of this.

But when I try to build our bots, I try
to use just pure JavaScript or pure Ruby

or pure Python, whatever it might be.

So that I really understand
like what's happening.

Cause in our docs, we're not documenting
libraries, we're documenting.

The APIs that the libraries are using.

And so I've been working on sending and
receiving audio over like through discord.

And so what I found is it's very hard to
debug because you're working with packets

and buffers and like you can't just
console log or use a debugger and say like

when you've chopped up audio files into
tiny pieces, you can't listen back to it.

And know, like, is it right?

And when you send it to discord,
you have to also encrypt it.

And so once it's encrypted,
I'm like, I have no idea.

And I was able to, like, I started looking
around to see if there's like anything

around debugging UDP and audio like
tools, and I didn't really find anything.

There's a dev tool company
called deep gram that does.

Text to speech.

And I think both the other
way and they accept UDP stuff.

So I was starting to play with their,
like, look at their docs and like play

around with their tools, because I was
like, if I can get my audio to show

up inside of their tool and translate
it or like caption it, then I know

it works and I know my code's working
because with this court, I'm sending it.

And I just don't even know if
it's just not encrypted, right.

Or what's happening, but
I did set up like a local.

UDP like echo server, and I was able
to unencrypted send a little Zelda

sound effect from my code to that
server and have it output as an mp3.

So like I know that part is
all working and so I'm probably

getting tripped up on encryption.

And I'll probably bug some engineers next
week about like, tell me what's wrong with

my code, do a little code review, but.

CJ: Is it So encryption is required or?

Mm-Hmm.

Colin: So there's this
handshake that happens.

So you have to connect
to our WebSocket gateway.

And then from that WebSocket gateway,
you then establish, like when you join a

voice channel, you establish a UDP socket.

And so you're managing all these
states and sockets and reconnects

and heartbeats and all this stuff.

And so I was able to recreate all
of that, which is good because

I've never done any of that before
in our, in Discord specifically.

And so just knowing like how the handshake
works and what events are sent and

being able to see it so that when other
people open other PRs, we can actually

like have this little playground to
verify, like, that's how it works.

That's how it's working as
documented or no, that's right.

It's, it's broken or
whatever that might be.

But in that handshake, we give you a
key to use for, and different types of.

Like hash like modes so that you have,
I think there's three different ciphers

that it could be and then a type of key.

And so you use that to send along in
your headers and you do that encryption

on your end so that way you're not just
sending through on encrypted voice.

CJ: got it.

Dang.

That sounds interesting.

Colin: Yeah.

CJ: Yeah.

Are

Colin: ideally, I mean, I don't know if
I'm, I'm hoping we can get it done so that

like maybe we have our own voice SDK or
something at some point, because sometimes

you just want the voice SDK or it's like
to have to like dig into this low level

stuff, um, doesn't make a lot of sense.

And maybe an SDK from us can also
be used as a reference for people

to implement their own SDKs or
whatever that might look like.

So,

CJ: Yeah.

Are you writing it with Ruby or

Colin: It's a JavaScript.

CJ: JavaScript?

Okay, cool.

Yeah, the gosh, just working with
any kind of sockets is yeah, it's

always a little bit harder than
I expect once I first get into

Colin: Yeah.

Well, and you're getting into also
like moving into Opus streams and

Aug streams and all these different
like audio formats that we don't

necessarily use on our computers.

And so then I have to like export it to
either Aug or mp3 so I can listen to it.

And then I could, what was nice is
that I'll be like, okay, export this.

And it's like, yeah, this file's corrupt.

You're like, okay, something's not right.

CJ: Right.

Yeah.

Interesting.

Yeah.

One thing that was kind of Tricky
about the 11 labs integration, not

necessarily tricky, but like, eh, I
don't think they have a Ruby library.

And so they just, you give them like an
HTTP request with text, and then they

just dump back like the audio in binary.

And so, or whatever, like it
comes back as just like, you

know, unreadable blob of whatever.

And so then I was like, I don't know,
let's see if we write this to a file.

Like if that ends up working and
it just magically was like, cool.

And then I was like, okay, now
let's upload it to active storage.

Boom.

It just kind of like that part.

I was surprised at how, how sort of
a straightforward that part was, but,

Colin: I mean, You're not far from
being able to take that story, getting

the binary, and then you can broadcast
it out to ice cast or something.

And now you've got live stream.

CJ: yeah, gosh, that's interesting.

Oh man.

Okay.

Yeah.

But yeah, along that path, I
was, I was similarly like, how

the hell do you debug this?

Right?

Like you're getting back something
and like, you just have to know

that, oh, that's an MP3 file
that I need to store somewhere.

Colin: I have been using chat GPT
to try to debug some of this stuff.

And I mean, it was kind of cool how
normally we're used to getting texts

back and there's these things called RTP
headers that you have to add to the front

of your packets when you send the audio.

Like, it, like created this like visual
representation of what an RTP header was.

And I, I was mostly using it to be
like, okay, I don't understand what this

header's for, what is it comprised of?

You're setting a lot of
different bits and things.

And so I was like, explain this to me.

Okay, explain it to me like I'm 10.

Explain it to me like I've
never written code before.

And it was pretty useful for that.

You know, you still gotta check your
homework and say like, 'cause half

the time it's like, oh, you need this.

And then I'm like, the docs
say we don't need this.

And it's like, oh, I'm sorry.

Right.

And thankfully these docs, I
think, were, have not changed

since before the cutoff window.

Which I think we talked about on
the show where the cutoff window was

gone and I still see a cutoff window.

So I don't know how that works to you.

CJ: I think you have to enable Bing
search maybe or something like in, in

like the pro GPT four dropdown, there's
like four options now or something or they

have, I don't know if you have to enable
them or plug plugins in or something,

but there's like the Bing search thing.

I think that's part of it, but they
also have like different models that

have cutoffs at different dates.

But man, I would say over the last
two weeks, it's been less valuable.

Like, I feel like the quality of
the responses I've gotten over the

last, definitely the last week is
way lower than it was before that.

And I don't know if they changed
something, but I'm like, this response

that you just said makes no sense at all.

And it is not going to be,

Colin: of the internet.

We should see if Cory Doctorow
wants to come on our show.

CJ: yeah, totally.

Colin: If anyone knows Cory, hit us up.

We should reach out.

I mean, he's on a podcast tour
right now talking about this

and it's top of mind for sure.

CJ: let's do it.

Should we wrap there?

Colin: I think that's, A
full featured podcast.

CJ: There you go.

Colin: This was not generated
automatically by Eleven Labs.

This is, these are, we confirm humanity.

CJ: Yeah, that's this is our watermark of
H generated, not C generated or something.

Colin: all of our, all of our
quips and, and weird vocal tics

and all, all, you get it all.

So,

CJ: Yes.

Yes.

All right.

As always, you can head
over to buildandlearn.

dev and check out the show notes
that'll have links to all the

resources and things we chatted about.

Colin: we'll see you next time.

CJ: See you next time.

Bye friends.