Keith Casey: Webhooks & webhooks.fyi

Colin: Welcome to Build and Learn.

My name is Colin.

CJ: And I'm CJ and today
we're joined by Keith Casey.

Keith is a product expert at ngrok and
has also worked at Okta and Twilio.

He's all about getting great tech
into the hands of awesome people,

and he lives in Austin and is big
participant in the Austin Tech community.

He also shares thoughts on
his blog @ caseysoftware.com.

He's also co-authored a
book about API design.

And so we're super excited
to have Keith here.

And, we're gonna talk about
webhooks today and webhooks.fyi.

If you haven't seen it, head over
and take a look at this knowledge

base that, ngrok has put together.

Before we get too deep into it, just
wanted to welcome you, Keith, and,

in case I said anything wrong there,
please, feel free to correct the record.

Keith: No, that was great.

That, that Keith Casey
sounds really impressive.

I'm not sure how it got
to be about me, but, no.

thanks for having me here, guys.

looking forward to this chat.

Colin: As we've been getting into, like
looking at who we want to have on the

show, it seems like the theme so far has
been finding people from our past and

putting them in front of a microphone and.

Keith is no exception.

There, sounds like inroads from
everything from coworking to Techstars

to startups that we've used, I think
you were dev at Twilio when we like

met in person for the first time.

so yeah, getting into some
hackathons and things.

Keith: And I think that's really cool
because the people I was meeting 12

years ago with Twilio, that's when I
was there way back in the ancient days.

The people that were there that were
passionate even that they're, even

if they're the most junior first
day on the job, developers after

that, Applying that passion for
6, 8, 10, 12 years in this case.

They've moved on to some
pretty cool, impressive roles.

I look around and I'm like, I remember
when this guy was writing his first lines

of code and now he's C T O over here.

Or I remember when this person just moved
to town and now they're a significant

like impactful community leader.

And it's awesome to see like friends
and allies grow up like that and move

into their own and see how that passion
has affected the world around them.

CJ: Totally.

Like just in those 12 years, someone
going from writing the first line of

code to being a cto O as a microcosm
tech in itself lets you see a co a full

career progression over the course of
12 years versus other industries where

it might take you 40 years or something
to kinda climb the corporate ladder.

Keith: Oh yeah.

CJ: fun to.

Keith: when I was first in the field
or Twilio way back in the day, we

had to tell people what an API was.

We had to teach people like
building on APIs was okay.

AWS had been around for a couple years,
but not many people were using it at

scale, When we think about how fast
people's careers have progressed, we

have to think, that's because part
of that is the technology under us

has moved so quickly and in such
radically different directions.

If you told us 20 years ago, no, most
people won't have their own servers.

They'll be living entirely in
this thing called the cloud.

And the cloud is really
just other people's servers.

But there'll be APIs connecting in
all these third parties, like really

defining and making your app work.

There's people that would've
said, wow, that's cool.

And there's other people
that would've been terrifi.

The correct answer.

Somewhere in between those, but that's how
see why our careers have progressed the

way they have, is that everything under us
has moved so quickly in interesting ways.

It's really powerful to see.

CJ: Yeah.

I think that's also A fun lens to look
at all of these AI changes through that.

okay, this is an inflection point, and
if you are feeling stuck in your ways

and you're not open to considering new
ways of doing things or being agile

and flexible about what this next year.

Is going to look like things
are going to get wild.

I don't know.

I think Colin, you were showing me
some stuff with the GitHub co-pilot

brushes or whatever the other day
and I started playing around with

it and I was really jealous having
spent spending most of my time in Vim

man, these co-pilot brushes in VS.

Coat are just blowing my mind right now.

so yeah, it'll be an interesting next
decade to see all of the progress that

happens within tech and our careers and
those folks that are getting into AI for.

Colin: Yeah.

And that's a great segue to
talk about what we're talking

about today, which is webhooks.

I was actually digging into kind of what
made webhooks popular, and you might

be able to shed some light on this too.

the idea that when you're at
Twilio, you weren't just trying

to get people to use Twilio's api,
you were trying to teach them how.

why they even wanted to use an API in
the first place, or what an API even is.

and just to set the stage, we found like
the earliest mentions of webhooks was from

Jeff Lindsay, all the way back in 2006.

And the idea that thinking of a world
today where webhooks are available on

almost every app, but there was a point
at which there, there were no webhooks.

And there were no APIs.

today we're gonna be talking
about webhooks and specifically

this webhooks.fyi site that
you guys have built at ngrok.

what have you guys seen as far
as like the rise of webhooks?

Popularization of webhooks?

Keith: Yeah, absolutely.

So I mean you call it, Jeff Lindsay
was one of the first people to

put the word together a webhooks.

And at that point it
was actually two words.

just a bit of trivia for you there.

but in 2006 and seven, he was
out there talking about this.

And fundamentally he went back to
the, some of the earliest principles

in object-oriented design and
the sort of the encapsulation

model of object oriented design.

Cause remember, when you're building
object, To rewind back to, CS 1 0 1

and you've got this object, you really
don't want to expose the internals

of how that works because now you
get tied into the specifics, the,

you get into all the idiosyncrasies.

You now have to worry about
that state of the object.

There's a concept called message passing.

And message passing is just the
state of this thing has changed.

Here's how it's.

that's it.

That's, that goes all the way back to,
small talk in the sixties and seventies,

and it's a exceptionally powerful concept.

Now when you're talking about
individual components within a

system, that works out really well.

When you then say, okay, we're gonna
take those components instead of

operating the same memory space where
we could just pass things over a bus

or a port or something like that.

Let's put them thousands of miles.

In different architectures, in
different data centers, with all

kinds of things in between them, we
now have to figure out how in the

world do they do message passing?

And it turns out webhooks
was the model to do that.

And simply it's, it's JSON
over HTTP or XML over HTTP.

It conceptually, when you realize,
oh wait a minute, it's this concept

that's now close to 60 years.

just updated for the tech
stack we're working on.

It's cool to think about, like we've never
gotten away from those core principles.

We've just applied them to the
medium that we're working in, and

He, Jeff Lindsay was started talking
about webhooks2006 and seven.

he joined a little company called
Twilio not too long after that.

And basically Jeff Lawson, the CEO
of Twilio, went to Jeff Lindsay

and said, I like what you're doing.

We have some problems that
I think that's a solution.

Let's blend this.

Let's pull this in.

And so I attribute a lot of
the early webhooks, one to

Jeff and the other to Jeff.

So Jeff Lindsay brought to Jeff Lawson
and they made it work in Twilio and.

Frankly we are blessed at Twilio.

We were one of the first people into
that space to do it at scale and we were

admired for a lot of different things.

And one of the things was the model behind
those webhooks, cuz fundamentally it's

this thing, this text message, this phone
call, this recording has changed state.

Being able to pass that message across
a wire, regardless of your tech stack

ended up being exceptionally powerful.

Then when you take the macro trend of, uh,
AWS, of GCP, of all these things coming

in the market, and you go, wait a minute,
my tech stack is no longer a stack.

It's a bunch of stacks sitting in
a bunch of different places, and

oh, by the way, it has dependencies
on apps that I'm building.

Has dependencies on apps
that you are building, has

dependencies on third parties.

WebBook is the only way
to do that reasonably.

There are other approaches, but when
you say, Hey, look, I don't care

about the tech stack, I don't care
about your architecture, I don't

care about anything else other than
are you speaking JSON over HTTP.

That makes life really easy.

CJ: Totally.

And I think also like when you think about
a normal web framework and you're building

out these typical features, whether it's,
the classic, you're just taking the post

body from a form and submitting that to.

Some endpoint, or if you're taking
the post body of that form and using

JavaScript with, fetch or Ajax or
whatever to submit that as j s o to a

server endpoint, you can build your web
application and your web hook handler

using all of the same tools under the
hood, which I think is also pretty

killer because yes, there are these
other solutions that, you might be using.

some message queuing system or some
other sidecar thing that happens to

reach into your database, but it becomes
much, much easier to reason about, in my

opinion, your web hook handlers if they're
following the same patterns that your, the

rest of your web application is following.

And yeah, I, it's, I don't know,
it's it's interesting to see how.

the trend in industry has landed
on or decided that JSON over

HTTP is the end all be all.

And I think it's a lot of it is
because like we don't really have

another pattern that has emerged for
just typical web development that is

significantly better and, something
that we might all want to migrate to.

and so yeah, at Stripe we.

Looked very closely at our web hook
solution and thought about oh, what

if we changed it to all of these
other different potential things?

Then, would people still adopt it?

It's okay, 99.9% of people building on
the web or building mobile applications

or whatever, are going to still use
webhooks because of the way that they're

set up and the way that they work.

Keith: Yeah, all of us know, and
we have a favorite HTTP tool.

All of us know we have
a favorite JSON tool.

When that's the baseline you're
starting from, it's hard to get

people to move to anything else.

and I appreciate that.

I appreciate that.

It's hard right now because we can
work with tools we're familiar with.

We don't have to introduce new
patterns, new technologies or god

forbid, a vendor specific protocol
that's we're done with those hopefully.

Colin: Speaking of tools, can you give
us kind of the high level of what ngrok

is trying to solve as a company, and
as the kinds of tools that you guys

are working on, and what inspired the
creation of webhooks.fyi, what were you

guys seeing as far as these simple tools
being used in really powerful ways?

Keith: Yeah, absolutely.

Ngrok, founder and CEO is Alan Shreeve.

He was employee number
six or seven at Twilio.

If you send an SMS using Twilio, you
probably used his code at some point.

But while he is working with those
webhooks, way back in the day, he ran

into this very simple thing of when
you pull out his phone and send a text.

he'd wanna route that to the
app he was building locally.

And there are a couple other
alternatives at that point.

And he said, you know what?

I wanna learn, go and I
wanna figure this out.

So he wrote the first version
of ngrok basically to do

that, to scratch his own itch.

And then he started adding things
like, request capture and replay to be

able to say, I've got this web hook.

It came in successfully instead of
having to pick up my phone again.

Let me just replay that.

. Awesome.

That was really cool.

And most people find ngrok through
some of those webhooks because if you

look at any web hook guide out there,
odds are step one of integrating

with stripes webhooksis probably
go install inro, which is awesome.

fundamentally what ngrok is,
it's a platform for providing

secure connectivity into any.

So today we're talking about
webhooksand local development.

It can also be IOT devices
on a customer site.

It can be a Kubernetes cluster
sitting inside your data center.

it could be the deep in the bowels of
that ancient D data center, getting

access to the database behind the scenes.

CJ: we're assuming that you, dear
your listener, know what a web hook is

Keith: Oh, yeah.

CJ: we'll just gloss over.

I think it's fine if we gloss over
it because we want people to go

read webhooks.fyi, which is gonna
teach you all about webhooksand

what they are and and how they work.

But I think it's worth at least
explaining the fundamental issue with

like why that request was not delivered
to the local running application and

Keith: Yeah.

Yeah.

So when you get, when you're working
with Stripe, Twilio, slack, whatever,

when the change in state happens,
a payment being processed as text

message being received, a new slack
message coming across, that web hook,

that payload has to go somewhere.

So the web hook is just telling you
the change in state has happened.

Here's some information.

And so when you're running in your local
environment, there's DNS issues, there

are SSL issues, there's all these things
to get from your local environment.

getting from your local
environment out is super easy.

that's what our browsers do every day.

Getting in is a horrifying complexity
of dns, of ssl, of na traversal,

of all these different things.

And so what Ingra does is it opens a.

over 4 43 outbound to our cloud.

And then the cloud handles a,
generates a URL that then that web

hook can call into that then gets
routed down into your machine,

CJ: Yeah.

So it's if you are running your app
locally, if you're building a web

application locally and you're running
it and it pops up on local host 3000,

like your friend that's in a different
city cannot go to local host 3000 and

see your program running because it
doesn't have access to the internet.

And I think it's also probably worth
mentioning that It's hard to get

into someone's computer on purpose.

, like the operating
systems are protecting us.

Yeah, it should be really hard for
any outside service to get in and

this is intentionally difficult
and yeah, luckily anytime we're

working with third parties, we can.

Depend on ngrok to poke a hole
through that magical firewall that

we've created and give access.

I, is there any like security
considerations, I guess this is of

like tangential, but that people
should be aware of when running

ngrok because you are poking a
hole in that, yeah, in that inbound

Keith: Yes.

Poking a hole is a phrase that
comes up pretty often because

at first glance, it's scary.

it's scary cuz you're like,
wait a minute, I'm doing what?

so what we did a few, about a year or so
ago now, we launched, or we had them for

a while, but we doubled down all of them.

We've got some capabilities you
can put on top of that tunnel

so you can do IP restrictions.

So if you've got IOT devices,
you can say only our command and

control server has access to it.

You've got, mutual t.

So you can have shared certificates
on each side and a handshake

to secure that connection.

you can also put an identity provider in
front if you do dash OAuth equals Google.

Now you can force someone to log
in through Google before they

have access to your connection.

And in fact, you can add
restrictions on top of that.

In fact, let's see, by the time this
airs, we added some updates to our

free plan to be able to add OAuth.

So you've got very simple OAuth
protections built into the free plan

itself, along with web hook verification.

So before you let any
traffic into that connection.

We ngrok in the cloud can verify
that payload is from stripe, from

Twilio, from Slack, whatever.

And then anything that
it can't verify gets.

and it never makes it to your server.

So yes, it is poking a hole, but
yes, we can layer on a variety

of security, policies on top.

Colin: Nice.

We're poking a safe.

CJ: Yeah.

Colin: There's a, these are your taglines
that you can have these for free, Keith,

so

Keith: a safe hole.

Thank you.

Thank you.

I appreciate the the
spirit it was offered in.

How's that?

Colin: Absolutely.

Yeah.

And I what's exciting to me, webhookshave
always fascinated me because I, when

I teach APIs, I like to think of it
as like turning a light bulb on across

the world and like a web hook is a.

A post request to a thing and
hue lights and other things.

you can have a web hook that, when
a payment goes through Stripe,

make this light flash green.

Or play, your cash register
sound in, in Slack, which is fun.

And there's a lot of ways, like
Zapier today is one of the bigger,

Low-code integrate X to Y type of apps.

And most of that today is run by webhooks.

So you may, if you're new to webhooks,
you may be using them already.

You're just not, catching
and sending them yourself.

things like lambda functions in
the API gateway and aws, like a

lot of that is really powering
webhooks, which is really great.

and I think that there's
two sides of this, right?

If you're a company or an app and
you wanna provide webhooks to an

end user, or if you're more commonly
your a developer who wants to consume

webhooks like stripe, webhooks,
slack, webhooks, things like that.

what did some of that inspire, what
webhooks.fyi is built for, or how

did that come about for you guys?

Keith: Y Yeah, it completely inspired it.

When we were working on the web hook
verification within ngrok, one of the

first things I was tasked with when I
joined up was, find a list of all the

webhooks out there and let's figure
out which ones we're gonna support.

it's a very reasonable request.

So I googled, web hook directory and web
hook, like best practices and everything.

And it turns out there
wasn't much out there.

Everyone had their list of
their favorites, but no one

actually dug into how are these
verifications actually handled.

So I was sitting there making a Google
doc of, 50, end up with like a hundred

and some different webhooks of.

API company you can think of
was probably on that list.

I think that list has grown
to 150 or so at this point.

and we are looking at how do you
verify the payload, which ones support

ssl, which ones don't, what are the
payload options for each of those?

And while we are building our own
implementation of that verification,

we realize, like realized like there
was no directory, there wasn't a

place for capturing this information.

And I'm sitting there with
this big Google Doc, I'm like,

why don't we turn this into a.

and just share it with people.

And then we started digging into what are
some of the best practices around this.

So we first analyzed, we, we took the
top a hundred or so and put them on

the site and we said, okay, how are
the different web hook providers, what

verification methods do they offer?

some are HMAC 256, some are this,
some are that, we dug into, which

ones are over HTTP versus HTTPS.

And we dug through probably 10 or
12 different aspects of all of.

And we released it.

And we released, here's what we see
in the wild in the field right now.

Here's what we all know are good things.

Let's make a list of the ideal
and call out and celebrate the

people who are doing it well.

And so on the site, webhooks.fyi,
there's a web hook directory and

there's a bunch of list of web hook
providers and some that have red X's

and some that have green check marks.

And we're trying to encourage
people to do things.

CJ: this is a fun list.

and also, there's people that are
not on this list that I'm like, oh,

I wonder if that would encourage
them to migrate from pub sub hubub to

webhooks or some, something similar.

But, I think also it's worth talking
and breaking down a little bit.

Some of those characteristics
that you went through.

So do they support, TLS or how
are they verifying signatures or

payloads or just, what are of the
differences there and maybe, yeah.

Do you have any that are like
the gold standard that like, oh

hey, everyone should go implement
their webhooks, like this company.

Keith: Yeah.

that was the fun thing is we had this
list of a hundred and some different

webhook providers and we categorized
all of them and we said, okay, which

of the ones we should implement first?

We narrowed, did the
list down to about 50.

Then we said, okay, now let's
actually start building the

verification code to verify these 50.

So you guys wanna take a guess of
how many different verification

processes there were for 50 web.

Colin: Probably 50

Keith: 50.

CJ: yeah, I think, I'm going
to cheat here a little bit

and guess like 10 based on the

Keith: there were 49,
49 different approaches.

There was one, I forget who it was
offhand, but there was one that

duplicated somebody else and that was it.

it was horrifying.

Colin: It's what happens
when you roll your own auth,

CJ: Holy smokes.

Keith: Yeah.

Yeah.

and this is just the authentication.

This is just a verification.

This isn't the additional capabilities.

but when we look at some of
those capabilities, we have

things like, Shared secrets.

So this is the model that, let's see.

I think Okta uses it, DocuSign.

Oh, Twilio implements this too.

In the dashboard itself, there'll be some
sort of secret that's generated and then

that gets used in the signature creation.

And then on the receiver side, you've
got that secret and then you generate

the same hash and you use that secret to
make sure, yes, this was definitely came.

Twilio, this definitively came from Okta.

What we've seen a lot of in the wild is
people not verifying payloads at all.

In fact, I do a demo where I
open a very simple Twilio app

and I have people texting in.

I go, Hey, by the way, if you
have Postman, there's my u URL

endpoint on the board right now.

Why don't you go ahead and do
a post and see what happens?

and people start posting, data to my
endpoint right there on the screen.

And sometimes things go off the rails.

It will keep it pg, pg rated here,
but sometimes it goes off the rails.

and I point out to people, I
say, look, this is a silly Twilio

app in front of an audience.

I said, what if you are waiting
for a Stripe web hook to confirm,

yes, Colin paid for this shopping.

if I know your endpoint and you haven't
verified that payment did come, or

that payment notification came from
Stripe, somebody else could make a

post and Mark Collins' order is paid.

And how would you know that?

How would you ever find that out?

And that's.

It's scary.

So anyway, we go through these different
processes and we wanna encourage

people to verify whether that is a
shared secret, like a Twilio or Okta.

There's some that use,
like asymmetric keys.

I think SendGrid and PayPal use those.

So you've got a, a key on each side,
and then it signs with the private.

And you can verify it with
the public key and say, yes,

definitively this came from PayPal.

some of 'em are using like is a
whole other challenge, but it's a,

it's still, it comes down to, it's a
way of being able to say, yes, this

definitively came from the source.

And some of the providers
actually support multiple ones.

I think SendGrid supports two or
three different approaches, so you

can kinda gauge your verification
method to, according to the

skills and the abilities that you.

CJ: That's cool.

one thing too that has been interesting
as we're like reviewing Stripes web hook

situation is if you're building a web
hook provider that's sending these web

hook notifications, one of the challenges
is versioning your event payloads.

So that if you have, you're adding a new
property, but you don't wanna break or

you're removing a property even worse.

You don't wanna break any of
those web hook consumers, and

so you have to version it.

So now like each web hook endpoint has
a different version and the payloads

have different versions and stuff.

And so when looking@webhooks.fyi, under
Web Hook security, if you look at this

like data list notifications option, that
is really attractive to me because like

then you don't actually have to include.

any like too much stuff that's
gonna get versioned, right?

Like you can build a payload that
will be very consistent across

decades worth of versions, ideally.

But then the consumer, when you receive
the notification, depending on your S D K,

that is then pinned to some specific A p I
version can fetch the latest copy of that

object that you're being notified about.

You get the latest state, but you
don't actually necessarily have to get

that in the payload of the web hook.

Keith: Yeah, so a little
bit of background.

a dataless notification, we said earlier,
like the message passing of saying, here,

the state of this thing has changed.

Here's all the information about it.

In a dataless one, instead of having
all that information, all it has is a

reference to the object that changed
and the fact that it did change.

And then you go back and you use
that your eye to then pull back and

say, okay, the thing has changed.

This shopping cart status has changed.

Tell me, that's actually what I
recommend, not just for versioning,

it mitigates some of that, but also if
you have really sensitive information.

So think of a healthcare use case where
you don't necessarily wanna broadcast

that information over the wire.

CJ: It's also if I think about it a little
bit, it's also, really great if you have,

like a globally distributed network.

Of like sharded databases where like
some of your customers are in, the

US and some are in the Australia.

If you're just sending this like super
lightweight dataless payload that

says, here's the object ID that was
updated, then like the API request

from Australia will be routed to
some AWS data center in Australia to

fetch the object faster than, yeah.

If it's located somewhere else.

I don't know, feel like there's.

Some,

Keith: a really good approach.

I hadn't thought about that.

I need to add that to the site.

CJ: it's, the, yeah, I think, yeah,
if you're looking at those web hook

security options, I think dataless is,
pretty attractive, like going forward.

Keith: we also have a section on there
for like operational experience because

while we started with a security
mindset, we said, okay, as people are

building and designing and working with
WebBook, how do they operationalize 'em?

you had mentioned like versioning
of the payloads and things.

There's also the, some capabilities
around forward compatibility

of when a web hook comes in.

How do you support multiple keys?

Because there's always that
moment of like with a shared

secret of my secret has been.

I need to revoke the old
one and generate a new one.

Depending on the traffic and your web
hook patterns and everything, you could

actually miss requests, so the ability to
revoke the old key and generate a new one.

If you can have both of them
running simultaneously, you don't

necessarily miss any requests.

CJ: Yeah.

And I've seen sometimes like where each
end point that you create in the third

party, so. If you imagine the way that
this usually works is you log into the

third party, whether that's Slack or
Zoom or Box or whatever, and there's some

input box in your developer application
settings where you're gonna enter a url.

In many of these, the ones that will
give you like a signing secret will

generally give you a different well,
Might give you a different signing secret

per web hook endpoint that you put in.

So sometimes if people wanna roll the
key, they'll just put the same u URL

twice , and then they'll get back like
different keys and then like they'll

have two running for a little while
and then you can delete the old one.

but yeah, I think it's, operationally it
sounds like a bit of a nightmare, cuz you

have to oh, let me run through the entire
process of checking and verifying the.

and then let me like catch an exception
and do the same exact thing again

to re-verify with like potentially,
N Keys that you're handling.

So

Keith: and we all know that complicated
processes scale really well, right?

CJ: Yeah, it is just like any, anything
that's complicated is going to break.

And I see that PagerDuty is one of
your web hook providers and so that's

something you're gonna need when you
build one of these complicated processes

and someone tries to roll the key.

.
Keith: Yeah, PagerDuty is
actually doing a number of things.

they've got, like the zero downtime
rotation is what we call that one,

so that you're not going to miss
webhooks in between and you're not

gonna break downstream systems.

But, I'm trying to think.

I think HubSpot also implements some
stuff around forward compatibility.

So not just changing the key, but
changing the signature method.

because frankly we know that
at some point we're gonna have

to change our signing methods.

I will also say there's, I think
at least one or two companies

that use MD5 for signatures.

Don't,

CJ: Yeah.

Keith: don't we do name them on the site.

I'm not gonna name them here cuz that's
impolite, but at some point we're gonna

have to change our signature methods.

So having that forward compatibility,
that multiple URL pattern, like having

those things ends up being super powerful.

Colin: Yeah.

If you think that might be you
can go to webhooks.fyi to verify.

but you can also submit your company.

So if you do follow these, there's
a way to submit and contribute,

like to add to that list, right?

Keith: Oh, yeah.

I should say we set this up
entirely as a community site.

so there's very light, any inro marketing
on it, I think it says in the header

and the footer, and that's about it.

but this is all a public GitHub repo.

go to, github.com/ngrok, webhooks.fyi.

See where we're really
tricky with the naming there.

but we accept poll
requests from every anyone.

So we have, there's a web hook company.

I think it's sw.

That submitted, themselves and a bunch
of their customers, we want to make

this the definitive place that people
can come to learn about webhooks

and still have it totally neutral.

So we don't sell webhooks.

We're not interested in that.

We just want the internet to be a
little more secure, and we think

this is one of the ways to do it.

Colin: I think it's a great tool
for if you're doing your first, you

have a guide for web hook consumers
as well as web hook providers.

A lot of these are on the provider
side, the security, things like that.

But then on the consumer side,
there's a whole bunch of kind of

conventions that you have to learn.

a lot of times, let's just say
you're, you don't have your

end groc tunnel on right now.

If you have that turned off and
Stripe tries to send a message to

you, they're gonna do things like
retrying until there's a certain

point when they just say, I'm gonna
give up and not keep sending it.

what does that look like on your end?

How do you verify as a consumer
that you're not playing the

same event more than once?

there's some reasons why you would
wanna replay a same event more than

once, but, even that, that idea of
dataless notifications becomes pretty

interesting in that instance where
it's have we seen this event before?

Okay, if we have, we're not
gonna do anything with it.

Or let's go check and see what's
changed each time versus some people I

do know try to use webhooks as a pure
integration framework, and there's some

challenges there if you miss events.

, right?

Because it's not polling, you're not.

but being able to go reach into the
system with your API key to go get

that latest data can help with that.

I think for us, we do use a lot
of webhooks to tell us something

changed, but we usually use that
to go pull the service to see.

All the changes and maybe all the
ones that we don't know about.

so it's more of, instead of polling every
hour or every day, we'll pull when we

get a message from the system, which is
also a good way of thinking about that.

Keith: Yeah, absolutely.

And There, there's the, provider section.

There's a consumer section,
best practices for both.

Even the people that mostly consume.

We found that internally,
they're still building systems

that are generating webhooks.

Even if they don't go outside
your firewall, outside your

infrastructure, that's fine.

Remember, originally, we start
talking about the value of webhooks

is the fact that we all know
HT d p and we all know J S O N.

Don't generate your own custom
protocol for these things.

Even if your only consumers are inside
your company, still follow best practices

because all it takes from, for that web
hook to then be shared publicly is some

senior exec at your company going, wow,
we have partners, we have customers

who would love that information too.

And now suddenly you're entirely internal
tool set just got published to the world.

CJ: Yeah.

I had a couple of questions.

If we could circle back to
the, like the building and

hosting of the site on GitHub.

it, I really like this approach in
building and public Also, I noticed that

it's using Mark Doc, which is pretty cool.

That's fun to see in the wild.

, but also, yeah, have you found
this to be like a pretty successful

way to engage in open source?

I, it's also seems like a
great lead gen for ngrok and

just generally how's it going?

Would you recommend this to
other companies that are trying

to participate in open source

Keith: we decided to, to launch
it on GitHub because it was quick.

It was easy.

we, had already collected all
the information behind it.

So really the marginal effort was
getting a reasonable design in front of

it and then generating some, Graphics
to show like what the different web

hook flows were and things like that.

So I think it's, I've been very
happy with the results so far.

Yeah, we have Google Analytics on it,
but we don't have a ton of like marketing

analysis tools or anything like that.

but realistically, like seeing people use
this, hearing how people Been informed

by this process, I think is great.

We had, I think two, maybe three weeks
ago, zoom announced that they're doing

a major update to their webhooks and
they announced the platform, they

announced all the changes, and in their
documentation itself it said, we use

webhooks.fyi to make these decisions.

CJ: Whoa, that,

Keith: Yeah.

That was so cool.

Like our initial goal with Web outside
FOI was to take the information we had and

organize and make it available to others.

The aspirational goal was that we start
nudging the industry towards better

practices, was amazing to have somebody
like Zoom come forward and say, your

work had inspired us to do this better.

it was a mic drop kind of moment.

Colin: Yeah, there've been these
webhook communities for a while.

Like I've been in the webhooks Google
group for a long time, but like people who

are gonna go consume or create their own
web hook system, I don't know that they're

going to join a Google group or go.

I think like webhooks.com used
to be a directory, but like

that site doesn't exist anymore.

There's a.

Webhooks, r slash webhooks.

Not a very lively place.

not really where people are going.

I think, having the powered by ngrok
and just like being a little bit of

an organizer and giving, some very
good opinions and also like making

it open so that people can contest
some of those, oh, you know, so.

Something new has happened that we now
believe should be, the new standard

for security or a new option, right?

People can do a poll request
and contribute to it.

it's very similar to like, whenever
someone decides to go write their own

authentication from scratch, you want
as many eyeballs on that as possible.

And that's why so many people use
these open source authentication tools

because so many people have seen them.

And I think, you guys are
giving some credibility.

But then everyone's gonna benefit from it.

Zoom's adding credibility to it.

We're at orbit.

We're gonna take a look at
our webhooks and see how we

stack up against these, right?

And figure out like, what are we
missing or what are we doing right?

and it's a good way to just close
any of those windows that might, we

might have left open accidentally.

Keith: Yeah, and frankly, Sometimes
you don't know what questions to ask.

Like when we first started digging
into this, like I wasn't even

thinking about forward compatibility
of changing the hashing algorithm.

I was thinking about changing
keys cuz that's something we just

have to do occasionally anyway.

But changing the ha hashing algorithm,
I said, oh yeah, Allen Shreve, our ceo.

He's yeah, we need to think about this.

And I was like, oh.

Yeah, actually that makes a lot of sense.

And despite being, in Web Oak for
10, 12 years at this point, it wasn't

something I'd really thought of.

And so we're trying to let the
next people know, Hey, look, here

are a consideration you have.

Even if you don't choose to think about
forward compatibility, that's fine, but

at least you knew that it's something
that you might want to consider.

CJ: Totally.

Another thing that comes to mind with
for forward compatibility too, and like

a lot of the comments that I'm making
come back to adding stuff into SDKs.

So like when we talk about Dataless or
Yeah, it's, you're gonna lean heavily

on the sdk, when looking at H Mac.

verification, which is
what we use at Stripe.

There are helper methods in the Stripe
SDKs in all of the official SDKs that will

do all of the fancy, crypto creating the
H Mac, you just have to give it, here's

the signature and here was the request,
and then let the SDK figure it out.

And what's nice about that is that,
again, you're, you can depend on

the SDK for forward compatibility.

So as maybe as the, as the payload
is using different signing.

Algorithms, you can upgrade
those inside of the SDKs.

And so then it's just a matter of
oh, let me bump my SDK version and

maybe put in a new key or drop the.

The public key somewhere in order to
handle the verification so that, like

I also, I have implemented lots of
web hook endpoints that use H Mac and

they don't have helpers in the SDKs.

And every single time it's
ah, how does this work again?

And like it's so finicky cuz you have to
get it perfect or it won't work at all.

And so it's great that you
have the code on, WebBook FYI,

Keith: that's one of the kind
of horrifying things we found.

Like that moment of,
how do I do this again?

We found, I don't know, 10,
15% of the docs didn't actually

have SDKs or code samples.

It was like a bulleted list of
get this field, then do this,

and I'm like, wait a minute.

You are trying to appeal
to developers in general.

You don't know their skill
level, you don't know they're

understanding these topics.

You're expecting them to
do it right the first time.

I'm like, come on, this is, you're
making life miserable for them.

And I always defer to use
the SDK if it's available.

but every company has to prioritize and
they're going to not be able to tackle

every language, every framework out there.

And I, I don't criticize anyone for
that, but there's gonna be times where

you are working in an environment
that they do not have the tooling for.

So you have to figure out,
okay, can I do this myself?

CJ: right?

Is that, so it sounds like inside of
Enro now you can use it in production

to handle message verification.

Is that.

is exposing SDKs from the
ngrok side that handle that?

Is that something that y'all
have thought about at all?

providing those open source
as oh, hey, this is just like

the standard H Mac thing.

You pass in Slack or whatever.

I don't know.

And then it just will
verify things for you.

Keith: Yeah.

the way it works on our side is that
you choose your web hook provider.

So if you're on the dashboard, you
choose Slack and you put in, I think

it's the shared secret for Slack,
and then we handle it in the cloud.

We've considered, how do we
provide general tooling for that?

That's really hard.

We've tried to say, can we do a inject
your own logic, like a Lambda style

kind of thing for web book verification
that scares the crap outta me executing

other people's code in our environment
as eh, Yeah, those are some interesting

questions that I'm not sure we're
willing or ready to answer at this

point, but as long as you're in the
flow, you can do a lot of things.

CJ: totally.

The fact that there were 49 different
implementations across 50 providers

also, is so frustrating because the
way that I like to implement a web

hook endpoint is just, I want like
slash webhooks and I want every third

party that I integrate with for that
application to all go to slash webhooks.

And now you have to like fork on the
source, like where did this come from?

And now you

Colin: big case statement,

CJ: Exactly big case statement.

And some people think that,
synchronous webhooks are cool and

asynchronous webhooks are cool.

Like all this stuff.

I'm like, Hey, that is another huge, I
think selling point of letting N Groc just

do it for you is that, hey, if you figure
out all of the verification piece and I

don't have to worry about that, then my
web hook endpoint is literally just, take

the payload, stick it in the database, and
then kick off a background job to process

this later and then respond with 200.

And I'm like gonna walk away.

and yeah.

Keith: there was a great article
going around recently called,

the Self-Provisioning Runtime.

And it was the whole mindset of that is
as you're building these stacks and these

operating environments, if we can build
tooling and support infrastructure so

that the developer building their app only
has to worry about their business logic.

they get more efficient, they get
more effective, and they can scale.

They can operationalize
things that much easier.

It's really the, that's the mindset
behind most of the DevOps philosophy

is how do we make sure that developers
stay focused on building business logic?

And so we actually had the same
concept on the ngrok side of things.

So how do we just take
the security stuff off?

just make it so that it works by default?

And you don't have to do this because
implementing OAuth, implementing

webhooks, like all these things suck,

. CJ: Yep.

Keith: it's the exact same patterns
over and over again, but it's

not entirely the same pattern.

It's really close to the same pattern.

It's 49 out of 50 times.

It's different, but it's just different
in little ways, and it's irritating.

Colin: Absolutely.

We found that link.

We'll put it in the show notes
cuz I think, that idea is I

think with the stripe, listen.

framework too, like being able
to listen to those webhooks.

I see this, we'll probably talk to
Alan about this when we have him on

the show, but ngrok go looks really
exciting too, just being able to

like, have your app start listening.

For you rather than have like I,
I like running ngrok locally and

just like seeing, those things.

But as a developer or even as someone
who's focusing on developer experience,

like that time to first hello world
or time to that first light bulb going

off and it's all these steps that
we have to do to be secure are only

gonna slow that down if you don't
provide ways of, Self provisioning,

runtime or helpers in your SDKs.

and so it's a good thing to think of
if you're developing developer tools

or if you're in developer marketing or
developer advocacy and you're trying

to figure out, yeah, everyone seems
to fall off at this point where we ask

them to go do a ed list of 10 things.

what can we take off their plate?

What can we push to the edge there?

And yeah, it's, I think the thing
that I'm most mind blown why

right now is just how many things.

I'll go back to Twilio right now.

and, between Jeff and Alan and yourself.

yeah, it was a, around that time
was a, was what a time to be on

the internet and all the things
that we get to benefit from today.

Keith: I, I think say 2009 through
2012, there was a lot of foundation laid

for kind of an internet renaissance.

and some of that has led to all the,
these API companies, a lot of it's led

to some of this machine learning stuff.

Machine learning like ChatGPT would be a
lot harder to use and implement right now

if we didn't have the APIs behind the.

And the APIs rewind 12 years ago and
people were just getting their hands

and their heads around these things.

So I think, yeah, it was a cool
time to get started in the space

and I look forward to what's the
cool thing to get started with now?

CJ: I was just checking to see if
OpenAI had webhooks, but I'm like,

what would that use case even be?

instead of webhooks, they just
have like really long requests.

you make an API call and it takes,
I'm not bagging on it, but yeah,

like two seconds later you'll
get back your check completion.

but.

earlier we talked about
the Stripe listen command.

So at Stripe we built a custom CLI
for integrating against Stripe, and

it has a bunch of different features.

One of them happens to be Stripe, listen,
which is a tunneling, a tunneling.

Mechanism so you can receive
your stripe webhooks without

having to set anything up.

So it will set up the WebBook
endpoint and also deliver it.

and as part of that we
have a VS code extension.

So like from inside of VS code
you can start the listener.

And I found that there's also
a ngrok for VS code extension.

And I assume this works very similar,
where you can just like fire up

ngrok directly inside of your editor.

Keith: Yeah, just like that early
thing of a wanting to set his phone

down, keep his hands on keyboard.

We have that exact same mindset for this.

Phil Nash, another Twilio
alum, built that vs.

Code extension years ago at this point.

but it's all about how do we keep you in
the environment that you're working in?

How do we not interrupt your flow?

you wanna be effective?

Let's make it so that you
are and just get outta your.

CJ: totally love it.

Colin: Awesome.

I think that's a great place
to leave people wanting more.

They we're gonna put a bunch of links.

I found a really good article
from 2007 from Jeff Lindsay

talking about how webhooks are
going to revolutionize the web.

It's a good read.

gets into a little bit of the
message handling and the small

talk type message passing that,
that we talked about here today.

but where can people find more about you
and all of your work on the internet?

Keith?

Keith: Yeah, absolutely.

my day job is inro, so ngrok.com,
that's the place for everything.

If you need to reach me specifically,
you can email me danger@ngrok.com.

I respond to that pretty quickly.

if you wanna find me online,
everywhere else, I'm Casey Software.

That's c a s e y software.

and definitely check out webhooks.fyi.

It is the source.

We really wanna make it available to
everyone and, inform the internet.

Make it a little more.

CJ: Thanks a ton for all your
hard work on that resource.

I think it's a wealth of knowledge
and information and yeah, it's

making the whole community better.

So really appreciate that, Keith.

yeah, so as always, you can head over
to build and learn.dev to check out

the links to the resources that we've
just mentioned and the show notes.

That is all for this.

Thanks folks.

We'll see you next time.