Do you struggle to make sure your code is always correct before checking it in?
What about your team member's code? That one person who never wants to run the linter,
tired of dealing with tons of conflicts and spurious Git changes? You need Git pre-commit
hooks. Well, we're lucky to have Stephanie Molin on the show today, who has done a bunch of writing
and teaching of Git hooks. This is Talk Python To Me, episode 482, recorded October 24th, 2024.
Are you ready for your host? You're listening to Michael Kennedy on Talk Python To Me.
Live from Portland, Oregon, and this segment was made with Python.
Welcome to Talk Python To Me, a weekly podcast on Python. This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
both accounts over at fosstodon.org, and keep up with the show and listen to over nine years of
episodes at talkpython.fm. If you want to be part of our live episodes, you can find the live streams
over on YouTube. Subscribe to our YouTube channel over at talkpython.fm/youtube and get notified
about upcoming shows. This episode is brought to you by Sentry. Don't let those errors go unnoticed.
Use Sentry like we do here at Talk Python. Sign up at talkpython.fm/sentry. And this episode is
brought to you by Bluehost. Do you need a website fast? Get Bluehost. Their AI builds your WordPress site
in minutes, and their built-in tools optimize your growth. Don't wait. Visit talkpython.fm
slash Bluehost to get started. Hey, everyone. Before we jump into the interview with Stephanie,
I want to tell you real quickly that I just released a blog for Talk Python. Now, we have had tons of RSS
over there because that's what powers podcasts. You can subscribe to the episodes. You can subscribe to
an RSS feed for new course announcements over at Talk Python Training. And I've had a personal blog
time over at mkennedy.codes, but no official Talk Python blog. And so I'm going to be posting
really cool things on there. I've already got a couple of articles posted, but I have plans for
some interesting series. And anytime there's some more interesting announcements or exciting news I
want to share with Talk Python, it's going to be over on the Talk Python blog. So if you're interested,
I would really, really appreciate it. If you go to talkpython.fm, click on blog, right in the
navigation or at the bottom and just subscribe to the RSS feed. That way we can stay in touch.
And with that, let's talk pre-commit hooks. Stephanie, welcome to Talk Python. It's awesome
to have you. Thanks for having me. Yeah, really looking forward to talking about pre-commit hooks.
You know, these are things that I'm sure a lot of people have heard of. I've certainly heard of,
but to be honest, it's not much I've done very much with. And I bet a lot of people out there
listening are like, yeah, that'd be a good idea. Just like continuous integration and writing tests.
Now let's get back to it. You know, something like that. So I think
there's a lot for people to take on, take away here. And we'll talk about what are these pre-commit
hooks, when to use them, how to build them, and a whole bunch of other things that you're up to.
So it should be a lot of fun. I'm looking forward to it. Me too.
Yeah. Now, before we get to that, how about your story? How do you get into programming Python and
pre-commit hooks and all these things? Hello everyone. I'm Stephanie Molin. I am a software engineer at
Bloomberg. And I would say, I guess I got into programming in Python. I initially was programming
in R and I was doing more data analysis while still building some things. And I needed to build a web
app. And one of my teammates had suggested that rather than battling with Shiny in R, that I just
learn Python. So I took a few weeks and just forced myself to do that. And I built something
with Flask. And that was how I got into it. Oh, that's really awesome. Yeah. You were doing work in
not finance, but in ads or something like that with R. What kind of work was that? Like we just generally
add, you don't have to go into details. Yeah. So it was, it was mainly reporting and doing analysis on how
client campaigns were going. But what really got me started with programming was more, I had gotten
involved with a hackathon team and we had built an alerting system. So just monitoring when something
weird went on with the campaigns. And I really enjoyed building more, more so than the analysis.
And so I had to find a way to, and I enjoy like a little bit of data and more on the coding side.
So I had to find something that would let me combine those two.
Yeah. Well, that sounds really fun. I definitely, I'm on the same wavelength as you with data analysis
is fun, but the building is, is really where things get interesting and, you know, look back and see
like, Oh, we built this thing. That's, that's a pretty awesome feeling.
Yeah. It was, it was a ton of fun and we ended up getting, I think third place on the hackathon,
but yeah, that was, that was really that moment where it was like, I got to taste of something else.
And I was like, this is, this is what I want to be doing.
Yeah. Oh, that's fantastic. Was that at your company or was that someone?
That was at the previous, previous role. It was the ad tech company. And so that was actually
all built in R, the alerting system. And then, Oh no. Yeah. Okay. Yeah. And then, and then as we
worked more on it, certain things ended up moving into Python. So a lot easier to work with and to
automate things and not have like some laptop running R somewhere.
Yeah, exactly. It's, that's sort of the promise of Python over a lot of these things that at first
blush seem somewhat equivalent, right? Is that it's, it's a real programming language that can go on to do
all the stuff. You don't have to try to automate some weird thing. That's not really meant to be that
way. Right.
I know. And now, I mean, I could not write R if I, if I had to, I wouldn't, I don't think I would.
Yeah. Well, I was going to ask you now, which side of the fence do you spend more time on R or Python?
It sounds like.
I haven't touched R in maybe six plus years at this point. So I, yeah. Other than the arrows,
that's probably the only thing I could manage too.
Yeah. No more equal size, just arrows. Okay. Got it. Awesome. Well, that's super fun. Let's talk
about pre-commit hooks, right? I've had Anthony Sotili on the show to talk about his pre-commit project.
It was a long time ago and I'm sure that project will get a bit of a shout out from your work as
well. But, you know, congrats, you put together a really nice series of articles and resources
teaching people what commit hooks are, how to debug them, how to build them, how to choose them. So I
think, you know, the stuff we're going to talk about, I'll link, of course, in the show notes.
It's a really nice resource for folks. So thank you. I appreciate that.
Yeah. Yeah. You bet. So let's talk about NumPy doc, doc string validation. This is, this was your entry
way into what this whole world of pre-commit hooks is, right?
Yeah. So, and I think July, 2022, I was at my first EuroPython and I decided to do the sprints
for the first time. I ended up working with the scikit learn team and they wanted to make sure that
all of their doc strings were conforming to the NumPy doc standard. They had a file in place or a test
file that you could run and just validate that whatever changes you made were now being validated
as far as doc strings. And I remember at one point, like I had, I think done 12 or so PRs in that sprint.
So I was very productive. And there was one early on, I think in the second or so, where it just wasn't
working and I couldn't figure out why it was telling me it wasn't valid. It was saying that it wasn't
ending in a period. And I had called over the, one of the maintainers and we both stared at it. To us,
it looked like a period. And I ended up just deleting the doc string and starting over. And it turned out
that it was a trailing space at the end. And so I had asked the maintainer, like, how do you not have
this happen to you? And the response was, you should install pre-commit. And by then I had, I was already,
I had to leave. So I was like, make, I made a note to myself. I need to research this when I get home.
And when I did, I was like, well, how did I not know about this before? And I set it up on things.
And then I went to look, does NumPy doc have that? This seems like exactly what you would want.
As you're writing code, you want to make sure that it's going to check the doc string there. You don't
want to have to run some other thing later on and remember to run it. So I looked and there was no
pre-commit hook for NumPy doc. And I had made something, something that initially we had just
used internally within my team. And then later on, I kind of wanted to use it for a personal project.
And so I set about seeing how we could actually open source it. And I had contacted the NumPy doc team
and they were very, very interested in it because there was a reason there was no hook. It's because
no one knew how to do it. Right. And at that point I had the horrible realization that what I had written
would never work outside because it was relying on things being installed. So, and then I felt pretty
bad about promising that to them. So I managed to come up with an entirely new solution in a weekend
and figured out how to use the abstract syntax tree to work through. And so I built an entirely
new version of it. And that is what is currently available in NumPy doc. And that actually led to
them inviting me to be a core developer for NumPy doc. Congratulations. How cool is that?
Yeah, I know. It's like the full spectrum, right? And just having heard about it and then just
seeing the connection between two things that weren't previously connected.
Yeah. Yeah. Well, I think your comment about the pre-commit hook not previously existing,
you know, for this project also is, it's pretty interesting, right? It's kind of like I hinted at,
I mean, a lot of people hear about this kind of stuff, but that doesn't mean they're putting it
into practice, right?
Yeah, for sure.
And so how do we, you know, let's, let's find our way over to pre-commit hooks in general. So how do we
encourage people or ensure that people follow coding rules, right? We've got tools like black,
we've got tools like rough. Now those will work awesome. If you give them a consistent config file
or config settings, not so much with black, but rough. Anyway, they'll make those changes and do a lot of the
kind of stuff that we're talking about here, but that requires, like you said, people to have it
installed, people to run it and people to buy into the whole concept of the project in the first place,
right?
Yeah, that last bit.
We're all using these tools and we're all going to run them and we're going to remember to run them
until one person goes, I don't like these tools. I'm not doing it. And then their settings fight with
your settings or their spacing fights with your spacing or whatever, right?
Yeah. I think what has, what really helped in my experience, when you incorporate these things,
even like going and approaching open source projects that didn't have a pre-commit set up and just asking
if they were interested in it, it's, you really see the value when you've, you think if you've ever
reviewed something or gotten review comments about, you should start a new line here. I don't like this
space here. And then you think about how much time you waste at that stage. And then you still have
zero consistency because you did it one way, someone else does it another way. And even further than that,
it's just the time you waste in your code. Oh, I should put this on a new line and reformatting files
when you could actually be writing things and thinking about how should I design this algorithm,
them. Right. And so I think a big part of making sure that once you find these tools that you're
going to use and actually make sure they're using, it's making it easy to use. Like you said, yeah,
you can just run black or rough, but you have to remember to run black or rough. And that is the
key problem. And what's so great about pre-commit or even extensions in your IDE is that these things
become automatic and that's what you need to get towards for these things to actually stick.
Yeah. To make them automatic and not part of it. And to some degree, continuous integration can do
those kinds of things. But a lot of times it's too late at that point. It's already checked in,
it's already committed. And then you've got the back and forth of now it's a diff, but it's only a diff
because they spaced it differently when they hit save in their IDE than when you hit save in yours and
all that. So pre-commit hooks run prior to actually leaving your computer, right?
Yeah. So it's actually prior to even the commit. So when you do get commit and you, let's say you pass
your message and if it's successful, you normally, you see the hash that gets generated. If you have
pre-commit hooks enabled, then if that, those checks don't pass, then that commit never gets created in
the first place. So you still have the files staged, but nothing has made it to the commit.
Yeah. That's great.
Yeah. I was just going to explain maybe a little bit about how they work if you're curious.
Yeah. Yeah. Well, let's start with just like, what even are, are these pre-commit hooks?
Yeah. So pre-commit hooks, and I think the naming is, is quite overloaded and that leads to a lot of confusion.
So at the lowest level, a Git repository in general supports a hooks system. So there's a variety of
different types of actions that Git will trigger a script on your behalf. And one of those such actions
is pre-commit. So as I described before, as you run Git commit, this gets triggered. Another thing
might be pushing. You can have Git wired to run some script when you push. Now that is Git's version
of pre-commit and hook, singular hook, because you can only have a single file run, single executable
can run.
Right. If you go to your, it's in the Git folder, there's a hooks subfolder and it's got little
samples for all the different lifecycle things, right?
Yeah. And yeah, they provide some, they have to, like I said, they had to be executable, but you can
be in any language that you have available on the machine. And so Git provides some examples. I do think
there are a few stages that don't have examples, but it's basically you take the name of the stage
that you're going to use and that's the name of the file. And that has to be an executable and Git
will run it at the designated moment.
Okay. So it could be a Python executable or it could be a Go executable or whatever, but it's just
one, right?
It's just one. Yeah. Cause it has to be named. And like in the case of pre-commit, it has to be called
pre-commit, nothing else.
This portion of Talk Python To Me is brought to you by Sentry. Code breaks. It's a fact of life.
With Sentry, you can fix it faster. As I've told you all before, we use Sentry on many of our apps
and APIs here at Talk Python. I recently used Sentry to help me track down one of the weirdest bugs I've
run into in a long time. Here's what happened. When signing up for our mailing list, it would crash
under a non-common execution paths, like situations where someone was already subscribed or entered an
invalid email address or something like this. The bizarre part was that our logging of that
unusual condition itself was crashing. How is it possible for our log to crash? It's basically a
glorified print statement. Well, Sentry to the rescue. I'm looking at the crash report right now,
and I see way more information than you'd expect to find in any log statement. And because it's
production, debuggers are out of the question. I see the traceback, of course, but also the browser version,
client OS, server OS, server OS version, whether it's production or Q&A, the email and name of the person
signing up. That's the person who actually experienced the crash. Dictionaries of data on the call stack and so much
more. What was the problem? I initialized the logger with the string info for the level rather than the
enumeration.info, which was an integer-based enum. So the login statement would crash, saying that I could not use
less than or equal to between strings and ints. Crazy town. But with Sentry, I captured it,
fixed it, and I even helped the user who experienced that crash. Don't fly blind. Fix code faster with
Sentry. Create your Sentry account now at talkpython.fm/sentry. And if you sign up with the code
TALKPYTHON, all capital, no spaces, it's good for two free months of Sentry's business plan, which will give you up to
20 times as many monthly events as well as other features.
So if you want to run more, you basically have to, potentially, write a program which then itself
figures out all the things to do and then delegates to running them. Like if you want to run ruff with
a fixed formatting issues and you want to run the checker fixer for NumPy doc strings and all those
things, you'd have to write a sort of orchestrating program for that, right?
Yeah, it's almost like you're writing in the case of like a bash script, like a giant bash script where you have
to decide, you know, do you fail early? How do you like and check, do I run this one and then this one? And then
even worse, you're actually, in that case, you're probably running everything, you know, you're running everything
sequentially. And if you don't do it carefully, then you know, maybe, maybe you want to fail early, maybe you don't.
So that becomes very, very challenging to configure and also to share because the thing about that file is that is not
included in version control. So that would be something that you would maybe have to store somewhere else and then do a
symbolic link. And then that becomes already a lot trickier for everyone to manage.
Yeah, I was just doing that last night and that's an AI question. I don't remember how to do that.
I know you can do it. It's not that hard. It involves LN, but you know, ChatGPT, what do I do exactly?
LN-S. I've had to do that quite a bit.
It's burned into the brain, huh?
So one of the things that you recommend so we don't have to build this orchestration piece is actually pre-commit,
which is a Python project, right?
Yes. And it's not the only one. So again, that's where like the naming becomes challenging.
But pre-commit is built in Python, but it can run hooks in a variety of languages.
And it interfaces with GitHub's system for you. So it creates that executable and plants it there.
But that executable is then pointing back to pre-commit so that you can just define a simple YAML file like you can see part of it on the screen right now.
And it becomes very easy because essentially you're just configuring what you want to run.
You're not actually coding the logic of the checks and how they relate to each other.
Right. So let's assume that all the pre-commit hooks that you want to run somehow exist out there in the world, right?
You don't have to create them for the moment.
So what you can do with pre-commit is you can set up a YAML file.
I always get those crisscrossed.
A YAML file, a pre-commit config YAML file, which then has a bunch of listings of here's a Git repository.
And if you install it as a Python package, here's a bunch of things that you can run on it, like check toml, check YAML and so on, right?
Well, it doesn't actually have to be a Python package, right?
So in that repo, and we're maybe jumping ahead, but there's a special file in that repo, which will tell pre-commit how it actually needs to install it.
So it could be anything.
Oh, that's interesting.
So the thing that integrates with the pre-commit project, it has to opt in in a sense in that it has to have a configuration file or a launch file or a setup file, something like that.
Yeah. So right now we're looking at pre-commit config.
There's pre-commit hooks, and that one is kind of registering it with pre-commit system.
So it tells pre-commit how to install it once it gets a hold of it.
And it also lists out these hooks that we see here under ID, but that will be defined over there so that pre-commit knows, well, what is check toml?
What is check YAML?
Okay. Yeah.
That's really cool.
And you can have more than one of these repositories in there, right?
Correct.
Yeah. So the repos section is a list of repo sections, and then each repo then has other config, like the individual hooks that you want to run from that repo.
Right, right.
So for the first example that you have in this, and this is your article, I guess, I don't know if I give this the proper announcement, but how to set up pre-commit hooks.
This is your, I perceive this as kind of your getting started article for this whole series.
I don't know if you see it that way.
Yeah.
Yeah, this was the first one.
I had gotten a lot of questions on how to do this.
And I think it's always interesting, especially when you think about, you know, speaking at conferences, I feel like, and which I do a lot of, and I feel like a lot of what gets more hits in that sense is like the advanced stuff, maybe more creating it.
But there's so much value in people just getting started and figuring out how do I even use this in the first place?
Because this saves you so much time.
So I really, this was where I got started for that reason.
I think a lot of people were able to benefit from this article.
Yeah, it seems like it.
I know it's fun to talk about the super advanced deep dive things, but most people, they just need to get started.
They just need some foundation, right?
And I think, I think that's actually where most of the benefit comes from, even though it is really fun to see some cool deep dive talk that people are going into, right?
So this next one is pretty interesting that we're adding here in this example, and that's the rough pre-commit from straight from Astral, right?
So this is just github.com/astral.sh, which is the company behind rough newbie.
And this is the rough pre-commit.
But what's interesting about this is, well, one, that it has nothing to do with the pre-commit project.
But two, that this one also takes special arguments that you can pass to it.
Yeah, so I think the rough pre-commit one is just a smaller version so that it works faster with pre-commit.
Because pre-commit will have to install these at some point.
It will have a cache.
So if you don't change the version in this case, it will be able to reuse that.
But that first time, you do have a bit of a delay.
And that's not something you want.
It's something you have to be very careful of when you want to be using these.
And then the args thing is nice because you have a few options when you configure these tools, depending on what the tool supports.
In this case, rough supports, as I think we mentioned a little bit earlier, configuration file.
So, for example, you could have stuff in your pyproject.toml.
But the key here is that maybe you're using rough in your IDE.
And maybe you don't want to do the same kind of changes that you want to do in pre-commit.
Maybe you wanted to ask you if it's going to change something.
Whereas in the pre-commit stage, you definitely want it to be fixed.
So you can use the args here to provide stuff that you only want to happen when it's running in the context of pre-commit.
Yeah, and rough has a exit non-zero on fix, which means if it goes through and you say to fix it, it will fix it.
But then it'll error out and say that wasn't a smooth transition or whatever, which is cool because that will then fail the commit itself.
Correct.
Give you the modified files and say basically have a look.
See if you like it now, right?
Before it actually just ships it off.
That's so important because sometimes you realize there was some rule that you hadn't reviewed before.
That's not quite doing what I want and let me tweak my setup.
So it's nice to have that bit where you can verify what was actually changed is what you want.
Yeah, I guess it's a little bit dangerous to just say change it and then commit it.
I've had people.
So I did a workshop on pre-commit both on setting it up and then making your own hooks at EuroPython this year.
And I did have a few people actually.
One very insistent asking me why wasn't there a hook or why don't they support just fixing it
and then automatically adding it and committing it on your behalf.
And to me, as a person who works in security, that just sounds very scary.
I don't want things doing that.
I want to see what is being changed and whether or not I agree with it or not.
Yeah.
Why doesn't it just go ahead and push it as well?
Come on.
Yeah.
Well, I think that was part of the suggestion.
I was like, I certainly don't want that running on my machine.
Yeah, it does skip out on some of the benefits of the multi-stage aspects of Git, I suppose.
But it is efficient.
You just get it done all at once.
That's pretty cool.
Yeah, but you don't know what else is grabbing, which is the scary part.
No, of course not.
I know.
Super bad.
So this example that we're talking about here where we've got a pre-commit hook that we're grabbing
and then it takes these arguments, I think this is an interesting point of discussion.
So the example you have in your article just says, what we're going to tell rough is dash,
dash, fix, dash, dash, exit non-zero fix, and show fixes, which is all good.
But rough can be pretty complex in its configuration, right?
You can say, disable flight gate, turn this one on.
These are warnings.
These are errors.
And there's a whole, you know, here's how many line columns I want and all of this stuff, right?
So you can either do this argument thing, or if it's supported, you could also potentially have,
say, a rough.toml, right?
Yeah.
So I tend to want to minimize the amount of configuration files I have.
So in my case, I think below I talk about having it in the pyproject.toml.
Yeah, exactly.
So you just add a rough section in there and then you configure things.
And this is stuff that you'd want to use both in your editor as well as in the pre-commit stage,
because you want them to agree.
And nothing worse than one telling you the lines too long and the other one like,
nope, that's good.
Go ahead.
Or put a space after the comma in parameters and then take away the space and put the space and take away the space.
Exactly.
You don't want them fighting.
You want them in agreement.
No, no, you don't.
So I suppose that's a massive bonus of having either the tool.rough settings in your pyproject or just a rough.toml,
however you go about that, it doesn't really matter.
Because then no matter how you're using rough via the pre-commit or for your project, it'll be the same thing, right?
Exactly.
Yeah.
Okay.
Yeah.
That's pretty awesome.
Now, I guess maybe we got a bit ahead of ourselves.
If I want to somehow install a pre-commit hook or pre-commit so that when I then give it one of these toml files,
it'll go subsequently grab them and do the things.
How do you get started with that?
I think I need a rephrasing of that question.
Yeah.
Sorry.
So if I have just a plain GitHub repository and I want to have pre-commit manage the hooks for that repository,
like what do I do?
Okay.
So the first thing is you have to actually install pre-commit.
And that's not the command that's on the screen.
This is more of a pip install.
So make sure you have the Python library in place.
And then you need to have this configuration file.
At least one hook in there so that you have a valid file.
And then you can run pre-commit install.
And I omitted it here, but what I talk about in a different article,
when you run this command, pre-commit actually tells you that it created the git hooks pre-commit file.
And if you open that up, and I have an example on that other article,
it's very simple and it's just calling pre-commit the tool itself.
So in all cases, you need to have it installed in your environment.
And a single time you run pre-commit install, which then does the wiring on the git side.
And this is something that everyone in your project has to run on any machine that they are using.
Because it's part of the repository itself, that file needs to exist there.
And that can only happen if you run this command.
Yeah.
So there's a .pre-commit.config.yaml file.
That's what you put into GitHub at the root of your project or something like this.
But then to actually configure git itself, you've got to run this pre-commit space install.
And it basically wires up the hooks to make that happen, right?
Correct.
So yeah, when you run this, that file gets created on your behalf.
And then you don't have to worry about wiring that up.
And then it's transparent.
All you have to do is tweak your config and then the changes happen.
Nice.
I don't know if the naming, how much to believe the naming.
Can it do things other than pre-commit?
Yes.
Can it do pre-push and those kinds of things?
They don't support every single one.
But there are quite a few that they do support.
For example, I once configured an open source project with a pre-push because it was a slower
check.
And that's something you definitely don't want running on each commit.
But it might be something where you want to make sure when you push the files that you've
addressed something that's maybe a little bit longer.
And that is really not any different than configuring with the pre-commit config YAML.
There's just a separate item that goes in there that says which stage to run.
By default, it's pre-commit.
So you don't see it.
But if you needed to change it, you can.
Yeah.
I figured that was the case.
But I'd never tried.
And given that it's named pre-commit, you know, it's kind of named after one of the hooks,
right?
But of course.
I think that's named probably the most useful one.
I would.
Yeah, I would think so.
I think a very popular example would perhaps be the commit message hook.
So there's a lot of tools that work on, you know, making sure your commits are following
a certain standard.
I think one of them is called committizen.
And so that runs on, my guess is on the commit message hook.
Committizen?
Yes.
Okay.
What is this committizen about?
I haven't heard of this.
I don't think their example uses that.
But I think they do have a pre-commit hook.
And I believe it works that way.
Yeah.
Interesting.
Okay.
What's this thing?
A release management tool for teams.
Yeah, sure.
That makes sense that you want to kind of be a little bit careful about what your commit
messages are.
Maybe you want to grab certain commit messages and add them to your changelog or something
like that, right?
Yeah.
I think there's been quite a bit of talk about this one at conferences I've been lately.
I think it's gotten a lot of traction.
Yeah.
2.5,000 GitHub stars.
That's pretty good.
I'll check it out.
This is news to me.
This portion of Talk Python To Me is brought to you by Bluehost.
Got ideas, but no idea how to build a website?
Get Bluehost.
With their AI design tool, you can quickly generate a high-quality, fast-loading WordPress
site instantly.
Once you've nailed the look, just hit enter and your site goes live.
It's really that simple.
And it doesn't matter whether you're a hobbyist, entrepreneur, or just starting your side hustle.
Bluehost has you covered with built-in marketing and e-commerce tools to help you grow and scale
your website for the long haul.
Since you're listening to my show, you probably know Python, but sometimes it's better to focus
on what you're creating rather than a custom-built website and add another month until you launch
your idea.
When you upgrade to Bluehost cloud, you get 100% of time and 24-7 support to ensure your
site stays online through heavy traffic.
Bluehost really makes building your dream website easier than ever.
So what's stopping you?
You've already got the vision.
Make it real.
Visit talkpython.fm/bluehost right now and get started today.
And thank you to Bluehost for supporting the show.
All right.
What other takeaways should we talk about in this first one?
I think we maybe have pretty much covered it.
Let's see.
I guess, you know, we mentioned before, but if people want to see sort of examples of pre-commit
hooks failing or succeeding or failing because they changed something, which is not exactly
a failure, but stopping and starting over, you have a nice example of what that's like
there.
So one thing that I guess might be useful is sometimes maybe you don't want to run the
pre-commit hooks.
Maybe you need to check in something in a certain way to fix the servers down, right?
We have to check this in.
I can't fix this hook, whatever this hook is upset about right now.
It needs to go in right away.
Just let me commit it, right?
You can do that.
I mean, I think there are probably several use cases or something like this.
Maybe you're going to be squashing things later and it doesn't, and it's, you don't,
maybe you don't even know what the API for you're doing, what you're doing is going to
look like.
It could be, and this kind of ties back to what we talked about earlier, perhaps roughs
doing something and you don't agree with, but you need to like check with the rest of
your team to make sure that everyone's in agreement with let's remove this rule.
Right.
So it's, I, this definitely don't encourage always doing this.
That defeats the purpose, right?
But there is kind of a break glass solution here where you, let's say you first run, get
commit and something fails and it's not something that you either want to fix at the moment or
really can fix.
Then you can just pass it, pass in dash, dash, no verify.
And none of the checks run at that point.
So it's like, as if the checks were never there in the first place.
Right.
Okay.
That's pretty interesting.
Like you say, hopefully people don't run that all the time.
At that point, just remove the pre-commit setup, save yourself.
Yeah.
Like what are you, what are you even doing?
Right.
I suppose there's an interesting interplay between pre-commit hooks and continuous integration,
right?
Like in a sense, they are often checking some of the same things.
What do you think?
So I think it's probably an example, like not, not quite a Venn diagram.
I probably, the circle for pre-commit is entirely contained within the circle for the CICD.
The difference is there are certain things where you can get immediate feedback, quick
feedback locally, and that should be something that you can put pre-commit things like linting,
formatting, et cetera.
And then CICD may be running your test suite.
That's definitely not something you want to be doing in a commit.
Imagine you have a test suite that takes three minutes to run, even maybe three minutes isn't
that bad, but every commit waiting three minutes is definitely not something you want to do.
No.
But it's still a check that you should definitely be running.
So in CICD, I would run everything.
Do the linting, do the formatting.
That's your final, that's your last layer of defense and you need to be checking everything.
And this just allows developers to get that feedback sooner.
Right.
So what you're actually checking in and you finally approve is much closer to what CICD
would kind of want in the first place, right?
Yeah.
Okay.
And it's also a much faster feedback, right?
So like if the thing has to run all the way through the linting, the formatting, the testing,
the type checking, whatever, you might be waiting 10, 15 minutes for all the things to run when
you could have had, you know, under a minute, hopefully way under a minute feedback instantly that
your file wasn't formatted correctly.
It should be near instantaneous, right?
I mean, instant maybe is asking too much, but some of that astral stuff is kind of ridiculous.
Yeah.
I think you have to be very careful, right?
Because there's all these checks and I think you had up on the screen maybe earlier, like
the pre-commit hooks, the general ones provided by the pre-commit organization.
Yeah.
There's tons of things in there, but you do have to be careful, right?
Because if you're like, oh, this could be good and this could be good and this could
be good.
Each check is adding time.
Assuming, like I say, assuming they're all running on Python files, you're adding time
to how long.
So you do have to be mindful of what you actually need.
And if you go to the point where you end up making the whole process take too long, people
are going to stop using it.
And then that defeats the...
Yeah.
Yeah, exactly.
As soon as it becomes a point where people go, I'm not using this thing, then you're kind
of kind of sort of lost unless you can just say, no, you have to use it.
But then you just have unhappy teammates.
Exactly.
Either way, it's not a real great outcome, is it?
I mean, if there's something that maybe only runs on a few files every once in a while, then
if you are having problems with speed, then you can also consider moving that to the CICD.
And I am definitely a big fan of rough, as you said, like just switching from black, flaky,
all that onto rough, you do save a significant amount of time on these checks and it's a huge
benefit.
Yeah, it's pretty ridiculous.
Now, this is not a get pre-commit thing.
This is a pre-commit the project thing.
But you can, if you're using this pre-commit project we've been talking about, you can say
pre-commit space run and do kind of a test without actually doing a commit, right?
Correct.
Yeah.
So there's a bit of nuances.
So if you just do pre-commit run, it's going to run all of your hooks, but on the staged
changes, because it's thinking essentially you're doing like a dry run.
If you, let's say, are adding a new hook and you want to make sure all of your files are
compatible with that new hook, then you might want to do something like pre-commit run dash
dash all files.
So look through your entire repository, regardless of whether you have changes in place.
So if you say pre-commit run, it only works on your, basically your changed files, not the
stuff that's already there and accepted.
Correct.
And another neat thing is in the case I mentioned where you add a new hook, you might just want
to run that hook.
So you can say pre-commit run and then the hook ID, and then you would just run that hook
and then you can define either a certain set of files or the staged runs, whatever.
Yeah.
That sounds pretty useful when you're building your own pre-commit hook, right?
So yeah, depending on how you build it, you can either use that or they have also a try
repo command.
Right.
Got it.
Well, let's see.
Maybe we could jump over and talk a bit through your hook creation guide, a step-by-step guide
to developing your own pre-commit hook.
I thought this was really, like I said, a good article.
And maybe one of the first things we talk about is just what makes a good hook in the first
place, right?
You said that they can't be too long or people will go crazy and turn them off or skip them
or whatever.
But what else?
So I think another big thing is if you're able to fix something, then you should fix it.
In the case of formatting and you're saying, oh, this should have a trailing comma, then
that's easy enough.
You can add the trailing comma.
You don't make more work for the user.
If you can't do that, then you should be very specific saying this file.
And if you have a line number saying exactly where it is, because just saying there's something
wrong in this file and someone has to hunt it is also not a good user experience.
No, that's going to be frustrating and super, super quick.
Yeah.
So be really descriptive about it.
And then also, maybe choose not to make it a pre-commit hook, right?
Not necessarily everything needs to run on every commit.
Yeah, I think that the speed thing is a huge factor.
And in general, I think one big thing that is key to note here is that it's even,
though, let's say you change files that, let's say you change a Python file, a Markdown file
and an image file.
If you're making a hook that only runs on a certain type of file, if you're careful and
specify that, then it's not necessarily a bad thing to include that in there because it will
only get triggered on those certain types of files.
And so like an example I have is the XF stripper.
Well, I created when I was building my website.
Your XF stripper is super interesting.
I'm starting to think maybe I want this as well.
Yeah, I was just very paranoid at one point about just working with images.
And so they come with, what's up here?
So exchangeable image file format data or XF as it's commonly called.
It's metadata that is in the image that you might not realize is there.
And so in this article, I talk about a picture of me presenting that I was given from a conference.
And this was something that was stored, I think, in a Google Drive.
So you have access to all the metadata that was available.
So I never met the photographer.
And yet I know the photographer's name, the camera they use, what type of computer they have,
how they edited it, all kinds of information.
And the dangerous part is the exact location of where this was.
Now, conference, not a big deal.
But you have to think about maybe you're blogging about something you did in your house or your apartment.
And now you have a photo up on your website where anyone can potentially see it that has the GPS coordinates for where you live.
Yeah, that wouldn't be great, no.
So I was very paranoid about this.
And I don't want the idea of like, oh, I'm going to add a new image.
Let me go through my checklist of what I need to do because I know at some point I'm going to mess something up or forget it.
And so this is a perfect use case for the pre-commit, right?
Because you want something that is going to stop you and tell you, nope, you can't do this, right?
And in this case, it can also remove the metadata because I am being super conservative and saying no metadata,
which has the nice side benefit of shrinking files, which is good for serving them.
Yeah.
Well, what value is it to have all that metadata in there for a blog?
Most of the time, most people are not, they just want to see, they want to read the blog.
They're not going to dissect your image, right?
I think it depends what you, I mean, maybe you have a travel blog and you want to know like, here's that location.
And then you have one off post where you introduce yourself and oops, you know?
Yeah.
There's so many ways.
And I think even just thinking, oh, I'm only going to be doing this.
There's always going to be something that later on happens.
So you have to be very careful just upfront that everything is going to go through this track.
Sure.
Can your exit thing, can it be selective about the metadata?
That's something I do want to do in the future.
Just remove the location if you say.
But the thing is, there's like, looking through all of that, it's hard to tell if there might be something in one subset of images you take that might be sensitive.
You can even think of certain situations where you might not want someone to know what kind of device you were using.
Right.
Because maybe they're like, oh, that device is vulnerable to something and I know they have it.
Right.
The worst of these is, I think, the multiple times, pretty sure it was the Samsung, but one of the Android companies posted a picture promoting the new phone.
And, you know, the exit information had the picture as being from an iPhone or something like that.
Oh, no, it was the other way around, I think.
Oh, the other way around.
I think I remember hearing that, yeah.
Well, it was like one phone company was posting it from, but the picture was actually, even though it was about the phone, it was, you know, implying this picture comes from or something.
It was like, nope.
Whoever is on the marketing team just happens to have the other kind of phone and there it goes.
Right.
And it's a huge scandal.
I mean, for those companies that talk about how awesome they're, how much better their cameras are or whatever.
Well, I see that's also the thing, right?
Because you never know who's going to look at the metadata either.
So, and it's interesting because certain things will, certain platforms will remove it.
So I mentioned like Google Drive, it's everything is preserved.
But the thing is, is you have to know ahead of time.
So you'd have to say, I'm planning to put this image here.
Let me upload a dummy image.
I don't care and check if the metadata is still there.
Yeah, exactly.
Yeah.
I think, I think Mastodon might remove it.
There's some certain platforms that will take away that metadata.
I think Facebook might.
It's been a long time.
I mean, it's a huge security concern.
So I imagine more and more places are, but I just wanted to have an abundance of caution and not risk anything happening.
Well, yeah.
And you're putting it on the internet as well, which there's, it goes straight from your computer through some sort of static website process.
And then it's downloaded, right?
There's very, there's no, nothing in between those two steps.
Exactly.
At least not in terms of image processing.
Yeah.
Cool.
Yeah, this is nice.
I'm thinking about grabbing it and trying out.
What file types does it work on?
Does it work on just JPEGs or does it do like WebP and all that?
Any image, anything that's classified as an image on pre-commit, the way pre-commit runs.
And it has to work with, I'm using Pillow.
So if Pillow can't read it, then it's not going to work.
Right.
Then I'll just skip over it or whatever.
Yeah.
So really quick, while we're talking about stuff on your website, your website's super nice.
Did you build this yourself?
Like, how is this thing built?
I did.
I did build it to myself.
I took a couple months in the beginning of the year and I had before a single page where
it was just like some boxes.
And then I was like, this needs to be revisited.
So it's built with Next.js and so React and TypeScript.
And then I use Tailwind CSS.
And yeah, it was kind of just like, I mean, a lot of these things are for me because sometimes,
you know, I like seeing all in one place where I'm speaking next or like stats about where
I've spoken, like a map and stuff.
And I went through, so kind of my process would be, you know, on my iPad, I would sketch out
what I kind of envisioned a page looking at and then I would prototype it in React and
see, okay, maybe this isn't fully work or like tweak things and iterate on a few times
and bit by bit the pages formed.
The latest thing I added was this timeline functionality.
At EuroPython this year, I had this idea for a timeline and I kind of got really, really into
it.
So it was funny.
I had a Python conference.
I was doing tons of React.
But if you scroll down a tiny bit, there's actually too much.
This one, right?
Yeah, yeah.
Versus the little text.
Oh, the complete upcoming.
Yeah, I got you.
So I built this.
Oh, that's beautiful.
I love it.
It's like a little infographic of your upcoming events.
Yeah.
So I was like very inspired and I did this in a few days.
But it's nice because, you know, going from the sketch to the React components, it's become
very natural, which it takes a bit to get there.
But it was nice because I did have to learn TypeScript for some changes in my team.
We were going to be starting moving to TypeScript.
So this was great to work on something that, you know, fit in my head as far as what needed
to be done.
And it was very, very helpful.
But yeah, so I'm very proud of this.
There's still more, tons more to do.
I have massive lists.
But yeah, I remember looking at Google.
This is a nice static site.
Very cool.
And I didn't even see this feature.
This is great.
Broadvon out in the audience says fire emoji for it.
Very good.
Thank you.
And also, thanks.
I see you put the podcast appearance on here as well.
That's cool.
So that's happening today.
Watch the live stream now.
If you're not watching now, then it's probably missed it.
But the recording will be there, of course.
But the reason I say that is you maybe want to give a shout out to some of your upcoming
events.
Yeah, why not?
So I'm going to be in San Francisco next week talking about my Datamorph project.
And I'll also be doing a book signing there for my hands-on data analysis with Pandas book,
second edition.
And then after that, I'm off to France to give a workshop on Pandas and then also talk about
getting started in open source contributions.
And then a couple of weeks after that, I will be at the final conference of the year in Australia.
And I will be talking about Datamorph once again.
And I'm hoping to run my third development sprint on Datamorph while I'm there.
Oh, that's cool.
Yeah, we'll talk about Datamorph in a second.
That's some interesting stuff.
But this is quite the agenda.
You got a full trip coming up.
No, I'm excited.
It's nice to see different cultures.
It definitely does land different, you know, the topics and just reactions.
Some people are at the top excited.
Some of them are just straight face.
You're like, I enjoy it.
I think it really comes into play as far as giving workshops.
I was in Portugal last week and I did the data analysis workshop.
And I think that was one of the best ones I've ever had.
It was very, very highly interactive and it was a really fun time for me.
And hopefully everyone else thought so as well.
Yeah, that's fantastic.
How did you get into public speaking?
Yeah, so I wrote the hands-on data analysis with Panda's book in 2019.
And at that time, if you had told me, go do some public speaking, I'm like, please no.
You're going to France and Australia and Portugal recently.
So I'm like, no, no, no.
Yeah.
And then, well, during pandemic times, a conference reached out to me about doing a workshop on pandas
because I had written the book and doing it virtually.
And to me, that felt like a good stepping stone to get over that fear of public speaking and
the fact that it would be virtual.
I wouldn't really have to look at anyone.
And I was still absolutely terrified when it came to actually delivering that talk.
And when you think about it, it wasn't a talk, right?
So it was my first thing was a four-hour workshop.
And now I'm at the point where a virtual thing is much less desirable because it's so hard when
you can't see people, you can't see our things landing, are they confused, are they with me?
Are they even still there?
So, and then after I did, you know, I made it to the end and I was like, okay, that's
definitely something I want to work on and do it again.
So I did, I came up with a second workshop on data visualization.
And then I think I did two or three more virtual sessions.
And then it became that some conferences were now in person.
And I was like, okay, I think I should try this.
And again, it was still a long one.
It may have even been a six-hour session that time.
So it's like crazy, right?
And then I did that in person.
And I was like, okay, I survived.
And then it kind of just felt like something, if I kept doing it, I would get over it or
at least get to the point where, you know, I could do it without being terrified for a
month ahead of time.
Right.
And I am at that point now.
It is like, I enjoy doing it because I enjoy, I'm very passionate about knowledge sharing and
just teaching people and getting that interaction that, oh, people are really like getting value
out of this.
And that to me is very nice.
Yeah.
It's super rewarding.
So, but yeah, this is quite impressive.
So just, I got the sense you kind of got started pretty soon.
You said 2019.
So that's, haven't been doing it for that long.
And this is great.
So maybe, you know, you brought it, maybe we could talk a bit about your book as well.
I don't know what to say about this one.
Just that it exists and people should check it out.
It's giant.
As you can see, 788 pages.
Holy moly.
That is giant.
Yeah.
So this is the second edition.
If you scroll down, there's also the covers for the Korean and Chinese editions.
Oh, awesome.
And I do not read either of those, but I do have copies.
You can act of faith to put your name on them.
You know what?
I've been told by people that read both of those languages that the name is not quite translated
correctly, but you know, I'll forget about that.
It's cool to have the copies.
Yeah.
So this book covers obviously pandas working through the basics of data analysis.
We also talk about data visualization.
And then there is a little bit towards the end about like actually applying this stuff
to use cases and also a little bit of machine learning.
Cool.
Yeah.
So I'll put a link in the show notes.
People can check it out if they would like to.
All right.
I feel like there's a few things.
We didn't make it very far in our creation guide.
So let's talk about the recipe.
All right.
What are the four steps?
At least Stephanie's recipe for pre-commit hook.
Yeah.
This is definitely my recipe.
I mean, I've, I think I've made two that are published ones and then obviously a few other
for trainings and explanation purposes.
And this, this is something that works well for me.
And I think makes sense as far as thinking about the pieces.
So the first thing, the hardest thing is actually to figure out what are you checking and how do
you actually code that up?
And if you want to do this in Python, this is just, okay, code your logic.
Yeah.
Right.
Yeah.
Well, and if it has a --fix, maybe that's even harder than just trying to
understand, right?
Because now you got to not break somebody's code or sorts of things like that.
Yeah.
But this would be where you start at the basic level, probably first, you know, find,
figure out, can you find the issue and show people where it is?
And then you can look into fixing it.
But yeah, you have to be very careful, especially if you're going to be touching things.
So I guess it's pretty straightforward, but the magic of Python is not just the language
and the static, the standard library, but the 500,000 external packages, right?
There's probably a ton of external packages that understand code, check different things.
And you could, you can use those in your hook implementation, right?
Just like a standard Python package, it can have dependencies and stuff.
Yes.
And so I talk about this in the third step, but I do like to make it as a package just
because you know that that's going to work and grab the dependencies as long as you follow
what you already know.
And pre-commit will, you will tell pre-commit in the fourth step in that pre-commit hooks
file how it should be installed.
So when you say this is, this is Python, then it will know, okay, so I should be using, for
example, pip to install this.
And if you have, for example, pyproject.tomo and you specify how it should be built, then
all of that just happens as it normally would.
It's just that pre-commit is doing it instead of you.
Yeah.
That's kind of, instead of you doing a pip install dashy dot or whatever, that it's
kind of figuring that out.
And I guess we haven't really talked too much about it, but when you pre-commit install, it
looks at the, this hooks YAML file and then it, it creates the environment and it downloads
all the packages the first time to kind of set it up.
Then it just runs over and over after that.
Right.
Yeah.
Unless you change something in your pre-commit config file, then it won't need to rebuild the
environment for this.
So if you keep the same version, then it's kind of like you said.
I installed this version of the package.
And as long as you don't say you need to update the package and it's kind of like a virtual
environment.
Okay.
You already have that.
There's no need to.
Yeah.
Excellent.
So your recipe is one, design the check function to turn it into a CLI, which there's some interesting
stuff in that one as well.
That's.
And I think that's kind of where the --fix comment comes into play.
Right.
So your logic, that check function, you should be able to say this was successful.
This was not successful as in stop the commit.
And then the CLI provides a very easy way to plug into that.
Maybe you want to say --fix or dash dash, you know, leave this type of file alone,
whatever kind of modification you want to do.
You can expose that in a CLI.
And that's also a quicker way to get started versus trying to, let's say, read the pipe,
find the pipe project.tongle, read it in, parse out things.
That's all stuff that can come later once you figure out exactly how you want your tool to
be configured.
Yeah.
Especially if it just has one or two arguments, it might not be necessary to be too, too over
the top with all the configuration.
And then you make it installable.
Basically, like you said, make it a package and then create the pre-commit hooks.
Yeah.
Well, those are the steps.
So I think write the function, that's pretty straightforward.
You just, whatever you want it to do, you just write a function that does it.
You do have an example in here about checking for valid file names and snake cased file names.
So things like it can't be just one letter and it has to be snake cased and so on.
Right.
But then to turn that into a CLI, there's a lot of options in Python these days, right?
You can click, you can type, but if you want something built in, yeah, if you want something
built in, argparse is pretty straightforward, right?
Yeah.
And I think also, I mean, if you look at the pre-commit hooks repo provided by pre-commit org,
a lot of them, or maybe all of them are just using argpars.
Because for most hooks, all you'll need to say is, I have an argument parser and it accepts
file names.
And at that point you have this boilerplate that you can just copy and you don't even
need to worry about configuring multiple, you know, different arguments.
It doesn't have to be too advanced with like sub commands and all that kind of stuff necessarily.
Yeah.
And then make it installable.
This is, you recommend a pyproject.toml, which yeah, for packages these days, that seems
pretty much the de facto standard, right?
Yeah.
And then what's nice is, yeah, you're using current things.
You're not relying on setup.py.
And also in there, there's a way to expose an entry point.
And that's line 24.
Yeah.
That's really nice.
I love entry points.
I think it's, I think they're massively underused in Python.
You know, people talk about how do I create a script that I can give it to somebody so they
can run something.
And that so often involves like, where is it?
Where is its associated files?
Where is its Python?
And where is its dependence?
All of that stuff you, if you just create a package and it has an entry point, you can
pipx install it or uv tool install it.
Or, and now you just have all these commands and people don't have to mess with all the Python
stuff.
Even if you know how to do it, you don't necessarily want to do that all the time.
Right?
Yeah.
And then it's just easy to, you can kind of call it from anywhere at that point.
Yeah, exactly.
So in this example, you give, you put a, a validate dash file name command and you
just point to, you know, what module and then what function to call.
And that's the CLI.
Yeah.
That's really nice.
And then of course that, that function in there is built and backed with arg parse.
So it all, it kind of all comes through a circle right there.
Yeah.
So it's like you, it's almost like you had created, you know, some command line utility,
like bash wise or something.
And you just have that available and it's hooks into your, your CLI.
I also want to call out on a 21 line 21, cause we talked about dependencies, right?
So anything you put in there, that's automatically will get grabbed when pre-commit installs.
So in this case, there's nothing.
And then the case of the exit stripper I mentioned, like we need to install pillow, right?
So this is how you can configure how pre-commit will grab everything.
And I also see it has, yeah, I see there's a requires Python version.
Does pre-commit help you get Python in any way?
Or is it just assume that there's a...
You need to have whatever languages you're relying on, you do need to have them installed
already.
Okay.
So in order for you to use this pre-commit hook on your machine, you'd have to have, for
example, Python 3, 10, 11, 12, something like that installed, given that it says 310
or greater.
So for example, like if you saw some hook that sounded interesting, but it's written in Go
and you don't have Go on your computer, you have to figure that out first.
That's a no-go.
It's a no-go.
All right.
Let's see.
Yeah.
And then last thing to do is you say, create the pre-commit hooks.yaml file.
And is this the thing that goes into your repo?
So when pre-commit sees it, it knows what to do?
Yeah.
So for example, in the exif stripper repo, there's this file exists.
So if someone uses exif stripper, they point to that repository.
And then when pre-commit goes and grabs it, it looks for this file, right?
And then the key things here, for one being language.
So language tells pre-commit, how does it try to install that?
So in this case, it says, oh, this is Python.
So then it knows, okay, pip.
The ID at the top, that's the name that you reference in the pre-commit config.
Like when you want to, like we saw check toml, check yaml in the beginning, those correspond
to entries in the pre-commit hooks yaml of that repository that they were being referenced
from.
So pre-commit can, so first finds this file, it can install, then it can see, oh, which
hook do you want?
Validate file name in this case.
And then how do I call this?
And that's entry.
And this is pointing to the entry point that we made, but it can be anything, right?
You could call rough and then add, you know, 20 different command line flags if you want.
And that can be your hook.
And that would be fine as well.
And what's very interesting here is it's optional, but it's the types one at the bottom.
So I talked before about XF stripper only running on images, right?
It'd be wasteful to have it look at toml and markdown, right?
If it's not going to do anything with it.
Can't find any XF information in the toml.
Yeah.
So this controls that.
So for example, this hook will only run on Python files.
And this logic, I'm blanking on the name of the tool that pre-commit uses to figure this
out.
But this is handled elsewhere.
So there's like certain names that you can use.
Right.
Some sort of category mapping over to these file extensions or these bombs at the beginning
of the file or whatever mean that it's this thing.
Exactly.
There is a very dangerous thing with this and that types is an and.
So if you say, if you wanted to do like this should run on Python and markdown, you can't
use this because it will look for files that are both Python and markdown and will not end
well.
Not too many of those exist.
Yeah.
There's a separate types or that you have to use.
That's like a little gotcha.
It's like an ORM sort of instead of a SQL statement.
Kind of you got to.
Yeah.
Those things always get weird.
Like import the or operator.
Like, okay.
Yeah.
Cool.
Okay.
That's actually that that is very good to know because it looks like a list of options.
It is.
Yeah.
But they combine.
So you might have something like it is a file and it's Python.
That might be one thing I've seen.
Right.
Okay.
Yeah.
Cool.
So if I wanted to have more than one hook, I could put it into one.
I could have more than one here.
Is that possible?
Yeah.
So this looks like a list.
Yeah, exactly.
It's structured as a YAML list.
So you just kind of could copy that block, paste the new one, and then just change whatever
fields you want.
And then that's now the second hook that you expose.
Right.
And working backwards, I suppose you just expose a different entry point potentially and then
just call it out or whatever you want.
Well, I mean, you could like maybe you have a validate file name and maybe you have another
one that's like validate long file names or something where you're like, now they have
to be this long.
And then it's just a shortcut for something else.
So it doesn't have to be a different thing.
Oh, yeah.
You just put an argument in there as a default kind of for people.
So we talked about args earlier and that was something the user could tweak.
Anything you put in here is essentially like it will always run with these.
So you could bake in certain things that have to happen.
Yeah.
Awesome.
I love it.
Okay.
We're pretty much out of time, but let's talk about one final thing.
Not this one.
Your Datamorph project.
Give a quick shout out to that before we wrap things up.
What do you think?
Sure.
So this project started related to the pandas workshop I had mentioned.
I wanted to have a visual to really drive home the point that we needed to visualize our
data because pandas very much data wrangling.
And after talking to people two hours about data wrangling and statistics, you can calculate
on tabular data.
Some people just feel like, oh, okay, we're done.
I mean, you know, we're done.
And that's definitely not the case.
And I was thinking about, and you had it on the screen before, but the data source doesn't.
So yeah.
So there was research in 2017 by Autodesk where they took the idea of Anscombe's Quartet, which
is, sorry, just a little bit above that, which is just a set of four, yeah, four data sets.
They share the same summary statistics.
So the mean in X and Y, the standard deviation in X and Y, and the Pearson correlation coefficient.
And they look very different.
And if you think of, naively, you think, well, I know the average and maybe how spread out
things are.
So I can kind of get a sense of what this data probably means.
But in reality, outliers and other weird things could just completely blow up those ideas,
right?
Yeah.
And so in 2017, they had developed this algorithm using simulated annealing.
So if you scroll down once more, where they take the dinosaur at the top and they use
simulated annealing to push the points.
Let me describe this really quick for just people listening.
So there's a matplotlib looking graph of some data points, and it has a certain standard
deviation, certain mean, et cetera.
But if you actually look at it, it looks like a T-Rex, right?
Something like this?
Yes.
Is that a decent enough description?
That's a perfect description.
Yeah.
So what the researchers have done is they use this simulated annealing algorithm to push
the points around.
So starting from that dinosaur and just moving the points ever so slightly in such a way where
the summary statistics are unchanged, at least to the two decimal places where they're currently
shown, and tried to make other shapes.
So some of the other shapes they have are a bullseye, a circle, lines slanted vertically
or a star.
And all of these can be formed from that dinosaur, some to varying degrees of success.
But they're visually recognizable, which is the point that is pretty important here, right?
So you cannot, as we said, rely on those summary statistics because you don't know.
Is it the star?
Is it the dinosaur?
Is it a line?
It could be anything.
And they also had animation that they included.
So basically, you could start from the dinosaur and then turn it into a circle.
And that's even more impractical because you realize at that point that it's not just the
dinosaur and the circle that have something in common, but it's the infinite number of
points arrangements that you can make between them that actually share that.
And so I wanted to explore if I could extend that to working for arbitrary data sets and also
different shapes.
So I found the research code and spent quite a bit hacking at it and even just trying to
get it to work for their example.
And that took quite a bit of time.
And then I had this idea of being that it was for a pandas workshop to take a panda and
turn it.
Initially, I wanted to turn it into the dinosaur.
I still have not found a good way to do that yet, but I also haven't been trying at all this
year on that, to be honest.
But I figured out how to, and by adding a lot of other things that didn't exist in the initial
algorithm, things like calculating bounds of the data and different metrics that I figured
out a way to get it to work regardless.
So I can give it a panda data set or a soccer ball and it can perform these transformations
and move the points around.
So on the screen, we have the first time I shared this publicly, what I had been working on,
it happened to be Easter.
So I made a bunny holding an Easter egg with the words, happy Easter off the side.
And it turns into two vertical lines all while preserving the summary statistics.
This is something I think makes it for a very good teaching tool in say like an introductory
statistics course to encourage people that they need to visualize.
There's an interesting study, I think called the hypothesis is a liability.
And they talked about taking students in a statistical analysis course and they split them into two.
And one set of students were just given the data set and say, here, explore, see what you find.
And then the other set were given a set of hypotheses to test.
And it turns out that the data is shaped like a gorilla.
And the students who were told here, test these hypotheses were five times less likely to even
realize that it was shaped like a gorilla because they never plotted it.
Yeah.
This is such a huge thing to like get people learning this early.
And the more shocking these visuals are, the better.
Yeah.
And I think these are super shocking, right?
Having T-Rexes and bunnies and go, you know, that bunny is, you know, equivalent.
And there's a continuous transformation from bunny to blob of dots with one outside dot, right?
That kind of stuff kind of surprise you, I think.
And one thing I see, especially when the dinosaur came out, but even when I posted some of my first
examples is you see people comment right away, wow, that there's something that's so cool that
that dinosaur is possible to do that with.
Like, no, no, no.
It's not, it's not just the dinosaur or just the panda.
It's really like anything.
And so the way this also works is that people can use their own data sets or they can add
something new.
And that's what I've had, that's what's what I've done this year in the two previous development
sprints that I had people just been, I did one in EuroPython and one in PyCon Taiwan earlier
this year.
And hopefully in Australia, we'll do some more.
But I had people add, for example, a target shape.
So what the, for example, the panda would turn into, we have a club, like the card suit,
which was quite a challenge, and the spade.
And I had already had the heart.
The heart is actually a trigonometric equation, which, you know, blew my mind at first.
There's actually a page I found on, I think, Wolfram Alpha, which was like, I want to say
like 10 or 15 different equations, trigonometric equations for different types of hearts.
And you can pick the exact type of heart you wanted.
Social media heart, the emoji heart, what are we talking about?
No, no, it was just like, this is longer, this is more curved.
Yeah, yeah, yeah, that's awesome.
But these are all now math problems when you think about that side of it.
So this could then be used maybe in a course where they want to focus on math, but also
some more coding.
So there's lots of different use cases, like just giving it the data.
And that's very much more just pure statistics.
But, you know, I think, and I've heard from a few teachers that, from what I presented,
that they're, it sounds like this would be something that they would like to use.
So hopefully that does happen.
If not, it's a fun thing to put in my slides.
And I did enjoy getting it to work.
Yeah, I didn't pull up any good videos for the YouTube video, but there's some really nice
animations of actually seeing it go from one to the other that you got.
And this is, you're doing a talk at PyCon Australia, and then you're doing a sprint on
this as well, right?
Coming up in November 22nd, about a month from now.
Correct.
So cool.
People can check that out if they happen to be at PyCon Australia and want to...
Well, I'll also be talking about it in San Francisco next week.
There won't be a sprint, but I will be talking about that.
So people can...
Okay.
It's not a PyCon.
Sure.
It's still cool.
All right.
Well, Stephanie, thank you so much for being here.
Let's wrap things up.
But I guess, you know, give us a final call to action for people maybe interested in pre-commit
hoax or other stuff that you're doing.
Yeah, you can find everything that we mentioned here and the projects on my website.
I'm putting much more effort into putting stuff on there this year now that I've rebuilt it.
So definitely check there and sign up for my newsletter.
Follow me on socials.
There's no links down here, but you can find them.
There'll be links on the episode page.
So we'll put them there.
All right.
Well, thanks.
Thanks for being here.
It's great to talk to you.
Thanks for coming on and sharing.
Thanks for having me.
Yeah.
Bye-bye.
This has been another episode of Talk Python To Me.
Thank you to our sponsors.
Be sure to check out what they're offering.
It really helps support the show.
Take some stress out of your life.
Get notified immediately about errors and performance issues in your web or mobile applications with Sentry.
Just visit talkpython.fm/sentry and get started for free.
And be sure to use the promo code talkpython, all one word.
This episode is brought to you by Bluehost.
Do you need a website fast?
Get Bluehost.
Their AI builds your WordPress site in minutes and their built-in tools optimize your growth.
Don't wait.
Visit talkpython.fm/bluehost to get started.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.