Byte My Python

Neovim TypeScript LSP Setup

2026-07-13T00:00:00+02:00

LSP is the technology that allows Neovim (and many other editor) to provide code completion, navigation, and contextual help. How it works is a server process that knows all the specifics of the language and is able to examine the files, and a protocol (LSP) which lets the editor communicate with that server the same way for all languages. LSP support is built in to Neovim, but you still have to enable and configure individual languages. The nvim-lspconfig package bridges the gap and includes default configuration for many languages.

What is special about TypeScript

TypeScript is a little different from other languages in that the standard tsserver language server does not implement the standard LSP (Language Server Protocol), but a variation of it (that actually predates LSP). So it needs a thin adapter to let the editor communicates with it as if it were a regular LSP server. There are two main options:

typescript-tools seemed really promising, but it does not seem to be actively maintained—the last commit was in 2025
ts_ls wraps typescript-language-server. It is more standard, but slower. For reasonably sized projects, it remains OK.

I was using typescript-tools until recently, but started running into some issues. When I noticed it was not maintained anymore, I decided to give the standard ts_ls configuration a try.

Enable `ts_ls` in Neovim

I use Lazy.nvim, so the config looks like this:

return {
    {
        -- pre-packaged LSP configurations
        -- see https://github.com/neovim/nvim-lspconfig/tree/master/lsp
        "neovim/nvim-lspconfig",
        config = function()
            -- for typescript-language-server
            vim.lsp.enable('ts_ls')
        end
    }
}

This relies on the executable typescript-language-server being present in PATH and on the project having a TypeScript installation.

If you have a warning about typescript-language-server not found, you typically need to do:

npm install -g typescript-language-server

Note: if you use a Node version manager, this is going to be specific to the version activated for the current project. So you may have to install a copy for each Node version.

Fixing a monorepo error

Beyond that, one common error I got with monorepos:

vim.schedule callback: .../neovim/0.12.4/share/nvim/runtime/lua/vim/lsp/client.lua:582: RPC[Error] code_name = InternalError, message = "Request initialize failed with message: Could not find a valid TypeScript installation. Please ensure that the \"typescript\" dependency is installed in the workspace or that a valid `tsserver.path` is specified. Exiting."

This happens because the language server looks for the TypeScript installed for this project. But in a monorepo, it might not find the right root. For example, GradeBee has this structure:

root
|
|-- pnpm-lock.yaml -> lock file for the workspace
|-- package.json -> workspace-level config, without TypeScript
+-- frontend
    |
    |-- package.json -> frontend dependencies, including TypeScript
    +-- tsconfig.json

The default ts_ls setup will detect the project root by finding the lock file—in this case, the root of the workspace—but TypeScript is not installed at that location.

There are two possible fixes:

Change the repo root detection. But then it still will not work to edit TypeScript files other than the ones under frontend.
Install TypeScript at the workspace level. It is a bit redundant, but it does make it clear that there is a tooling dependency on TypeScript.

I chose the latter option. Since I use pnpm, I can add it to the catalog key in pnpm-workspace.yaml:

catalog:
  typescript: ~6.0.2

Then, in both the root and frontend package.json, under devDependencies:

"typescript": "catalog:"

Links and final words

This is the current state, but the tools continue to evolve. Typescript 7.0 just came out and the compiler was rewritten in Go. Likely, Microsoft will follow this with an implementation of the LSP server in Go to address both the speed, and the compatibility concerns, making packages like typescript-language-server redundant.

Some links:

Organizing Agent Skills

2026-07-12T00:00:00+02:00

Most agents (Codex, Pi, OpenCode, and so on) will by default look for global skills under ~/.agents/skills, but it’s not an established standard. Claude Code and Cursor, for example, only use their own skill folder. The same applies within a project. Regardless of where skills are stored, the structure inside the skills folder is similar: individual folders that each contain at least a SKILL.md file, plus some reference documents or scripts. That part is getting standardized—see agentskills.io.

I started my skills repo a few weeks ago, in an attempt to organize a growing collection of both custom and third-party skills.

The challenges

My challenges are:

I periodically want to try out new third-party skills, to test a new workflow or just play with something new.
When I have a third-party skill, I want to be able to keep it updated...
...but I might also want to customize that third-party skill.
And then, in addition to that, I have my own custom skills to edit and maintain.

Managing installed skills

A new-ish tool from Vercel makes that possible, aptly named "skills."

It lets you:

install a new skill to the global or project folder, given a Git repo or a path within a repo;
track installed third-party skills in a .skill-lock.json file;
update those skills using npx skills@latest update;
list installed third party skills, based on the skill lock file: npx skills@latest ls -g
remove individual installed skills using remove (though only individual skills right now)

The .skill-lock.json has a package-lock.json feel but is more focused on updates. It’s useful because it lets me install and uninstall skills without messing with my custom skills. By keeping everything under Git, I can also make tweaks to third-party skills if I want to try small changes—being careful when updating them, since npx skills update will silently overwrite changes.

My layout

I organize my repo like this:

.agents/.skill-lock.json — the lock file from skills
.agents/skills/* — all third-party skills
.agents/skills/custom/* — all my custom skills

Agents will recursively discover skills, so you can use any organization you like within the repo. I haven’t done that yet, but I have seen developers divide skills between engineering-focused and research-focused work. There is not yet a way to activate a specific portion of the skills for a given agent session, though, so the organization is just to keep things tidy, not to increase agent performance.

Take away

This is a growing area of exploration and one where we as senior developers can really leverage our software engineering experience to boost the whole team. The need for composable, modular skills is crucial and good organization is paramount - I feel this is going to be a crucial need and one worth mastering.
I have been doing quite a bit of experimentation - as always, I mostly learn by doing - and will be sharing that in the future on this blog as well, so stay tuned!

GitHub PR Review "View Deployment" button, with Dokku

2026-07-01T00:00:00+02:00

I have been playing with review apps linked directly from GitHub PRs. The goal is simple: when someone opens a pull request, GitHub should show a link to a running preview of that branch.

I use Dokku for deployment of the app on the VPS and it works fine for the purpose of creating the one-shot, PR review application. It's all driven through CLI so very scriptable to fit into a GitHub Action workflow. I ran into a few snags I should document.

Give each PR a URL

The first thing to set up is wildcard DNS. I want each PR review app to get a predictable hostname, something like:

pr-123.example.com

Once the wildcard points at the Dokku host, the workflow can set the app’s domain during deployment:

dokku domains:set "${APP_NAME}" "${APP_HOSTNAME}"

Then Let’s Encrypt can issue the certificate for that review app:

dokku letsencrypt:set "${APP_NAME}" email "${LETSENCRYPT_EMAIL}"
dokku letsencrypt:enable "${APP_NAME}"

Nothing especially magical here, but it is nice that this is just Dokku commands. The PR gets a real HTTPS URL without provisioning a separate load balancer, ingress controller, or whatever other bit of cloud ceremony usually comes with “temporary app with HTTPS”.

Storage

If the app needs persistent files, create named storage for that PR’s review app and mount it into the container:

dokku storage:create "data-${APP_NAME}"
dokku storage:mount "${APP_NAME}" "data-${APP_NAME}" --container-dir /data

That gives the app a stable /data directory for the lifetime of the review app. It is still disposable, just not “lose every uploaded file on the first restart” disposable.

I also used another trick to initialize the storage from production. This gives me 2 things:

make sure the database migrations are going to work correctly on production data
make the production data available during review

In my case it was simple... in some cases there are references to external systems in the production data and things are not quite that easy.

TARGET=`dokku storage:info data-$APP_NAME --format json | jq -r .host_path`
sqlite3 /data/production/production.db ".backup ${TARGET}/${APP_NAME}.db"

(just before the dokku storage:mount command)

You might also use pg_dump to take a postgres backup, or rsync to just sync directory content.

Linking from the PR

The important part is on the GitHub side: expose the review app as an environment URL. That way GitHub puts the link right on the PR instead of making everyone dig through CI logs.

The workflow shape is roughly:

  deploy-review-app:
    name: Build, push, and deploy review app
    runs-on: ubuntu-latest
    environment:
      name: pr-${{ github.event.number }}
      url: ${{ steps.review_app_url.outputs.review_app_url }}

This will create an environment using the PR number, and associate an URL based on a step output (review_app_url). I created a step just for the output:

      - name: Generate review app URL
        id: review_app_url
        run: |
          # Pass the URL to the environment URL via step outputs
          APP_NAME="gradebee-pr-${{ github.event.number }}"
          echo "review_app_url=https://$APP_NAME.test.gradebee.app" >> $GITHUB_OUTPUT

One gotcha: GitHub secret masking can prevent the URL from showing up.

If the environment URL matches a secret value, GitHub may redact it and refuse to display it. So if the domain is stored as a secret, and the final URL contains that exact domain, the PR link can mysteriously disappear. In this case, the domain name itself is not really a secret, so it is better as a variable or plain configuration value. Keep the actual credentials in secrets; do not make the public hostname one.

One other thing: in my case I was fine with re-using the production secret for the review app (it has to do with user authentication, and I wanted to have the same users on both sides). But if you need to differentiate, you'll have to do a layered approach:

use a live environment with production secret
put the test secret at the repository level

This way when the production deployment runs it uses the live secrets, and the test deployments (which have a blank, just-created environment) will fall back to the repo secrets.

Cleanup

The other half of PR review apps is making sure they actually go away.

On the GitHub side, delete the temporary environment when the PR closes:

  - name: Clean up Github environment
    env:
      GH_TOKEN: ${{ secrets.ENV_ADMIN_TOKEN }}
    run: |
      gh api \
        --method DELETE \
        -H "Accept: application/vnd.github+json" \
        /repos/${{ github.repository }}/environments/pr-${{ github.event.number }} || true

For this to work you need to generate, and store in a secret ENV_ADMIN_TOKEN, a fine-grained "Personal Access Token" with read-write access to administration of the repository (doc link). The regular GITHUB_TOKEN is not enough.

On the Dokku side, destroy the app and its storage:

dokku apps:destroy "${APP_NAME}"
dokku storage:destroy "data-${APP_NAME}"

The storage cleanup is easy to forget, especially if the app itself disappears cleanly. But if every PR creates a named volume, those leftovers add up.

Takeaway

This is the kind of setup I like: a few boring shell commands glued into CI, rather than a bespoke review-app platform. GitHub already has a place to show the PR link, and Dokku has enough primitives to stand up the preview behind it.

The main thing is to be deliberate about what is disposable: app, domain, GitHub environment, storage. If you create all four on PR open/update, clean up all four on close. Otherwise your “temporary” review apps become a very quiet little infrastructure archaeology project.

Existing, public github projects can be a good source of inspiration, and the whole workflow as described above can be viewed on the GradeBee repository.

Loop Engineering: A Balance of Human and Agent

2026-06-25T00:00:00+02:00

Like many engineers these days, I spend a lot of time on so-called "loop engineering". The core question isn't really which model or tool is fastest—it's about the balance between human and agent. How do you get the agents to do the heavy lifting, while making sure you intervene at exactly the times where you, the human, add the most value?

I have been working on a skill based on the kanban-md tool. It already ships with a skill, but I wanted to tailor it to my workflow—run more controlled bites and know when it needed to involve me. And then I have some other skills I use once in a while or experiment with, most notably the superpowers skill, which just came up with some enhancements.

I ran the same fleshed-out plan two different ways.

First run

I did a test with a plan broken down into tasks, using the superpowers skill and the subagent-driven workflow. This is a lot of agent control and it decides exactly where to spawn sub-agents (which it does a lot).

It took a long time, almost 1h30 (it actually took a lot longer because I had to keep granting permissions, but that was my setup). TBH I'm not sure I would not have been faster coding it by hand. The permission friction is really my problem there (part of loop engineering).

For the record, the cost:

  Total cost:            $37.26
  Total duration (API):  1h 18m 28s
  Total duration (wall): 15h 42m 18s
  Total code changes:    1661 lines added, 170 lines removed
  Usage by model:
       claude-opus-4-8:  35.0k input, 116.0k output, 15.9m cache read, 3.3m cache write ($31.79)
     claude-sonnet-4-6:  197 input, 56.8k output, 10.0m cache read, 364.7k cache write ($5.22)
      claude-haiku-4-5:  425 input, 9.7k output, 1.4m cache read, 49.9k cache write ($0.2511)

Second run

Next I tried the same plan with the kanban-loop skill I devised, using kanban-md (it's in my skills repo). I first asked it to split the plan into sequential tasks (it created 3), then set it to work.

This took 1h4. It is really nice to be able to check the progress on the kanban board—that's a much healthier place for the human to sit in the loop: glancing at status rather than approving every step. Though I still need to get the agent to be a bit more communicative on its progress when updating the tasks. It was cheaper at $13.92 (I used pi-crust to calculate that, which is based on estimates—it might not be as accurate). To take with a huge grain of salt - this was absolutely not a formal benchmark. But I am not convinced Claude's strategy of aggressively spawning sub-agents always pays off.

Where the human actually added value

I asked an agent to run a comparison between the two branches. The first implementation let a significant bug through! But it had a more thorough implementation for one of the tasks. Apparently they were both solid (mind you, I hadn't looked at the code yet at this point).

Once I looked at the code myself I figured out very fast that both implementations were actually not great—because the plan sucked and I didn't review it carefully. So it goes to show: crap in, crap out.

But here's the loop-engineering lesson. Seeing the code made the problem obvious in a way that staring at the plan never did. The most valuable human intervention wasn't babysitting permissions or even reviewing the plan up front—it was reading the output once the agent had taken a first pass. For $13, that wasn't too bad. Sometimes there is a lot of value in letting the agent run ahead so you can find the rough edges in your own thinking more quickly than you would by reviewing a plan. There is another version of the skill that takes a hard break after each sub-task for the human to review, and I think, even though we would like for this to be fully automated and hands-off, that's a better proposition. Then it really comes down to where you want to place the seams, to have something that is reasonably to review, but large enough that you still have a good gain of time from it.

Switching to the Kitty Terminal Emulator

2026-06-23T00:00:00+02:00

This weekend I moved from Ghostty + Zellij to Kitty.

The main reason I wanted to drop Zellij was the overhead—especially in terms of memory (Ghostty regularly got up to 8GB of RAM, thanks to it). But there were still a few things missing when using Ghostty without Zellij... Customizing the click action for example was very difficult (in Zellij you can make a plugin to customize that, and I did one to be able to match the links in a way that fit my workflow better—miserly-link).

I tried another terminal, Kitty. You could say Ghostty is actually greatly inspired by Kitty which introduced extensions to the terminal such as the Kitty Image Protocol. Kitty is (like Ghostty) very fast, but is a bit more mature so it has many more features. A lot of the features are very keyboard-centric which I love.

Quick start and config

Unlike Ghostty which starts with a barebone config, Kitty ships with a very long config file with most of the defaults commented out.

A lot of the shortcuts work with a shared modifier called "kitty_mod". The default is ctrl+shift but I felt that was a little hard to reach and remapped it to alt+shift.

Some useful configs:

inactive_text_alpha -0.6
# fade text for inactive windows (focus indicator)

Cheat sheet - some of the most useful shortcuts

kitty_mod+Z = Zoom
Cmd+Enter or kitty_mod+Enter: new kitty window (= new pane within the current tab). This will place automatically in function of the selected layout. Remap the command to new_window_with_cwd to have it open at the current working dir.
kitty_mod+[ or ]: previous / next window
Cmd+t or kitty_mod+T: new tab
Shift+Cmd+[ or ]: previous / next tab
kitty_mod+H = edit scrollback in nvim (there is a sample configuration line to uncomment for that). The q key is remapped to exit.
kitty_mod+L = switch layout... there are 7 of them. You can configure which ones are enabled—more on that below.
kitty_mod+P,F = pick a file, insert in terminal (this is one of several uses of the “hint kitten”, more on that below)
kitty_mod+F3 = command palette
kitty_mod+F8 = swap 2 windows, that's pretty handy
kitty_mod+1,2,3,4 = go to window

Layouts

Something I used a lot in I3. Instead of deciding whether you want to split a window to the right or the bottom, the layout decides that for you. There are 7 layout total. I just kept 4:

Fat (large window at top, split vertically at bottom)
Tall (large window on left, split horizontally on right)
Grid
Stack

The stack layout is very handy for zooming a single window, but I mapped that one separately to Mod+Z. There is a split layout which lets you do windows however you want, with vertical / horizontal splits, but I'm going to try and live without that one.

Kittens

Extensibility is the strong point of Kitty, I feel, when comparing with Ghostty. Ghostty has a very nice out of the box experience but still a little immature in terms of extensibility. This is done through "kittens". There are a number of super useful ones, out of the box.

Choosing a font

Use kitten choose-fonts to pick a font. I tried a few, settled on Victor Mono, downloaded with Homebrew:

brew install font-victor-mono

In some (most?) terminals you are advised to install a “nerd font”, which has some extra icons. But kitty will actually transparently detect these icons and render them so this is not necessary.

Open anything

Kitten hints, this shows a letter or number by everything that looks like a file, so you can open it in your favorite editor, little like Vimium. I added a shortcut to open in nvim (mapping q to quit, like in the scrollback pager edit action):

map --allow-fallback=shifted,ascii kitty_mod+p>e kitten hints --type path --program="launch --type=overlay nvim --cmd 'nnoremap q ZQ'"

This is a really awesome feature for the mouse-adverse crowd.

Notifications

Another seemingly minor but very useful feature, "kitten notify" replaces terminal-notifier but is smarter about knowing which kitty window triggered the notification. I use this as completion hook for my coding agent.

Final (first) impression

After a weekend of use, performance is good, I'm getting more and more used to the different shortcuts and way of working things, and keep finding little gems. I've only scratched the surface - I'll keep updating this as I find more of them!

Why Git's Linear Log Can Lie to You

2026-01-30T00:00:00+01:00

When your team uses merge commits instead of rebasing, git log can deceive you. That flat, linear list of commits doesn't tell the whole story — and if you're trying to figure out what's actually in a release, you might draw the wrong conclusions.

Here's a scenario I ran into recently.

The Setup

I opened a PR yesterday for a bug fix on our release branch. The PR was approved but only merged today. Meanwhile, yesterday the CI pipeline built and tagged a release image from the same branch.

Looking at git log, you might think my fix made it into that image:

9c2600b (HEAD -> release/2.9.0, origin/release/2.9.0) Merged in fix/user-session-timeout (pull request #1532)
faf20e5 Merged in fix/dashboard-label-update (pull request #1528)
31b3a38 (tag: release/2.9.0-build-1678) Merged in fix/array-null-check (pull request #1529)
a992bf9 DASH-456 update E2E with fixed label update
72da568 (origin/fix/user-session-timeout) SESSION-789: set up connection pooling
7ca61c1 SESSION-789: fix for mock use in unit tests
a598fd9 SESSION-789: work around timeout in tests
346236a SESSION-789: ensure session is not eagerly bound
3d2d424 Merged in fix/picklist-value-type (pull request #1530)
095b0c1 fix PICK-234 incorrect value for picklist

See the problem? The commits from my branch (72da568, 7ca61c1, etc.) appear before the release tag at 31b3a38. A quick glance suggests they're included in the build.

But they're not.

The Reality

Run git log --graph --oneline --decorate instead, and the true picture emerges:

*   9c2600b (HEAD -> release/2.9.0, origin/release/2.9.0) Merged in fix/user-session-timeout (pull request #1532)
|\  
| * 72da568 (origin/fix/user-session-timeout) SESSION-789: set up connection pooling
| * 7ca61c1 SESSION-789: fix for mock use in unit tests
| * a598fd9 SESSION-789: work around timeout in tests
| * 346236a SESSION-789: ensure session is not eagerly bound
* |   faf20e5 Merged in fix/dashboard-label-update (pull request #1528)
|\ \  
| * | a992bf9 DASH-456 update E2E with fixed label update
| * | 7b14e0c (origin/fix/dashboard-label-update) DASH-456 fix label update issue
* | |   31b3a38 (tag: release/2.9.0-build-1678) Merged in fix/array-null-check (pull request #1529)
|\ \ \  
| |_|/  
|/| |   
| * | f85fe64 (origin/fix/array-null-check) ARRAY-567: handle undefined array
| |/  
* |   3d2d424 Merged in fix/picklist-value-type (pull request #1530)

Now you can see that my fix branch runs parallel to the main line. The merge commit that actually incorporates those changes (9c2600b) comes after the release tag. My fix wasn't in the build.

Why This Happens

Without rebasing, feature branches keep their original commit timestamps. When merged, Git interleaves these commits chronologically in the flat log view — even though they weren't actually part of the main branch until the merge commit.

The flat git log essentially overlays the log from the feature branch onto the trunk. You end up reading multiple parallel histories stacked on top of each other based on date, which can be deeply misleading.

Reading the Log Correctly

Two approaches to get the truth:

1. Use --graph to see the actual branch structure:

git log --graph --oneline --decorate

2. Use --first-parent to see only the merge commits on the main line:

git log --first-parent --oneline

This shows you exactly what was merged and when, ignoring the individual commits within each branch. The order of these merge commits can be trusted.

Most Git hosting platforms (GitHub, GitLab, Bitbucket) also show this graphically in their commit views — branches in flight run parallel until the merge commit actually brings them in.

A Case for Rebasing

This experience reinforced why rebasing is worth the effort, especially for longer-lived branches (more than a week or so):

When syncing with the main branch: Rebase rather than merge. Don't use the "Sync" button in Bitbucket or similar — it creates a merge commit that interleaves histories. Instead:

git fetch origin
git rebase origin/main

Before merging your PR: Squash trivial commits (typo fixes, "oops" commits, etc.) into logical units. This keeps the history readable and makes each commit meaningful.

Why it matters:

git log reads linearly and accurately represents the state of the repo at each point in time
git bisect works correctly — you can binary search through commits to find regressions without getting lost in parallel histories
Rolling back to a pre-merge state is straightforward
Code review is easier when commits tell a coherent story

When you merge without rebasing, the histories become intermingled. Going back to the pre-merge version of a branch or using git bisect to analyze regressions becomes challenging because you're navigating through multiple parallel timelines that have been flattened into one.

Takeaway

If your team uses merge commits, remember that git log without flags can mislead you about what's actually in a given commit or tag. When debugging "is this fix in production?", reach for --graph or --first-parent to get the true story.

And if you want to avoid these headaches altogether, consider rebasing your feature branches before merging — especially the long-lived ones. Your future self (and your teammates) will thank you when bisecting a regression at 2 AM.

The QuerySet Database Binding Gotcha

2026-01-27T00:00:00+01:00

When working with Django's multi-database support, there's a subtle issue that can trip you up: QuerySets remember which database they're bound to, and this can lead to unexpected conflicts.

The Setup

In our codebase (Prancer), we have custom managers that use using(self.db) to bind a QuerySet to a specific database. This is a common pattern when you want to route queries to a replica for read-heavy operations.

class SomeCustomManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset().using(self.db)

The key thing to understand is that once you call .using(), that QuerySet is bound to that database from that point forward. Any further filtering or chaining will still use that database.

The Problem

We ran into trouble with PatientDataSource (and generally when using HistoryTableRule). These components accept a QuerySet that is constructed at initialization time — essentially acting as a configuration object.

The issue: this QuerySet is effectively global state. It gets created once, bound to the 'default' database, and then reused across requests.

Later, when a query actually executes:

Filters are added to this pre-constructed QuerySet
The result is used to filter the main Data Source QuerySet
But the main QuerySet is bound to replica0

And now you have a conflict: you're trying to combine QuerySets that point to different databases. Django does not handle this gracefully.

Why This Happens

The root cause is the disconnect between:

Initialization time — when the QuerySet is constructed and bound to default
Execution time — when the actual query runs and should use replica0

If you're building reusable components that accept QuerySets as configuration, you need to be aware that the database binding travels with the QuerySet.

The Fix

The solution depends on your specific case, but generally:

Defer QuerySet construction until execution time, rather than initialization time
Pass a callable (like a lambda or method reference) instead of a pre-built QuerySet
Explicitly rebind the QuerySet with .using() right before combining it with other QuerySets - this lets Django re-evaluate the database context

The last option is often the quickest fix, but the first two are cleaner architecturally — they avoid the "global state" problem entirely.

Takeaway

When you see .using() in your codebase, think carefully about when that QuerySet is created versus when it's used. If there's a gap, you might be setting yourself up for a database routing conflict.

Testing Lambda Functions with LocalStack and pytest

2026-01-25T00:00:00+01:00

When integration-testing code that calls AWS Lambda, you don't always want to hit the real thing. LocalStack gives you a local, containerized version of AWS services — including Lambda — that you can spin up, use, and tear down as part of your test suite.

Here's a pattern I've been using: create a Lambda function on the fly inside a pytest fixture, wait for it to become active, then hand off a boto3 client to the tests. Sounds straightforward, but there are a few gotchas.

The fixture

@pytest.fixture(scope="module")
def update_schema_lambda(testdata_dir: Path, config: Config) -> Generator[LambdaClient, None, None]:
    with (
        LocalStackContainer()
        .with_services("lambda")
        .with_volume_mapping(str(testdata_dir), "/testdata")
        .with_volume_mapping("/var/run/docker.sock", "/var/run/docker.sock")
    ) as container:
        endpoint_url = container.get_url()

        result = container.exec(
            """sh -c 'cd /testdata && zip /tmp/lambda.zip mock_update_schema.py'"""
        )
        assert result.exit_code == 0
        result = container.exec(f"""
        awslocal lambda create-function \
    --function-name {config.update_metadata_function} \
    --region us-east-1 \
    --runtime python3.9 \
    --handler mock_update_schema.handler \
    --memory-size 128 \
    --zip-file fileb:///tmp/lambda.zip \
    --role arn:aws:iam::000000000000:role/lambda-role
                       """)
        assert result.exit_code == 0
        result = container.exec(f"""
        awslocal lambda wait function-active-v2 \
    --region us-east-1 \
    --function-name {config.update_metadata_function}
    """)
        assert result.exit_code == 0
        client = boto3.client("lambda", endpoint_url=endpoint_url)
        yield client

Let's break it down.

Starting the container

The LocalStackContainer (from testcontainers) spins up LocalStack with only the services you need. Here we're mounting two volumes:

/testdata — where the Lambda handler code lives
/var/run/docker.sock — LocalStack needs this to run Lambdas in their own containers (Lambda-in-Docker)

Creating the Lambda

Inside the container, we use awslocal (LocalStack's wrapper around the AWS CLI) to zip up the handler and create the function. Nothing too surprising here, except for one thing: always pass --region.

LocalStack can be picky about regions. If you omit --region, it might default to something unexpected, and then your boto3 client (which defaults to us-east-1) won't find your function. Save yourself the debugging headache and be explicit.

Waiting for the Lambda to be ready

This is the part that bit me:

awslocal lambda wait function-active-v2 \
    --region us-east-1 \
    --function-name {config.update_metadata_function}

Lambda functions aren't invocable immediately after creation. The wait function-active-v2 command blocks until the function transitions to the Active state. Skip this, and your first invoke will likely fail with a cryptic error about the function not being ready.

Getting the endpoint URL

Once the container is up, container.get_url() gives you the endpoint URL to pass to boto3:

client = boto3.client("lambda", endpoint_url=endpoint_url)

This tells boto3 to talk to LocalStack instead of real AWS. Without it, your tests would try to hit production (and probably fail with auth errors, or worse, succeed and cost you money).

Putting it together

With this fixture, any test that depends on update_schema_lambda gets a fully-configured boto3 Lambda client pointing at a fresh, ephemeral LocalStack instance with the mock function already deployed and ready to invoke.

The container tears down automatically when the module's tests finish, thanks to the context manager.

One last tip: if your tests are slow, consider scope="session" instead of scope="module" — but be aware that tests will then share state, so make sure your Lambda is idempotent or reset between tests.

Getting Started with MicroPython (Raspberry Pi Pico)

2026-01-23T00:00:00+01:00

If you’ve just unboxed a Pico (or any similar MicroPython-capable board), the first hour can be surprisingly fiddly: flashing the right firmware, getting a REPL, and figuring out how to copy a script over without reinventing the wheel. I’m just getting started with MicroPython myself, so this post is basically my “here’s what worked” notes from that first setup.

One thing that helped early on was doing a quick “sanity check” with the Pico C/C++ SDK: build and flash a tiny example once. It doesn’t teach you Python, but it does confirm your USB connection and board setup are solid.

After that, here’s the shortest path I found to a working MicroPython REPL.

Flash MicroPython (UF2)

Download the MicroPython firmware (a .uf2 file) from the official docs: Raspberry Pi MicroPython documentation. Make sure you pick the firmware for your exact board.
Unplug the board (if it’s connected).
Hold BOOTSEL (the small white button on top), then plug the board back in. It should mount as a USB drive.
Copy the .uf2 onto that drive. The board will reboot automatically once the copy completes.

Connect to the REPL

After flashing, connect to the serial REPL - the interactive Python command line:

minicom -o -D /dev/ttyACM0

Then you can run Python command (if you don't see a prompt at first, try just pressing enter to activate the REPL). Good to know: to exit Minicom, press Ctrl+A then X.

If you hit permission errors accessing /dev/ttyACM0, add your user to the dialout group and re-login:

sudo usermod -aG dialout "$USER"

First test: turn on the built-in LED

At the REPL, try turning on the built-in LED:

from machine import Pin

led = Pin("LED", Pin.OUT)
led.value(1)

Copy scripts to the board with `rshell`

To upload a script (and have it run automatically on boot), copy it to main.py using rshell:

# first, install rshell
uv tool install rshell

# copy a file to run automatically
rshell cp my_file.py /pyboard/main.py

# inspect files on the device
rshell ls /pyboard

Restart the device, and it will run the program. To restart, the most straightforward is unplug / replug, but you can also connect to the REPL using minicom and press Ctrl+D, or use machine.reset():

import machine
machine.reset()

That’s it for the initial setup. Next up I want to explore the ecosystem a bit more (sensors, simple displays, maybe a small servo project). So far, MicroPython looks like a great way to make programming feel tangible quickly—and a really promising tool for introducing kids to coding and robotics without getting stuck in tooling.

Managing Python Monorepos with uv Workspaces and AWS Lambda

2026-01-19T00:00:00+01:00

uv workspaces are a super tool when developing interconnected Python packages, especially in mono-repo setups. uv will set it up pretty automatically if you have a pyproject.toml at the root of the repo and you run uv init inside a subfolder.

What this means is: - only one venv will be set up, with all dependencies for all projects (this is pretty nice for the IDE because you don't need to keep switching venvs, but note the important caveat below) - you can reference a local project by adding it in [tool.uv.sources], and uv will automatically install it as editable (so you won't need to build each time you make a change and keep reinstalling your project):

[tool.uv.sources]
common_logging = { workspace = true }

one caveat: if your projects have conflicting dependencies, uv will fail the install! For microservices, it is probably best to keep dependencies on compatible versions anyway
another caveat: since there is only one venv, it's possible the IDE will detect a package that is not really in the project dependencies, and you accidentally import it! This can easily go unnoticed during local development.
for that, you might be better off forgoing workspaces and using a path dependency:

[tool.uv.sources]
common_logging = {
    path = "../common/logging", editable = true
}

I have decided to keep using uv workspaces, but we have a CI check to catch that type of problem before it reaches production.

Running uv in a workspace

From the root of a workspace, uv sync will install only the workspace-level packages. Likewise uv run will not install all the packages. This is not a way to specify common dependencies, because they will only be installed if the workspace root is installed (so for example uv export inside of the package will not include those "shared" deps).

Now, as you go into a package and run uv sync, uv will add the dependencies from that package to the venv... so eventually you will have everything installed. But you can't be sure that is always the case. This is also nice if you want to install deps for only one package, before running tests in CI, to make sure it does not accidentally depend on the other packages' dependencies.

To run and include all package dependencies (needed for example for a type checker!), use

uv run --all-packages

Use in Docker build

The following examples assume AWS Lambda container images, which explains some of the paths (/var/task, /var/lang/lib) and the base images used. The general approach still applies to non-Lambda containers, but paths and base images would need to be adjusted.

There are two catches:

you don't want to install all the workspace dependencies necessarily — some are not shared among all your microservices
you don't want to install the local dependencies in the same layer as the core dependencies, because the core dependencies are less likely to change (and, more importantly, are much bigger)

This is especially important for Lambda images, where image size and layer caching directly impact cold start performance.

I struggled a bit with the latter, but in fact uv makes it rather easy. It's not super well documented though — there are many different ways to get there that will sort of work but leave you with some cruft in the final image.

First attempt using uv sync

you do need to copy all the pyproject.toml files and the single uv.lock file that you are going to need. This includes all your local dependencies, and that makes sense — if you changed a requirement in a local dependency, it's now part of your core deps, and you can include it in the base layer

COPY pyproject.toml uv.lock /build
COPY services/common/logging/pyproject.toml /build/services/common/logging/
COPY services/${SERVICE_NAME}/pyproject.toml ./

then, install the core deps, making sure you specify the --package argument to install only for that part of the workspace, and the --no-install-local argument to prevent including other workspace members:

RUN uv sync --frozen --no-install-local --no-dev --package ${SERVICE_NAME}

next, the local dependencies… uv will figure out which ones of those are needed for the project based on the workspace graph:

COPY services/common /build/services/common
RUN uv sync --frozen --no-editable --no-install-project --no-dev --package ${SERVICE_NAME}

and finally, the code for this service:

COPY services/${SERVICE_NAME} ./
RUN uv sync --frozen --no-editable --no-dev --package ${SERVICE_NAME}

I refined this a little bit, because the above will leave two copies of the source code (one under /build/services and one under the venv), which is undesirable in a Lambda image where size matters:

# Copy dependency files: workspace, and service project
COPY pyproject.toml uv.lock /build
COPY services/${SERVICE_NAME}/pyproject.toml ./

# Install core dependencies
RUN uv sync --frozen --no-install-local --no-dev --package ${SERVICE_NAME}
ENV PYTHONPATH=/build/.venv/lib/python3.14/site-packages

# Install local dependencies
RUN --mount=type=bind,source=services/common,target=/build/services/common \
    uv sync --frozen --no-editable --no-install-project --no-dev --package ${SERVICE_NAME}

# Finally, copy the project source
COPY services/${SERVICE_NAME}/src /var/task

Now, the downside is that you are left with a few unneeded things in this final image: there are some leftovers from the venv and the uv command itself. I was also worried that running the uv command directly in the final image could leave me with some surprises, especially in a production Lambda environment.

Second attempt using pip

To work around the above problem, I install the dependencies in two steps into separate folders. Then, I do another stage in the Docker build to collect all these dependencies into a clean final image.

Despite the title, this still relies on uv for dependency resolution — pip is only used for the final installation step, which keeps the runtime image simpler.

# --- Build Stage ---
FROM public.ecr.aws/lambda/python:3.14-arm64 AS builder

# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

ARG SERVICE_NAME

# Set up the working directory
WORKDIR /build

# Copy dependency files: workspace, and service project
COPY pyproject.toml uv.lock /build
COPY services/${SERVICE_NAME}/pyproject.toml /build/services/${SERVICE_NAME}/pyproject.toml

# Install core dependencies
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --no-install-local --no-dev --package ${SERVICE_NAME}

# Install common dependencies
RUN uv export --package ${SERVICE_NAME} --no-editable --no-dev --frozen --format requirements.txt | \
    grep '^./services/common' > common.txt
RUN --mount=type=bind,source=services/common,target=/build/services/common \
    uv pip install --no-deps -r common.txt --target /build/common

# Application code (we could use uv pip install here too, but this is simpler)
COPY services/${SERVICE_NAME}/src /build/app

FROM public.ecr.aws/lambda/python:3.14-arm64 AS final

# Copy core dependencies from venv
COPY --from=builder /build/.venv/lib /var/lang/lib

# Copy common
COPY --from=builder /build/common /var/task

# Application code
COPY --from=builder /build/app /var/task

# Set the CMD to your handler
CMD [ "ingestion_worker.main.handler" ]

This results in a smaller, cleaner final image that contains only what Lambda needs at runtime.

Exploring Modern Python Type Checkers

2026-01-12T00:00:00+01:00

I’ve been playing around with alternative Python type checkers and language servers, looking beyond the more legacy options to find a setup that’s powerful without getting in the way. Along the way, I compared ty and Pyrefly, tweaked extensions and inlay hints, and ran into a few interesting differences in behavior, ergonomics, and typing semantics. The result isn’t a definitive winner yet, but it did surface some useful insights—and a couple of gotchas about Python typing that are easy to miss.

Both extensions were very easy to install from the Extension Marketplace in Cursor.

I disabled the “Python” extension from Anysphere (keeping the ms-python one). Anysphere’s extension is essentially pyright, which was doing double duty with ty.

For ty, I also disabled the language server, as suggested in the documentation:

{
  "python.languageServer": "None"
}

After that, there were still some logs being output:

ty
ty language server

There were also inlay hints showing inferred types. They’re somewhat useful, but overall a bit too much visual clutter, so I disabled them. I can still see variable types via tooltips when needed.

I also tried Pyrefly. The setup was similar, but I didn’t need to manually disable the language server—Pyrefly automatically disables Pylance. It offers a few more customization options. One thing to note: by default, it will not display type errors unless a configuration file is found in the project. This can be forced via the “Display Type Errors” setting.

Where Pyrefly’s language server was better:

In one instance, it was able to automatically locate missing imports when I used Cmd + .. This involved a type alias, where ty took a few more shortcuts.

Inlay hints comparison

Ty was a bit more obnoxious with inlay hints out of the box, while Pyrefly tended to include them only when they were genuinely useful. For example:

def merge_data(self, source: str, dest: str):
    source = check_sql_id(source)
    self._validate_source_schema(source)
    stmt = build_update_statement(self.con, source, dest, namespace, scope)

Both tools correctly inferred the type of source as SqlIdString (rather than the original str), but:

ty added an inlay hint
pyrefly did not, unless I was also renaming the variable

Ty also tended to union types with | Unknown when using list comprehensions or list literals, which was pretty weird and annoying. Thankfully, there’s already an open issue for this, so I’ll keep tracking it.

Pyrefly had one case where it refused to redefine a variable’s type via annotation. That’s valid Python, though arguably questionable style.

Finally, I use Protocol for my interfaces on the project, and Pyrefly had generally more helpful messages when an interface was not correctly implemented.

Both of those are easy to integrate in the CI workflow, and they play nicer than pyright with Python's package manager.

Overall, I’m more aligned with Pyrefly, but I’m sticking with ty for now—mainly because I resonate more with Astral’s values than Meta’s. I’ll revisit that choice if ty starts to cause friction.

One final typing takeaway: a type alias is not the same as NewType. A type alias doesn’t provide real type safety—it’s just a nickname for an existing type.

Stop Fighting Circular Imports

2025-10-05T00:00:00+02:00

The code is good, but you are missing the type hints!

Oh, I know, but as soon as I add them, I get these circular import errors!

Mmm, that's often a smell, let's dig a little

Sounds familiar? Yet sometimes even when digging in, the design is fine, and the modules really need to import each other, especially if you want to keep things simple and not introduce a Java-style interface hierarchy.

Turns out, modern Python has a solution for this, in fact, several of them. Let's check it out :)

Here a simple example:

# post.py
from models.user import User

class Post:
    def __init__(self, author: User):
        self.author = author

# user.py
from models.post import Post

class User:
    def __init__(self, name: str, posts: list[Post]):
        self.name = name
        self.posts = posts

This gives:

ImportError: cannot import name 'Post' from partially initialized module 'models.post' (most likely due to a circular import) (/home/nico/scratch/models/post.py)

So, first solution, from PEP-563, was to be able to do the annotations as strings.

class Post:
    def __init__(self, author: "User"):
        self.author = author

But that's not enough for most tools, you also need to tell it where to find User... yet you can't just import it as it would be back to a circular import, so you use a special "TYPE_CHECKING" guard:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from models.user import User 

class Post:
    def __init__(self, author: "User"):
        self.author = author

And finally you can turn on these string annotations for the whole module using a future import:

from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from models.user import User 

class Post:
    def __init__(self, author: User):
        self.author = author

Now, as often in Python, there are a few layers added over time. For the full details, you'll want to read the PEPs:

start with PEP-563, which is rather simple, and is available today (and as early as Python 3.7)
then move on PEP-649 and its little sister PEP-749. They change the way this is done, but this won't be available until Python 3.14.

Then, under the hood things will be quite different and it won't use the "string" annotations anymore. For most usage (when annotating code for static type hints), you will just be able to drop the __future__ import.

My 5 Best Tricks using Pytest with Django

2024-04-25T00:00:00+02:00

Test driven development is THE way I code, without compromise. Maybe if I am doing a quick spike of throw away code... or some strictly UI code that I know is going to evolve a lot... but anything on the backend, I always start with setting up tests. Often I work with legacy or semi-legacy code bases that don't have good tests and a big part of the work is setting those up - after that the code practically writes itself :)

Anyway, back to pytest. Pytest is that awesome library and test runner for Python that makes it easy (well, easier) to write small and readable tests, decoupling the part that is going to create all the bits of data that the test needs, from the test code itself. It does this using 2 neat features: fixtures, and test parametrization. It also has a quite large ecosystem for integrating with application frameworks, like of course Django.

Of course, all the examples in the Django documentation, refer to the TestCase classes they provide, and it's not always straightforward to adapt to pytest cases. The big problem I ran into with TestCase is that they tend to lead to large, self-contained classes that become very hard to understand. You can get away from that by crafting utility classes... or mixins for your test cases, etc... but if you have complex relationships between your models this can devolve into a spiderweb since your utility methods have to account for the potential intricacies of the tests. Fixtures are one way to address that, and the big advantage they have is that they can be locally overridden as the test needs it - I will get to that.

I had used pytest before, quite a bit, but I ran into some unique difficulties with Django that I wanted to document. I am not going to go into the details of pytest itself because there is already quite a lot about it, except in specific cases where the interaction with Django is important.

Install the pytest-django package

Probably the first hit when you googled "pytest" and "django". Install that. It will give you a bunch of fixtures that you can use to test the Django application. The full documentation is at https://pytest-django.readthedocs.io/en/latest/ and worth a read. The fixtures I use the most are the ones related to the database (db, transactional_db), there are also some wrappers for the custom assertions normally available through Django TestCase subclassing (like assertRedirects, assertTemplateUsed, etc) and some Django helpers that are made available as fixtures instead (like rf for RequestFactory, client, or django_assert_num_queries)

One thing you have to do is to point pytest to your Django settings module. You can do that by setting DJANGO_SETTINGS_MODULE in your pytest.ini file (if it does not exist, create it in the root of your project):

[pytest]
DJANGO_SETTINGS_MODULE = myproject.settings

Using --reuse-db

For us it was a big deal to be able to reuse the database between tests - initializing the schema from scratch takes about 15 minutes on a good day. The --reuse-db flag is a lifesaver. It will keep the database around between test runs, and only reinitialize the schema if you add migrations. Keep in mind that if you modify migrations, it will not pick that up - you have to either manually reset the test database, or use the --create-db flag to force a new database to be created.

You can add that flag in your pytest.ini file:

[pytest]
addopts = --reuse-db

Understand transaction scope and database fixtures

This is a really important one to understand because the tests are less "self-contained" than with TestCase. By default, pytest will prevent any database access. This is a good thing, because it makes the tests run faster, and it makes it easier to reason about the tests. But sometimes you need to access the database, and that's where the fixtures come in. pytest-django provides 2 different fixtures for this: db and transactional_db. Internally this will map to Django TestCase and TransactionTestCase classes. The difference is that db will run the whole test within a transaction and roll it back at the end of the test. transactional_db will start by serializing the state of the whole database, then run the test without a transaction, and at the end it will reset the database using the saved state. This is useful if you need to test code that will manipulate transactions explicitly or use multiple database connections. With a large database it's also very slow. Because pytest encourages small, focused tests, and can use test parametrization to quickly generate a large number of tests, transactional_db can be very slow. Another problem of transactional_db is that if the test crashes in an abnormal way (say, your computer crashes, or you run in a debugger and stop the session), the cleanup will not run, and the database will be left in the dirty state. Again these problems are not specific to pytest, they also exists with TestCase and TransactionTestCase, but as pytest lets you write more tests more easily you are more likely to run into them.

Generally, use no database if you can afford it (but often Django apps are designed to use the ORM extensively so this is not always possible), or use the db fixture. Use the transactional_db fixture only if you absolutely need to.

If you have fixtures that need to access the database, you have 2 choices:

you can have db or transactional_db as a dependency to the fixture. But then this means the fixture, not the test, will decide how the db is managed (if multiple fixtures request different database fixtures, pytest will use transactional_db).

you can request the fixture from the test itself. I prefer this method because it makes it clear that the test is using the database. You can put a mark at the module level, if all tests within the file are going to need the database.

Use django_db_blocker for custom database setup

This is where you absolutely need to master the concept of fixture scope. By default, fixtures are function-scoped, which means that they are created and destroyed for each test function. This is also the case for the db and transactional_db fixtures, which means you cannot have a fixture with a wider scope that uses the database. This is a problem if you want to share fixtures between tests. You can then use django_db_blocker to manually control the database access. This is a context manager that will allow you to run code that access the database. You also need to request the django_db_setup fixture in that case, which runs once per session and ensures the migrations are run on the test database. Here is an example:

import pytest

@pytest.fixture(scope='module')
def my_model(django_db_setup, django_db_blocker):
    with django_db_blocker.unblock():
        # this runs once per module, but will be seen by all the tests
        some_complicated_model_setup_that_takes_a_long_time()
    yield
    with django_db_blocker.unblock():
        # I need to do this since nobody is going to do it for me!
        some_complicated_model_teardown()

def test_my_model(my_model, db):
    # I run within a transaction
    assert my_model.objects.count() == 1
    # and at the end of the test, the transaction is rolled back

def test_my_model_more(my_model, db):
    # I also run within a (different) transaction
    assert my_model.objects.count() == 1

Normally, with a function-scoped fixture, the my_model fixture would be created and destroyed for each test. But with the module scope, it only runs once. Note how I still have to request the db fixture in the test, because I exited the context manager before the test, so that the tests are still isolated from each other.

I don't really like this code because it is not very readable and it makes the tests less isolated. But if you absolutely have to do a long-running setup you might need this. There is no equivalent to the class level setUpTestData of Django, because this conflicts with the concept of fixture scope in pytest - you have to achieve the same result by using a module-scoped fixture, but this also requires you to do a little bit of work to ensure the database is properly set up and torn down. The next tip will show a safer way to do this.

Use a larger-scope transaction for complex initialization

By combining the above 2 tips, it is possible to offer a fixture that runs at a higher than function scope and provides a transactional context. The tests will then operate within their own, nested transaction, and will still be isolated from each other. This is only possible if the database backend supports nested transactions (or savepoints, as Django will automatically use that within a transaction). Which is the case for most popular databases supported by Django.

You use django_db_blocker to open the transaction at a higher scope, yield within it so that your test function can run and create a nested transaction within that scope, then close the transaction at the end of the test (simply by exiting the context manager).

Here is an example:

import pytest
from django.db import transaction

@pytest.fixture(scope='module')
def my_model_transaction(django_db_setup, django_db_blocker):
    with (django_db_blocker.unblock(), transaction.atomic()):
        # this runs once per module, but will be seen by all the tests
        some_complicated_model_setup_that_takes_a_long_time()
        # yield within the transaction context
        yield
        # no need for teardown, since we use a transaction

def test_my_model(my_model_transaction, db):
    # I run within a nested transaction
    assert my_model_transaction.objects.count() == 1
    # and at the end of the test, the transaction is rolled back
    # (but not the outer transaction which has the model setup)

def test_my_model_more(my_model_transaction, db):
    # I also run within a (different) transaction
    assert my_model_transaction.objects.count() == 1

One big downside is that this cannot be combined with transactional_db since there is always a transaction running. So if you have even one test that needs to run without a transaction, this will not work. In my case I had a pretty neat setup using this but realized toward the end that some tests needed to run outside of the transaction, so I had to refactor the whole thing.

Django TestCase still work!

Pytest will still pick up and run your Django TestCase classes. This is useful if you have a lot of tests already written in that style, and you don't want to convert them all at once. You can run them in parallel with your pytest tests, and slowly convert them over time.

Bonus: Some useful links

Your database is not elastic!

2024-04-17T00:00:00+02:00

Recently, I had a tough experience with a web app that didn’t scale as planned. We built the app to easily add more servers, but the database couldn’t keep up. This is a common issue in web development and often gets overlooked until it’s too late. Databases, especially relational ones, aren’t as flexible as other parts of the app. While you can quickly add more web servers or queue workers, scaling a database horizontally (adding more servers) is tricky and costly.

You can create read-only replicas, but setting them up isn’t easy or cheap. Sharding, another option, involves splitting the database into pieces, but it’s complicated and expensive. Handling transactions across these pieces is also a headache. Scaling up (making one server bigger) isn’t a long-term fix. Eventually, you’ll hit the limit of the biggest server you can afford.

It’s a strange reversal. In the past, when I worked on distributed apps like Windows-based CRM systems, the database was often the only part that could scale, while client machines couldn’t. Now, it’s the opposite.

This reality hit home for me last week: a deployment which had been carefully planned and tested for weeks quickly went south when it became obvious that too much of a burden was shifted to the database. The web workers happily scaled out to try and answer the requests faster, which actually compounded the problem since it added to the load on the one thing that would not, could not scale: the database.

Eventually we were able to resolve this with not too much broken bones by scaling up the database, until the application could be corrected. The cloud managed database service made that easier but it was still not a seamless process (and is also rather costly since it can't automatically scale back down either). And the feature had to be re-architected to be less database intensive - thankfully the change was rather isolated so we could turn that around quickly.

This experience taught us an important lesson: databases are a critical part of web development that require careful planning to avoid major problems down the line.

A Python-powered blog on AWS Cloudfront

2024-04-08T06:29:00+02:00

To start this blog, naturally I wanted to use a Python tool. I was attracted by the philosophy of Jekyll for static site generation, and as it seems every language has his static site generator (SSG) nowadays, I picked a popular Python equivalent: Pelican. Why Pelican? I actually started with "Nikola", but ran into a few minor annoyances. The Pelican package seems quite polished: they have a large plugin repository, they support reStructuredText by default (and I liked that they used docinfo for the metadata which seemed more appropriate than directives), and a modular architecture. Their documentation is pretty good too. Importantly, having a static site generator means my data remains easily exploitable by another engine, so it was not too important to spend much time on this decision now (indeed a large part of the job of a software engineer is knowing which decisions are important to spend time on, which ones can be deferred, and how to keep these decisions flexible). I decided to host it on Cloudfront, I have a lot of experience with the more "backend" AWS services so it should be a fun learning experiment (again, an easy to modify later decision). Cloudfront used to be rather pricey but now has a generous free tier that is more than sufficient for a personal blog.

This short article serves to document my journey as a howto if you are interested in replicating it.

Create a repo

The first step to anything :) I picked github, by default. Remember having to pick between RCS and CVS? Yeah, I'm that old.

Install Pelican

I used pipenv install pelican. By default it does not have Markdown support. Installing with pipenv means it will be kept separate from the system packages and installed within its own isolated environment. It's a good practice in general for Python development. From that point on you have to prefix all commands with pipenv run, or use pipenv shell to drop into a shell with the path to the virtual environment activated.

Generate blog skeleton

Use pelican-quickstart (but actually pipenv run pelican-quickstart!) to generate a skeleton. Initially I had already created the content folder and it errored out, but no biggy as it only serves to create an empty directory anyway. When running the quickstart, enter the name of the S3 bucket you will publish to. It doesn't have to exist yet.

I added a post under content so it would have something to show. Pelican will automatically determine the category based on the post path, so I put mine under content/tools//pelican-aws-cloudfront-blog.rst.

At that point pelican -r -l will build and serve locally. Pelican generated a Makefile so you can also do make devserver. This was a nice touch, I like Makefile. Just stop with Ctrl+C when done.

Push to AWS

You must have a correctly configured AWS CLI for this step. Refer to the Amazon documentation for this.

First, create the bucket. The easiest is to do it in the AWS console. Make sure you uncheck "Block public access". Also check "ACLs Enabled". The best practice is to leave ACL disabled, and create a policy, but it's a bit of a hassle and does not really make a difference in this case since we'll be the only one uploading to the bucket.
Run pipenv make s3_upload to upload the files. You may have to edit the Makefile to correct the name of the bucket. I also edited the Makefile at that point to remove
Create a CloudFront distribution in the AWS Console. Specify the bucket as the "origin". Leave the WAF (web application firewall) disabled. Also change "Origin access" to public. Again, it would be possible / best practice to specify this using a policy... but no real purpose here. Lastly, set the "Default root object" to "index.html". We'll come back later to configure the custom domain. Without configuring an alternate domain, CloudFront will set up a random cloudfront.net domain name I can use for testing (something like d2dl3dxwttz643.cloudfront.net I ran into some "Access Denied" error here... to troubleshoot, first run a curl command to retrieve the file directly from S3:
```
curl www.bytemypython.com.s3.us-east-1.amazonaws.com/index.html
```
That got me "Access Denied" so I knew something was up with the bucket permissions. I found that I had messed up one of the previous steps: I had left ACL disabled on the bucket. I went back and re-enabled them through the console, and redid the upload.
Register a domain. This is pretty easy with AWS Route53 and it will integrate with the rest. Go to AWS Route53 in the AWS Console, click Register Domain, and follow the prompt - it will take a little while to complete.

The way CloudFront works, it will download the files from S3 the first time they are requested, then cache them for up to 24h by default. Just something to be aware of when you publish new content. You can modify this behavior under the "Behaviors" tab of the CloudFront console. You can also create an immediate invalidation for all the cached files, with a little bit of effort it would be possible to combine to automate that within the upload command.

Create or customize a theme

I wanted to customize the theme a bit. I found a nice one called "Pneumatic" that I liked, quite minimalist, very clean. Since I am a full stack dev, I wanted to fiddle with it. I copied the templates but restarted the CSS from scratch using the pico framework. I have not experimented with it before, and it's a good occasion to try a different approach to CSS - at work we usually have the safe choices of Bootstrap or Material Design.

This lead me to discover the plugin feature of Pelican, which is very easy to use with pip. The webasset plugin is a good one to use for this purpose. It can preprocess files and generate a new file name that can then be included in the HTML template. I had a bit of fun with CSS and off we went :)

Update (July 2026)

The build process to publish to AWS was a bit annoying, and CloudFront added needless friction - I updated the workflow to do the build in Github Actions, and publish to Github Pages. The configuration is visible in the repo.

Byte My Python

Neovim TypeScript LSP Setup

What is special about TypeScript

Enable ts_ls in Neovim

Fixing a monorepo error

Links and final words

Organizing Agent Skills

The challenges

Managing installed skills

My layout

Take away

GitHub PR Review "View Deployment" button, with Dokku

Give each PR a URL

Storage

Linking from the PR

Cleanup

Takeaway

Loop Engineering: A Balance of Human and Agent

First run

Second run

Where the human actually added value

Switching to the Kitty Terminal Emulator

Quick start and config

Cheat sheet - some of the most useful shortcuts

Layouts

Kittens

Choosing a font

Open anything

Notifications

Final (first) impression

Why Git's Linear Log Can Lie to You

The Setup

The Reality

Why This Happens

Reading the Log Correctly

A Case for Rebasing

Takeaway

The QuerySet Database Binding Gotcha

The Setup

The Problem

Why This Happens

The Fix

Takeaway

Testing Lambda Functions with LocalStack and pytest

The fixture

Starting the container

Creating the Lambda

Waiting for the Lambda to be ready

Getting the endpoint URL

Putting it together

Getting Started with MicroPython (Raspberry Pi Pico)

Flash MicroPython (UF2)

Connect to the REPL

First test: turn on the built-in LED

Copy scripts to the board with rshell

Managing Python Monorepos with uv Workspaces and AWS Lambda

Running uv in a workspace

Use in Docker build

First attempt using uv sync

Second attempt using pip

Exploring Modern Python Type Checkers

Stop Fighting Circular Imports

My 5 Best Tricks using Pytest with Django

Install the pytest-django package

Using --reuse-db

Understand transaction scope and database fixtures

Use django_db_blocker for custom database setup

Use a larger-scope transaction for complex initialization

Sharing fixtures between Django apps

Django TestCase still work!

Bonus: Some useful links

Your database is not elastic!

A Python-powered blog on AWS Cloudfront

Create a repo

Install Pelican

Generate blog skeleton

Push to AWS

Create or customize a theme

Update (July 2026)

Enable `ts_ls` in Neovim

Copy scripts to the board with `rshell`