We treat shipping software like it needs five different computers.

Local dev runs on your laptop. CI spins up a fresh runner. Preview gets a container somewhere. Staging lives on its own box, and prod lives on another one again. Five environments, configured five different ways, drifting apart a little more every week.

Then you lose a day to a bug that only shows up in one of them, because the five were never actually the same machine.

Subscribe now

It is one workload, observed at different moments

Here is what I keep coming back to. All five are the same thing. The same app, the same dependencies, the same code. We are just looking at it at different points in its life.

Local dev is the workload while you are writing it. CI is the workload under test. Preview is the workload someone clicks through. Staging is the workload one step from users. Prod is the workload live. One process, five snapshots of it, and we built a separate tool to host each snapshot.

So we spend real engineering time, every day, keeping five copies of one thing in sync. Dockerfiles that have to match the runner that has to match the base image that has to match prod. The work is not building software. The work is fighting the drift between five machines that were supposed to be one.

Subscribe now

The field already moved off containers, halfway

A lot of teams saw this and reached for better isolation. Some moved to microVMs for the security boundary, which is the right instinct. A container shares the host kernel, a microVM does not, and for anything running untrusted or agent-generated code that boundary matters.

That solves isolation. It does not solve drift. You can run five well isolated microVMs and still rebuild the environment from scratch five times, still configure each stage separately, still watch them diverge. Better walls between the boxes, same five boxes.

The position: one primitive you fork

The fix is to stop treating the stages as separate products and treat them as branches of one running machine.

boxd runs every stage as a forkable microVM. You start a machine in about 30ms. You fork it in about 100ms, and the fork comes up already live, sharing the parent’s warm state. You pause a machine and resume it sub-ms, with memory, processes, and warm caches exactly where you left them, not a fresh boot from a disk image.

That last part is the piece most people skip. Filesystem snapshots are table stakes now. Plenty of platforms will snapshot your disk on stop and boot a new session from it on resume. boxd resumes the running machine itself, the live process state, not a new session that happens to share your files.

Put those together and the five stages collapse. Your CI run is a fork of the machine you developed on. Your preview is a fork of that. Staging is a fork. They cannot drift, because they came from the same place. The agent and daemon run inside the guest, so this is something you can self-host and run yourself.

Subscribe now

What this looks like if it is right

Branching an environment becomes as cheap as branching code. You stop maintaining five definitions of one workload and start forking one. “Does staging match prod” stops being a question you have to answer, because staging is a fork of prod.

I know this sounds futuristic. The honest reaction is to ask what happens when real production shows up. Microservices architectures, load balancing, replication, high availability, geo replication, all the things you actually run a system on. Those are real and we are not waving them away. They get solved along the way, one at a time, as we build toward this being where you run the whole thing and not just where you develop it.

So far I have been describing a single box. One machine you start, fork, and resume. The reason that matters past the single box is that a system running across many servers is still just servers. A microservices setup is twenty machines instead of one. A high availability cluster is the same workload replicated across boxes. Geo replication is the same machine, closer to more users. The unit never changes. It is always a server in some state, and the thing you want is to fork that state, pause it, and bring it back live. The single box is where the dynamic is easiest to see. The same dynamic, in a slightly different shape, is what you are fighting every time you run a system that spans more than one of them.

This is the spine of what we are building this quarter. I will go deeper on the fork primitive, on live-state resume, and on why filesystem snapshots alone do not get you here, in the pieces that follow.

If you are tired of keeping five machines in sync to ship one app, drop me a message. I will get you a machine to fork.

Michiel - boxd.sh

Series: where agents actually run. Part 2 of 3. Part 1 covered why containers should not run sensitive agentic workloads. This piece covers the MicroVM, the runtime the whole industry moved to, and why most of them are still built for the wrong workload. Part 3 covers what an agent-native MicroVM looks like.

Part 1 landed on a boundary. Agent code should not run on a kernel it shares with the host. Containers fail that test.

Subscribe now

The teams running agent code at volume already agree. E2B and fly both run their sandboxes on MicroVMs. Anthropic ships a sandbox runtime rather than trust a container. The move off the shared kernel is settled, and the thing everyone moved to is the MicroVM.

So the boundary question is closed. This piece is about what comes after it, because getting the boundary right does not mean you got the runtime right. Most MicroVMs were built for a workload that looks nothing like an agent.

What a MicroVM is, and what it was built for

A MicroVM is a virtual machine stripped down for speed. On Linux the kernel’s own virtualization layer, KVM, does the isolation, the device model is minimal, the guest boots in milliseconds instead of tens of seconds. You get a real kernel per tenant and a hardware boundary between workloads, with most of the weight of a traditional vm removed.

Firecracker is the reference implementation and it is genuinely excellent. AWS built it for lambda and fargate. Read the design goals. Boot a tiny vm fast, run one short function, tear it down, pack thousands on a host. The unit of work is a function invocation that lives for milliseconds to seconds.

That is serverless. The machine has no past and no future. It boots, computes a result, dies. State lives somewhere else. Nothing about the machine needs to survive the request.

Every assumption in that sentence is wrong for an agent.

Subscribe now

The shape of an agent run

Plenty of agents are short-lived, a single call that returns and ends. Those are a clean fit for the serverless shape, and nothing here argues otherwise. The agent this series is about is the other kind. It boots once and runs for hours. It reads a task, spawns a shell, installs packages, writes files, runs them, reads the output, decides what to do next. It accumulates state the whole way. Installed dependencies, a populated filesystem, environment, half-finished work, the context of everything it has done so far.

Then it stops to wait. For a tool response, for a user reply, for a rate limit to clear. It sits idle, often for most of its wall-clock life. Then it wakes and keeps going from exactly where it was.

Sometimes it needs to try five approaches to a hard problem. The clean way to do that is to branch the whole machine, live, and explore each path in parallel from the same starting state.

It also rarely works alone. The thing being built now is not one agent, it is many of them, spawning sub-agents, splitting a task across a fleet, comparing results. For that to work the machines have to talk to each other freely, and the context has to travel with them. An agent that hands work to another agent should pass the live state it has built, not start the other one from a cold slate and a serialized blob. The machines are nodes in one running system, not isolated functions that happen to share a host.

Long-lived. Stateful. Idle most of the time. Branching when it counts. Talking to each other with context intact. That is the opposite of boot-run-die on every axis that matters.

Subscribe now

Persistence is bolted on, not native

Firecracker assumes the machine is disposable and gets replaced. So if you want an agent’s state to survive a crash or a host reboot, you build that yourself. External database, external object store, a snapshot-and-restore dance, disk state registered somewhere outside the sandbox.

You end up building a persistence system around a runtime that was designed to throw state away. That is engineering you do because the primitive will not do it for you. The agent needs the opposite default, hours of accumulated context surviving a crash and coming back exactly where it left off, and the serverless MicroVM gives you a clean slate every time.

Subscribe now

Idle is billed, not suspended

A serverless runtime does not need an idle machine to suspend to zero and wake instantly, because serverless work does not sit idle for an hour. An agent does, most of the time.

So you get two bad options. Keep the machine alive while it waits on a user, and pay full rate for a machine doing nothing. Or tear it down and lose the live state you spent an hour building. Neither is what the workload wants, which is to suspend to zero bytes while idle and resume in milliseconds with state intact.

Subscribe now

Fork goes through a snapshot, not the live machine

This is the one that matters most, because branching is what turns one agent into a swarm. And it is where the serverless MicroVM works hardest to do something it was not built for.

Firecracker forks through snapshots. The shape is the same everywhere. Boot a machine, bring it to a useful standby point, snapshot it to a file, then start copies from that snapshot. Disk is cloned copy-on-write so children only allocate blocks when they write. Clever, and it works.

But read what the teams running it say about the cost. From Kernel.sh, after disk and vm state can be forked cheaply, “it’s loading the snapshot memory file that remains expensive.” Each vm resume “incurs a significant disk read,” and that read “scales linearly with the number of vms being launched, which means the physical limits of host disks throttle speed during bursts.” So they reach for userfaultfd paging to load memory lazily, a shared snapshot page cache, and hot pools of pre-booted machines so the common request never cold starts at all.

Look at what that stack is doing. Snapshot to a file, clone the disk copy-on-write, lazily fault memory back in, and keep a pool of pre-warmed machines so users do not feel the round trip. Each layer hides the same fact. The fork goes through a serialized snapshot of memory on disk, and pulling that memory back is the expensive part. It is a real engineering achievement, and it is a workaround for a primitive that was never meant to fork a live machine in the first place.

Subscribe now

The pattern

The MicroVM is the right boundary. That part is not in question, and the whole industry landing on it is the proof.

But most MicroVMs are still serverless underneath. Persistence is something you build around them. Idle is something you pay for or lose state to. Fork is a snapshot pipeline with three layers of optimization hiding the cost. None of that is a flaw in Firecracker. It is Firecracker being used far outside the boot-run-die workload it was designed for, by teams who picked the right boundary and then had to fight the lifecycle.

The agent workload asks for the boundary and a different lifecycle at the same time. Native persistence. Suspend to zero. Live fork that does not route through a file. A MicroVM shaped like the agent from the start, not a serverless one with the gaps papered over.

What comes next in this series

Part 3 covers what that looks like, treating persistence, suspend-to-zero and live fork as requirements rather than features bolted on, with concrete numbers on each. Cold boot fast enough to never pool. Native persistence so state survives a crash or a reboot. Suspend to zero so idle costs nothing. Live fork of a running machine, in parallel, without the snapshot round trip.

The boundary was the first decision and the industry already made it. The lifecycle is the next one, and it is still open.

If you’re building on agent infra and re-evaluating the runtime layer, drop me a message. Happy to compare notes.

I posted this on x earlier this week and the responses split into two camps. Half nodded. Half asked what I’d use instead. Fair question. Before we get to the answer, we need to be honest about the starting point.

Every AI company shipping agents picks a runtime. The default pick today is a container. Docker, kubernetes pods, fargate tasks, whatever the cloud wraps around runc. It’s fast, cheap, the team already knows it. For trusted internal services this is the right call.

Agents are not trusted internal services.

What a container actually is

A container is a process tree on the host kernel with namespaces wrapped around it. Cgroups for resource limits, capabilities for permission slicing, seccomp filters for syscalls. Nice software primitives. All sharing one kernel with the host and every other container on the box.

NIST said this plainly in SP 800-190: containers “do not offer as clear and concrete of a security boundary as a VM.” That document was written for normal multi-tenant workloads, before LLMs were deciding what to run at runtime.

The escape pattern keeps repeating

April 2026, three weeks ago, CVE-2026-31431 landed in the linux kernel algif_aead crypto module. CVSS 7.8 HIGH. Incorrect handling of in-place AEAD operations across memory spheres, exploitable from inside a container to corrupt host kernel memory. CISA added it to the Known Exploited Vulnerabilities catalog with a due date of May 15. Active exploitation in the wild. Every container on every patched-too-slow host is exposed because they all share that kernel.

This is the freshest case. It’s not the first one.

November 2025, CVE-2025-31133, CVE-2025-52565 and CVE-2025-52881, all in runc, all enabling container escape, all disclosed in a single advisory. The masked-path bypass let an attacker write to /proc/sys/kernel/core_pattern from inside the container. That’s host code execution.

January 2024, snyk disclosed leaky vessels. Four CVEs across runc, BuildKit and Moby. CVE-2024-21626 let a process escape to the host filesystem because of a leaked file descriptor in the runc working directory logic. Every version of runc up to 1.1.11 was affected, which means almost every docker, kubernetes and containerd deployment on the planet.

Subscribe now

The CNCF writeup is worth reading because it shows the pattern. Runc was audited. These bugs sat in the codebase for years. Mount race conditions, symlink swaps, procfs writes. The kernel surface is too big to fully reason about.

Look further back and the same shape repeats. CVE-2019-5736 overwrote the runc binary on the host. CVE-2022-0185 was a kernel filesystem bug usable from inside containers. CVE-2024-1086 was a netfilter use-after-free that ransomware groups picked up and used for privilege escalation through 2025. One kernel under all the containers means one bug class.

What changes when an LLM picks the command

A normal container threat model assumes the workload is trusted code with untrusted inputs. You wrote the binary, you control what runs, the attacker pokes at the api.

Agents invert that. The model is the entity choosing what to execute. It reads a webpage, it reads a github issue, it reads a slack message, and then it decides to run pip install something or curl | bash or chmod on a path. Anyone who can influence the text the model reads can influence the command it runs. That’s the prompt injection threat model and it’s well documented.

Now overlay the runc CVEs. A malicious image, a poisoned dependency, a RUN --mount= line in a dockerfile the agent decided to build. Each of those is a documented escape vector. The agent is happily running them because the prompt told it to.

Containers were built assuming the operator picks the workload. Agentic systems break that assumption.

Subscribe now

Hardening helps. It doesn’t solve

gVisor sits in front of the kernel and translates syscalls in userspace. Kata containers wrap a vm around the container. Both reduce attack surface in real measurable ways.

Both also pay for it. gVisor adds latency on every syscall. Workloads that are heavy on file io or networking can lose 30% or more. Kata is essentially admitting you wanted a vm in the first place, then bolting container ergonomics back on top. You pay for both the vm boot and the container runtime, and you still have to keep the container layer patched.

There’s also the deeper point. You can harden one runc deployment. You can’t audit out a class of bugs that has been resurfacing since 2019. The security history is the security forecast.

The teams closest to this already moved

The pattern repeats across the companies running agent code in production.

E2B runs AI agent sandboxes as their entire product. Their isolation primitive is firecracker microvms. When their docker partnership shipped in october 2025, docker tooling landed inside the microvm. The kernel boundary lives in hardware, the container is just packaging. They also inherit firecracker’s ephemeral lifecycle. Firecracker was built to boot, run a function, die. Agents run for hours and accumulate state across sessions. We’ll come back to this.

Fly.io took the same direction with sprites, built on the same firecracker base. Their checkpoint/restore lands around 300ms via CoW metadata. Modal took the direction with their own microvm runtime, optimized for short serverless invocations. Same kernel-boundary instinct, different lifecycle bets, the same lambda-shaped assumptions baked in at different layers.

Every team that’s actually shipped agent code at meaningful volume concluded the security boundary belongs off the shared kernel. That’s the first move and it’s the right one. The runtime layered on top of that boundary is where the choices start to diverge from what agents actually need.

The counter-example is daytona. Their architecture runs privileged docker containers on bare metal. They have HIPAA, SOC 2 and GDPR certifications which means their operational controls are real. The isolation primitive is still a shared-kernel container with elevated privileges. Compliance certifications and kernel boundaries are different things. An auditor checks that you have controls. An attacker checks the kernel.

The strongest validation comes from a company that has no commercial stake in this market. Anthropic, the team building claude, doesn’t trust containers as the boundary for claude code. They ship a sandbox runtime built on bubblewrap and seatbelt with a network proxy. Read their threat model: “a prompt-injected claude” modifying sensitive files, exfiltrating ssh keys, abusing network access. That’s the threat. Containers don’t address it.

Whether the runtime e2b, fly and modal moved to is the right primitive for agents specifically is a separate question. Hardware isolation is necessary. It isn’t sufficient. That’s part 2.

Subscribe now

Why this matters now

Agent infrastructure was a research demo in 2023. It’s a $139B forecast for 2034 with lovable running 1M+ sandboxes in a single weekend. The runtime layer is being standardized right now. The choice teams make in the next twelve months gets locked into production architectures for years

If you’re running agents on shared containers today, you’re betting that no one in the next five years finds another runc bug. The last six years say that’s a bet you lose.

What comes next in this series

Part 2 covers why standard vms and most existing microvms also fall short. Firecracker is a great primitive built for lambda. Lambdas boot, run a function, die. Agents boot, accumulate state for hours, fork themselves, pause for a user reply, resume. The lifecycle assumptions sit on the wrong end of the spectrum

Part 3 covers what an agent-native runtime actually needs. Cold boot times, fork semantics, pause/resume, persistence. With concrete numbers

If you’re building on agent infra and re-evaluating the runtime layer, drop me a message. Happy to compare notes.

Michiel - boxd.sh

$139B agentic infrastructure market by 2034. every AI company shipping agents hits the same question. where do these things actually run

the answer matters because the workload looks nothing like what came before

the shape of the workload is new

agents run for hours, not milliseconds. they accumulate state across tasks. they fork themselves to explore in parallel. they pause between user turns and resume later. they spawn shells, apt install packages, execute arbitrary binaries decided by the model at runtime

this is a persistent, stateful, unpredictable process that runs code we don’t fully trust. the existing primitives were built for something else

containers

fast, cheap, universal. they share a kernel with the host though. an escape in agent code hits the host directly. fine for trusted workloads, risky when an LLM decides what to execute next

gVisor, Kata, and friends harden containers but come with real tradeoffs. you either keep the shared kernel (and the attack surface) or you bolt on a VM underneath and pay for both

unikernels

Ivan Velichko wrote a sharp intro to unikernels that’s worth reading in full. short version: small, fast to boot, truly isolated. sub-200ms startup, sub-2MB images, no kernel/user split so syscalls stay cheap. the guest doesn’t share a kernel with anyone

the catch is what they ask of you. single-process by design. fork() isn’t supported, POSIX IPC goes over the network, there’s no shell to exec into when something goes wrong. agents spawn bash, install python packages, run arbitrary binaries at runtime. that whole pattern sits outside the unikernel model

great for narrow appliances. wrong shape for agents (could turn out to be wrong down the line here)

microvms

hardware isolation, real kernel per workload, KVM doing the heavy lifting. agents get a full linux they can actually use. a genuine virtual machine boundary between tenants

the isolation story is right. but most microvms were built for serverless. Firecracker, for example, was shaped around function-as-a-service workloads at AWS. boot, run, die. short-lived, stateless

agents live on the other end of that spectrum. they run for hours, accumulate state, fork, pause, resume

Subscribe now

agents need a new primitive

a variant of the microvm, with four characteristics the serverless generation never needed

cold boot fast enough to spin up a fresh machine per task. slow boot forces you to pool and recycle VMs, which leaks state between agent runs. under 50ms, you stop pooling. every agent gets a clean machine

memory-level fork so an agent can branch itself. this is the big one. agents hit hard problems all the time. “which of these five approaches works best” is a fork-join-reduce workload. if you can fork the entire live state of the machine in milliseconds, not just the disk, agents explore in parallel and keep the best result. without it, they serialize

pause/resume so idle agents cost zero. agents are idle most of the time. waiting on a user, on a tool response, on a timer. if the machine suspends to zero bytes of memory and resumes instantly, idle time stops burning money

persistence as a first-class property. most microvms are ephemeral by design. Firecracker assumes the VM dies and gets replaced. agents need the opposite. hours of accumulated context, installed packages, file system state, in-progress work. if the host reboots or the process crashes, the machine comes back exactly where it left off

that’s a compute primitive shaped like the workload

Subscribe now

what we built

at boxd.sh, this is the product. hardware-isolated microvms with sub 50ms cold boot, sub 10ms pause/resume, memory-level fork, and persistent disks

Laurentiu Ciobanu wrote ignition, our hypervisor, in rust, running directly on KVM. owning the hypervisor end to end is how we hit the numbers. no Firecracker, containerd, or gVisor between your code and the CPU

isolation is enforced at the network layer with eBPF/TC filters on the tap device, compiled at VM startup. egress restricted to tenant IPs and the infra subnet. every VM gets its own kernel, its own scheduler, its own file system

what comes next

the agent workloads we see today assume one machine per agent. that’s the baseline. once fork is cheap, the pattern changes. one agent orchestrates ten, each a branch of its own live state. after that, a hundred. an agent that hits a hard problem forks ten copies of itself, explores the solution space in parallel, merges the best result, keeps going

that’s the shape of compute behind the $139B number. a new primitive

try it

if you’re building agentic infra and figuring out the runtime layer, we’re shipping a new release which makes us the fastest in the market. drop me a message if you want to test it.

Michiel

Vercel recently open-sourced Open Agents, a reference platform for building coding agents. It's well-built, well-documented, and we respect the team behind it. We also think its core architectural decision is wrong, and that it will limit what agents can become.

The decision

Most agent platforms today separate the agent from the execution environment. The agent process runs on one machine and controls a sandbox on another through tool calls: file reads, file edits, search, shell commands. The execution environment is passive. The brain lives elsewhere.

Open Agents makes this separation explicit and central. From their docs:

The agent does not run inside the VM. It runs outside the sandbox and interacts with it through tools. That separation is the main point of the project.

They list four reasons:

1. Agent execution isn’t tied to a single request lifecycle

2. Sandbox lifecycle can hibernate and resume independently

3. Model/provider choices and sandbox implementation can evolve separately

4. The VM stays a plain execution environment instead of becoming the control plane

These are reasonable engineering arguments. They’re also, we believe, optimizing for the wrong thing.

Subscribe now

What separation gets you

Let’s be fair. The separated architecture has real advantages.

Observability. Separation makes it easy to log what the agent did. Every action goes through a defined tool interface.

Orchestration. The agent can control multiple sandboxes simultaneously. Frontend in one, backend in another, tests in a third. An agent inside a single VM can’t easily do this.

Resilience. The sandbox can crash without killing the agent. The agent can switch models mid-session. The two scale independently.

It ships today. Current LLMs operate through tool calls. The architecture matches how models actually work right now. It’s pragmatic.

We don’t dismiss these benefits. For controlled enterprise workflows where predictability and auditability matter most, this architecture makes sense today.

What separation costs you

Everything goes through a keyhole. Yes, shell.exec() exists. An agent outside the VM can technically install packages, write scripts, set up databases. But everything is serialized into shell command strings, sent over the network, and returned as text output. An agent inside the VM is a process on the OS. It reads files at disk speed, interacts with running processes in real time. Every interaction from outside carries the overhead of serialization, a network round-trip, and deserialization. An agent inside the VM just does the thing.

Latency becomes a larger tax over time. Today, most agent time is spent on LLM inference, so the overhead per tool call is small relative to thinking time. But as models get faster and agents make more calls per task, the tool-call overhead becomes a larger share of total execution time. An agent inside the VM doesn’t have this tax at all.

Fork breaks without the agent inside. If you fork a VM but the agent is running outside it, you’ve cloned the environment but not the agent’s context. The forked sandbox doesn’t know what the agent was thinking, what it tried, what it learned. The agent has to re-inject all of the relevant context into that new sandbox, costing tokens, time, and execution power. If the agent runs inside the VM, forking captures everything: environment, agent state, in-progress work. You can explore ten approaches in parallel, each with full context, and keep the most preferred outcome, while documenting non-preferred outcomes.

What we’re building instead

At boxd, we made the opposite bet. Every agent gets its own computer. Not a sandbox it controls through tools. A real Linux machine it runs inside. Here’s how we address the same four concerns without separation:

“1. Agent execution isn’t tied to a single request lifecycle.”
Agreed, it shouldn’t be. In boxd, the agent process runs inside the VM and persists across requests. The VM is the durable context. No external orchestrator needed to maintain continuity.

“2. Sandbox lifecycle can hibernate and resume independently.”
In boxd, the VM suspends and resumes in milliseconds, with the agent inside. Hibernation captures everything: environment, agent state, in-progress work. Resume is instant and complete. No context re-injection, no lost state, no token cost to rebuild what the agent already knew.

“3. Model/provider choices and sandbox implementation can evolve separately.”
The agent inside the VM can use any model via API. Swapping providers is a configuration change, not an architectural one. The VM doesn’t care which model the agent calls. Coupling agent to VM doesn’t mean coupling agent to model.

“4. The VM stays a plain execution environment instead of becoming the control plane.”
This is where we disagree most. A computer is the control plane for anyone doing real work on it. A developer thinks, experiments, and builds on the machine itself. Making the VM dumb and the external orchestrator smart limits what agents can become. Agents should be freed, not managed.

Full root access. Full filesystem. Full network. Whatever the agent needs, it can get. No permission, no tool call, no round-trip. The isolation comes from the hardware: each VM runs its own kernel. Full flexibility and full security, not a tradeoff between them.

Subscribe now

The training wheels argument

We think the separated architecture is a temporary crutch for a temporary problem.

Current models need structured tool interfaces because they’re not reliable enough to use a raw computer well. They make mistakes. They get confused by complex filesystem layouts. They run commands that don’t make sense. The tool interface acts as guardrails.

But models are improving fast. Every generation is more reliable, more capable, more autonomous. The tool abstraction that helps today’s models will constrain tomorrow’s.

You don’t give a junior developer a five-command terminal and an approval workflow for every action. You might start there. But the goal is to give them full access and trust them with it. The same trajectory applies to agents.

Vercel’s architecture makes the training wheels structural. When the models outgrow them, the architecture has to be replaced, not just updated.

We might be wrong

The honest version: we’re making a bet on the direction of AI capabilities. If models plateau and continue to need heavy guardrails and structured tool interfaces, the separated architecture wins. It’s more controllable, more auditable, more predictable.

If models continue to improve, if they become reliable enough to operate a full computer without hand-holding, then the separated architecture becomes the bottleneck. The companies whose agents have full computers will outperform the companies whose agents work through tool keyholes.

We’re betting on the models getting better. Given the trajectory of the last three years, we think that’s the right side of the bet.

Subscribe now

The bigger picture

Start with one agent, one computer. That’s today. But models get better and agent handoffs get smoother. Tomorrow, one agent coordinates ten computers. Then a hundred. Each one a full, isolated machine that the agent can fork, specialize, and orchestrate. An agent that hits a complex problem forks ten copies of its environment, explores in parallel, and merges the best result.

This scales indefinitely. The limiting factor stops being software architecture and becomes energy. That’s where we want to be: an infrastructure primitive so simple and fast that the only constraint on what agents can do is how many machines you’re willing to power.

You don’t get there by routing every action through a tool API on a remote orchestrator. You get there by giving agents computers and getting out of the way.

We think Vercel, respectfully, got it backwards.

We’re building boxd: hardware-isolated, persistent, forkable computers for AI agents. If you’re interested, check out boxd.sh.

Sources:

• Open Agents on Vercel

• Guillermo Rauch’s announcement on LinkedIn

• Open Agents docs

The Problem

The Software Development Lifecycle is outdated. It was built for an era when moving code to production was a dedicated step in the process. Coordination, environment configuration, dependency management, planning. That step created friction. It still

does.

The SDLC originated in the 1960s.¹ Royce described sequential phases in 1970.² The Agile Manifesto in 2001 prioritized working software and frequent delivery over rigid process.³ DevOps in 2009 focused on making deployment faster and more frequent.⁴ Every major shift changed how teams move through the phases. None removed deployment.

The new generation of builders enters the stack higher than ever before. They produce real, working software without understanding the underlying layers. Abstraction has always worked this way. Coding agents are achieving near-perfect performance on implementation tasks.

As agents take on more of the implementation and begin making architectural decisions autonomously, the lifecycle must be designed for them. Solutions built today should be agent-friendly by default.

Subscribe now

The World We Want

The current software development lifecycle has seven phases. Planning, Requirements Analysis, System Design, Implementation, Testing, Deployment, and Maintenance. We remove Deployment.

The six-phase software development lifecycle:

1. Planning - Define scope, goals, feasibility

2. Requirements Analysis - What are we building?

3. System Design - How will it work?

4. Implementation - Build the logic

5. Testing - Does it work correctly

6. Maintenance - Improve and iterate

Deployment disappears because the environment you build in is the environment your users access. There is no second environment to push to. Code runs the moment it’s written.

This changes what’s possible.

An agent builds a complete app without a human touching infrastructure. It creates a machine, installs dependencies, writes the code, starts the server. The app is live at a public URL. The agent tests it, iterates, and ships a working product. The only step between idea and live product is the code itself.

A solo founder ships three MVPs in a single afternoon. Each one runs on its own machine with its own domain. She shares the links with early users and gets feedback the same day. Nothing stood between the idea and a live product except the time it took to write the code.

A team replaces their environments with forks. A developer wants to test an experimental feature against real data on beta. Too risky to merge. She forks the beta machine in a second. Same disk, same database, same services. The fork shares the original state and only diverges when she writes to it. She runs the experiment, shows the team, destroys the fork. Beta was never touched.

A model/RL lab runs 500 agents in parallel, each on its own computer. Every agent gets a real VM with its own kernel, IP, and persistent disk. They fork a known-good state, experiment independently, report results. Fully isolated. The lab tears down all 500 machines when the run is done. Total cost: minutes of compute.

Builders focus on application logic. Agents handle the implementation. The separation between development and production is gone.

Subscribe now

Core Beliefs

Deployment was a solution to a problem. Software used to run in a different environment than where it was built. Bridging that gap required coordination, tooling, and process. That gap is closing. The solution should close with it.

Abstraction compounds. Every generation of tooling raises the floor for the next. Builders today operate at the application layer because the layers below have been solved. Each generation builds higher because the previous one made it possible.

The lifecycle should reflect reality. The SDLC has been patched and extended but never rebuilt for a world where agents write code and builders operate at the application layer.

Agent-friendly by default. Agents are already writing code, choosing architectures, provisioning resources, and making tradeoffs. This is accelerating. A lifecycle with steps that require human judgment at every transition is a ceiling on what agents can do.

Start where the volume is. Applications, websites, APIs, internal tools. That is the majority of software being built today. Critical workloads with strict compliance requirements can follow. The default should change first.

Call to Action

At Azin, we are building infrastructure that eliminates the boundary between development and production. VMs that boot in milliseconds. Machines you can fork in seconds. HTTPS domains that appear the moment your code starts listening. Agents that manage their own infrastructure through a CLI.

ssh boxd.sh new --name=myapp and you have a live computer on the internet in about 1 second. A beautiful gimmick: it starts and stops in under 10ms to have close to zero idle run time.

We are looking for builders, teams, and companies who want to explore what software development looks like without a deployment phase. If you have a use case, bring it.

boxd.sh

There are seven phases in the software development lifecycle. We’re making it six.

Michiel

Share The Application Layer

¹ Elliott, G. (2004). Global Business Information Technology. Pearson Education.

² Royce, W. (1970). “Managing the Development of Large Software Systems.” Proceedings of IEEE WESCON.

³ Beck, K. et al. (2001). Manifesto for Agile Software Development. agilemanifesto.org

⁴ Debois, P. (2009). First DevOps Days conference, Ghent, Belgium. See also: Bass, L., Weber, I., Zhu, L. DevOps: A Software Architect’s Perspective. Addison-Wesley, 2015.

Been watching this debate play out for months now. The data keeps pointing the same way. Anthropic launched MCP in December 2024. By March 2025 the hype was real - I was building a one-click deploy solution for remote MCP servers at the time. The promise was compelling: a universal connector between AI models and the world. But something in me kept questioning it. The context overhead, the awkward auth flows (way better now with OAuth 2.1), the servers you had to build and maintain for every single integration.

The token math

As an example of the context overhead: the GitHub MCP server dumps 55,000 tokens into your agent’s context before a single action runs. That’s just one server. Stack four of them - say GitHub, a database connector, Microsoft Graph, and Jira - and you’ve burned 150k+ tokens on plumbing before your agent does anything useful.

CLI equivalent? Your agent already knows gh. Total cost: around 200 tokens.

This goes beyond the Github server. Here’s what a typical enterprise agent comparison looks like on a real task - “list all non-compliant Intune devices and export to CSV”:

• MCP approach (Microsoft Graph MCP Server): ~28,000 tokens for schema injection alone, ~145,000 tokens total for 50 devices.

• CLI approach: ~8,000 tokens total for the same task. That’s an 18x difference. On one task.

That’s an 18x difference. On one task.

@LLMJunky tested it in a more controlled way and found 4 MCP servers used 380% more tokens than 14 CLI tools. The stat is making rounds because it’s hard to argue with.

Subscribe now

The implementation burden

Beyond tokens, every MCP integration is a small project. You need to learn the SDK, implement tool schemas, handle edge cases, manage deployment, and keep up with protocol changes as MCP evolves. If a service doesn’t have an MCP server yet, you build one from scratch.

CLI flips this. Most tools you’d want your agent to use already have a battle-tested CLI. git, gh, kubectl, aws, gcloud, stripe, railway - they’re all there. Your agent already knows how to use them. Zero setup.

What the builders are saying

@kepano, creator of Obsidian, put it plainly earlier this month:

“CLI is better than MCP. Faster. Direct access to your local files. No need for a server or auth. Easier for both humans and agents to access and explore. Composable with pipes.”

@hishboy was equally direct, quoting a viral CLI launch:

“For agents (and humans): CLI is a way better interface than MCP”

@Phillip Ninan on what he’s actually using day to day:

“Skills + CLI > MCP. My coding agents seem to work much better with CLIs than MCPs. Most of the MCPs I use nowadays are just for docs.”

And @Shreyas Rao wrote a full deep dive: 33% token savings with CLI, Unix pipes beat protocols, battle-tested beats bleeding edge.

So is MCP dead?

Not exactly. The builders I respect aren’t abandoning MCP entirely. But they’re reaching for CLI a lot more than they did a year ago.

MCP still has genuine strengths: structured permission scoping, clean third-party integrations where a CLI doesn’t exist, and read-only context sources like documentation. If you’re building a tool that needs to be consumed by many different agents in a standardized way, MCP makes sense.

But for the actual work agents do - running tasks, querying systems, executing operations - CLI wins on almost every metric that matters for efficiency.

Most of the conversation around MCP is about what it enables. If you flip it - and ask what agents actually need to run efficiently - the math looks pretty broken.

CLI has been on every Unix system since 1971. No schema injection. No server to maintain. No auth overhead. Composable with pipes. And your agent already knows how to use it.

It’s been on every machine since 1971. We just forgot to look.

Whether it’s OpenClaw on a Mac Mini, a VPS running 24/7, or a managed instance humming along in some cloud provider’s datacenter, the model is the same: always on, always paying, almost always idle.

We inherited this from web hosting. Keep a server warm, charge by the hour. It made sense when apps needed to hold state, maintain connections, and be ready at a moment’s notice. It doesn’t make sense for agents.

Subscribe now

Agents are fundamentally different

A web server serves requests continuously. An AI agent mostly waits - for a cron trigger, a message from Telegram, a webhook, a scheduled task. The actual compute happens in bursts. The rest is idle.

The current generation of agent hosting treats this like a traditional workload. You’re paying for a running process that sits there doing nothing, waiting for something to happen. It’s the equivalent of hiring a full-time employee to occasionally check their email.

The numbers are ugly

Take a typical OpenClaw deployment. Most users configure a handful of cron jobs, a Telegram channel, maybe some scheduled automations. The agent is actively running maybe 5-10% of the time. The other 90%+ it’s idle - but you’re paying the full rate.

At scale, this becomes absurd. The entire industry building agent infrastructure on top of VMs and containers is leaving efficiency on the table. The next layer of cloud infrastructure needs to be built around how agents actually work: bursty, event-driven, and mostly dormant.

Subscribe now

What agent-first infrastructure looks like

The model that makes sense for agents is the same that powers modern serverless functions - but faster. Sub-10ms startup so the “always on” illusion holds from the user’s perspective, but underneath, compute spins up when needed and disappears when the job is done. You pay for work, not waiting.

This means:

- Event-driven by default. A Telegram message arrives, the agent wakes, processes, sleeps.

- Startup so fast it’s imperceptible. Under 10ms means users never notice the difference.

- True pay-per-use. Not per hour. Not per month. Per unit of actual work done.

- No cold start hacks, no scheduled pings to keep things warm, no VM babysitting.

The gap between traditional cloud and agent-ready infrastructure isn’t small. It requires rethinking the runtime layer entirely - from how containers start, to how billing is calculated, to how orchestration handles bursty workloads.

We’re building it

At azin.run, we've been working on this problem for a while.

Our current product is a console that lets developers deploy agents on their own cloud infrastructure - think Railway, but designed for agentic workloads. Bring your own cloud, get a sane deployment experience without the usual devops overhead. openclaw.azin.run is one of the products on our marketplace, making it dead simple to run OpenClaw without touching a terminal.

But the console is the beachhead, not the endgame.

The real work is on azin cloud - our own infrastructure layer built specifically for agent workloads. Sub-10ms cold starts, true pay-per-use billing, compute that spins up on demand and disappears when the job is done. No idle, no waste, no babysitting VMs.

That's what agent-first cloud actually looks like. We're building it now.

Dependencies Are Vulnerabilities

Your business relies on external AI APIs. Tomorrow, new export restrictions could cut off access to your region. Your AI-powered fraud detection, customer service, and analytics systems stop working. Your operations halt.

A couple of examples that highlight this:

• Geopolitical tensions drive semiconductor export controls restricting AI chip access globally

• Closed-source cloud services (OpenAI, Anthropic via AWS/Google/Azure) remain subject to government oversight and potential restrictions

• Data sovereignty concerns make GDPR compliance with closed-source cloud AI legally problematic for European businesses

• Apple's reported $6.2B interest in Mistral AI (recent reports) signals another EU AI asset potentially moving to US control

Solution: Run open-source AI models on your own infrastructure with complete control over your data and operations.

Subscribe now

Quality Gaps Are Narrowing

Let's be honest: closed-source models from the leading foundational model providers still generally outperform their open-source counterparts, especially the smaller ones. But here's what's changing rapidly:

Stanford's 2025 AI Index shows that in January 2024, the top U.S. model outperformed the best Chinese model by 9.26 percent; by February 2025, this gap has narrowed to just 1.70 percent. Similar convergence is happening between closed and open-source models.

The same report reveals that many benchmarks we use to gauge AI systems are becoming "saturated"—AI systems get such high scores that the benchmarks are no longer useful for differentiation. This suggests we're approaching fundamental performance limits across all model types.

Asymptotic Convergence

My thesis: Model quality will asymptotically converge. Eventually, there won't be meaningful differences between closed-source and open-source models of similar scale. Therefore, most value capture will shift to the application layer - how you deploy, optimize, and integrate AI into your specific workflows and business processes. Why?

1. Architectural innovations spread rapidly - Research gets published, techniques get adopted

2. Training data is becoming commoditized - Similar datasets are available to all players

3. Fundamental physics limits - There are computational and data efficiency ceilings

4. Talent mobility - Researchers move between companies, knowledge transfers

We're seeing early signs of this convergence already.

Smaller Specialist Models

While everyone obsesses over larger general models, I find it highly interesting to look at smaller specialist models. In some domains, they even outperform general monolithic bigger models.

Looking in the medical AI corner, an interesting one I found is the Diabetica-7B model, which scored 87.2% accuracy on diabetes queries—beating both GPT-4 and Claude-3.5. That's a 7B model which can runs easily on the latest macbooks.

Improved small model performed can also be found in the coding corner. MIT researchers in 2025 developed an architecture that enables small, open-source models to outperform specialized commercial closed-source models more than double their size in Python code generation. DeepSeek-R1, an open-source reasoning model, excels at complex reasoning and mathematical problem-solving through self-verification techniques.

This pattern can be found in more domains. Financial services use specialized fraud detection models that crush general-purpose ones. JetBrains talks about "focal models" under 10 billion parameters that deliver better domain performance at a fraction of the cost.

The efficiency gains are appearing across general-purpose models too. Microsoft's Phi-3.5 series includes models like Phi-3.5-mini-instruct with just 3.82 billion parameters that are doing pretty well in language, reasoning, coding, and math. Gemma 2 comes in 9 billion and 27 billion parameter versions that can run locally on personal computers. StableLM 2 offers a 1.6 billion parameter model for specific, narrow tasks with faster processing, while their 12B model provides more capability without massive resource requirements. Qwen2.5 supports 29 different languages and scales up to 72 billion parameters for code generation and mathematical problem-solving.

Subscribe now

Therefore: Get familiar with running Models in your own infra

Enough talking. Running models locally can be super straightforward. I’m on a macbook pro with an M4 chip and 24gb of RAM. For this tutorial, I’m using Ollama. While Ollama provides the simplest path to local deployment, you have several other options: you can run models directly using Python frameworks like Transformers and llama.cpp for more control, deploy via Docker containers for scalable production environments, or use specialized platforms for high-performance inference on GPUs.

Step 1: Install Ollama

Head to the Ollama Download page and follow the instructions for your platform—Mac, Linux, or Windows. Ollama supports GPU acceleration on Nvidia, AMD, and Apple Metal, so it'll harness whatever hardware you've got.

Step 2: Choose Your Model

Browse the Ollama Library to explore available models. For this walkthrough, I'll be running a Mistral model, but you can pick whatever model you want as long as your hardware can handle it. Check the model size to ensure it fits in your available memory—the library shows the requirements for each model.

Step 3: Load and Run

Copy the command from the Tags tab (it'll start with ollama run) and paste it into your terminal. The model will download and start running locally. Use /bye to exit when you're done.

You can use it in the terminal like this:

In addition, you can also use your models in a self-hosted version of witheve.ai to enjoy Agent and workflow automations with access to over 1500+ apps (reach out for support). Below how I connected a llama and mistral model:

That's it. You now have a powerful AI model running entirely on your own hardware, with no cloud dependencies, no per-token costs, and complete control over your data.

Keep building,

Michiel

What is Proof of Reserves?

POR is a method for companies to demonstrate that they have enough assets to cover what they owe to their customers. Around 2022, the concept came to crypto as a way to verify that for every digital coin or token a company has issued or is holding for its users, there's an equivalent amount of real assets backing it up.

The Current State of Proof of Reserves

Currently, most major players in the crypto world, either just claim their backing or provide POR reports infrequently. For example, popular stablecoins like Tether only provide POR reports quarterly. These reports are typically attestations (evidence or proof of something) verified by third-party firms, offering a snapshot of the company's reserves at a specific moment in time.

The Tether Example

Tether is a stable coin that should always be worth one US dollar. Despite claiming billions in excess reserves, Tether has faced criticism for the lack of detail and independent verification in its quarterly reports. This infrequent reporting leaves users in the dark about the true state of reserves for months at a time, creating significant risk in a market that moves quickly.

The Need for Real-Time Verification

Reports are better than just claims. However, a quarterly frequency is simply not enough. The crypto market operates 24/7, with millions of transactions happening every day. The market needs real-time Proof of Reserves. Real Time POR could offer:

1. Instant verification of asset backing

2. Early detection of potential issues, possibly preventing major collapses

3. A stronger foundation for the growth of decentralized finance

4. Increased transparency and trust in the whole crypto ecosystem

Current Solutions and Challenges

There are a few existing solutions in the market working on real-time POR. These solutions typically involve pulling off-chain data and making it available on-chain for smart contract verification. However, they face several limitations:

1. Cost: Existing PoR solutions are often expensive to implement, making it challenging for smaller exchanges and protocols to adopt them. This high cost barrier limits widespread use across the industry.

2. Limited blockchain integration: Many current solutions are primarily designed to work with specific blockchain networks, such as those compatible with the Ethereum Virtual Machine (EVM). This limitation restricts their applicability to a broader range of blockchain ecosystems.

3. Complexity: Implementing these solutions can be technically complex, requiring significant resources and expertise.

4. Third-party reliance: Most current offerings are provided via third parties, introducing an additional layer of trust. This goes against the trustless principles that blockchain technology aims to achieve.

These challenges highlight the need for more accessible, cost-effective, and versatile PoR solutions that can cater to a wider range of blockchain networks and service providers in the industry.

The Future of Proof of Reserves

As the industry evolves, there's a clear need for more accessible and cost-effective real-time PoR solutions. The ideal future system would:

1. Provide real-time or near-real-time verification

2. Be affordable for both large and small companies

3. Offer easy integration with existing systems

4. Scale effectively to handle the growing crypto market

5. Implement a trustless setup, eliminating the need to rely on third parties for verification

Subscribe now

Core Technological Components of Proof of Reserves

To understand how real-time POR works, it's important to grasp some of the key technological components that make it possible. The core components are essentially Merkle Trees and Zero-Knowledge Proofs (ZKPs). The idea behind building such cryptographic proofs is a really interesting one and goes back to as early a 2014 (worth reading the proposal on the Bitcointalk forum). An idea was proposed to use a Merkle tree-based technique to verify whether certain sites really hold the bitcoins they claim to hold. So it was already a point of attention at that time.

Merkle Trees

So what are those ‘Merkle tree-based techniques’ where (self declared) Cyberpunk Greg Maxwell was referring to in 2014? Merkle trees are a fundamental data structure in cryptography and blockchain technology. They allow for efficient and secure verification of large data sets. In the context of POR (specifically for exchanges), a variant of the Merkle Tree is often used called a Merkle Sum Tree. In such a tree, leaves are one-way hashes of user account private data.

The Merkle Sum Tree works as follows:

1. Each leaf node contains a hash of a user's account data (e.g., account ID, balance) and the actual balance value.

2. As we move up the tree, each parent node contains: a) A hash of its child nodes' hashes b) The sum of its child nodes' balances

This structure allows for two critical features:

• Efficient verification of individual account inclusion (through Merkle proofs)

• Verification of the total sum of all account balances

The root of the Merkle Sum Tree represents both the cryptographic proof of all included accounts and the total sum of all balances. This allows exchanges to prove they have sufficient reserves to cover all user balances without revealing individual account details.

However, this approach faces a privacy challenge: to verify a proof in a summation tree, one needs to know the balances of neighboring nodes. This is where Zero-Knowledge Proofs come into play.

Zero-Knowledge Proofs

Zero-Knowledge Proofs (ZKPs) are cryptographic protocols that allow one party (the prover) to prove to another party (the verifier) that a statement is true without revealing any information beyond the validity of the statement itself.

The Famous Cave Diagram above illustrates a simple Zero-Knowledge Proof:

• Peggy (purple figure) claims to know a secret password that opens the magic door between paths A and B.

• Victor (green figure) wants to verify this without learning the password.

The process:

1. Peggy enters the cave, choosing path A or B randomly.

2. Victor shouts to Peggy that he should take back path A or B.

3. Peggy must exit from the called path, using the door if necessary.

Repeating this many times, Victor becomes convinced Peggy knows the password without learning it himself. This demonstrates ZKP's core principles:

• Completeness: Peggy can always convince Victor if he knows the password.

• Soundness: Peggy can't trick Victor if she doesn't know it.

• Zero-knowledge: Victor learns nothing about the password itself.

Subscribe now

In the context of POR, ZKPs address the privacy concerns raised by Merkle Sum Trees. Instead of users directly validating their Merkle Proof by hashing their private data and rebuilding the root hash (which would require knowledge of other users' balances), they can validate a ZKP that's validating their Merkle Proof.

Here's how ZKPs enhance the POR process:

1. The exchange generates a ZKP for each user's Merkle proof.

2. This ZKP proves that: a) The user's account is included in the Merkle Sum Tree b) The user's balance is correctly incorporated in the tree's total sum c) All balance values in the path are non-negative

3. Users can verify this ZKP without learning anything about other accounts or the specific path their account takes in the Merkle Sum Tree.

By combining Merkle Sum Trees with ZKPs, exchanges can provide a robust POR that:

• Proves the total sum of user balances

• Allows individual users to verify their account inclusion

• Maintains privacy for all users

• Prevents manipulation of reported balances

This approach significantly enhances the trustworthiness and transparency of crypto exchanges and protocols, addressing many of the concerns raised by recent platform collapses.

E7.2: Building a More Trustworthy Ecosystem With Accountable Proof of Reserves

POR is a crucial tool for ensuring transparency in the crypto ecosystem, but the current model of infrequent reporting falls short of the needs of a fast-moving market. As we move forward, real-time PoR should become the industry standard—not just for compliance, but for securing the future of the entire crypto space.

But what if we told you that the future is already here?

At Accountable, we've developed a state-of-the-art system that combines real-time data aggregation, advanced cryptography, and automated auditing processes to deliver a comprehensive and continuously updated Proof of Reserves solution - all in a trustless way.

Next week, together with CTO Ioan Moldovan, we're pulling back the curtain on a our solution that's set to redefine how we approach PoR. In doing so, we're working towards a more trustworthy, transparent, and efficient financial ecosystem.

Thanks for reading and see u next week!

Keep building 🛠

Michiel

Subscribe now

I genuinely feel sorry for everyone affected by this exploit, especially those who lost their funds and the Penpie team who worked hard to build their platform. It's heartbreaking to see the impact that such vulnerabilities can have on both individual investors, the entire project, and the DeFi ecosystem.

Despite this setback, it's important to recognize the significant contributions both Penpie and Pendle have made to advancing the DeFi ecosystem. Fortunately, the funds on the main Pendle platform were unaffected by this exploit.

Michiel

A quick Summary

On September 3, 2024, Penpie, a subDAO (Decentralized Autonomous Organisation), operating on the Pendle platform (a protocol for trading tokenized yield), fell victim to a severe exploit that resulted in the loss of $27 million. The incident shows not only the technical vulnerabilities within smart contracts but also underscores the importance of acting on identified risks. In this article, we’ll explore the mechanics of the exploit, the audit, and the broader implications.

Subscribe now

The Exploit: What Happened?

The attack centered around a vulnerability in the internal _harvestMarketRewards function of the PendleStakingBaseUpg contract of Penpie. This function is responsible for harvesting rewards from various markets and distributing them accordingly. The attacker exploited a (kind of) reentrancy vulnerability, allowing them to repeatedly call the function and withdraw far more rewards than they were entitled to.

The vulnerability was rooted in how the function interacted with the redeemRewards method of the IPendleMarket contract:

IPendleMarket(_market).redeemRewards(address(this));

This external call allowed the attacker’s malicious contract to manipulate the flow of execution, re-enter the function, and trick the contract into thinking it had more rewards to distribute than it actually did. By artificially inflating the balance of rewards, the attacker managed to drain the contract, causing significant financial loss.

Subscribe now

A Known Issue

In June 2023, an audit conducted by WatchPlug flagged a related issue with the same piece of code. The auditors identified that the redeemRewards function call allowed any sender to claim rewards on behalf of the user, potentially leading to inaccurate reward tracking if manipulated.

Despite this critical finding, the vulnerability was not adequately addressed, leaving Penpie exposed to the very attack that WatchPlug had warned about over a year earlier. The issue was marked as resolved. However, in the audit the repo /pendleMagpie was mentioned while I found the exploited code in repo /penpie_contracts. Therefore, I couldn’t check how the vulnerability was addressed.

Market Reaction and Implications

The exploit led to a dramatic 40% drop in Penpie’s native token (PNP) and an 8% decline in Pendle’s token (PENDLE). The incident not only shook investor confidence but also started discussions again about the importance of good and responsive security practices in the DeFi space.

Lessons Learned

1. Act on Audit Findings: It is crucial for DeFi projects to take audit findings seriously and implement recommended changes promptly. The Penpie incident demonstrates the potential consequences of ignoring or delaying such actions.

2. Reentrancy Guards and Security Best Practices: Ensuring that smart contracts are protected against reentrancy attacks, using tools like reentrancy guards, is essential. So not only on external functions that call internal functions. If there is an internal function making an external call, please be cautious and take adequate steps to secure this call. Additionally, following best practices such as the Check-Effects-Interactions pattern can mitigate many common vulnerabilities.

3. Ongoing alertness: As DeFi protocols continue to evolve, continuous monitoring and regular audits are necessary to maintain security. The dynamic nature of the crypto space requires projects to stay ahead of potential threats.

Conclusion

The Penpie exploit serves as a harsh reminder that even well-documented vulnerabilities can lead to catastrophic outcomes if not addressed. For the DeFi space to thrive, developers and project teams must prioritize security at every stage, from initial development through to post-deployment monitoring. By learning from incidents like this, the community can work towards a more secure and resilient ecosystem.

Michiel

Share

YourOwn, inspired by the serie Suits, is building Donna—a powerful Whatsapp based AI assistant tailored for SMBs. Donna comes equipped with a suite of tools such as reading documents, setting reminders, and searching the internet. However, despite her capabilities, Donna had a significant limitation: she couldn’t access or utilize company-specific data.

The Vision: Enhancing Donna with RAG

The goal was clear: to integrate a RAG system into Donna, enabling her to access and retrieve company-specific data, thereby making her more useful and relevant to the businesses she serves.

What is RAG?

At its core, RAG consists of two primary components:

1. Data Retrieval: This involves fetching and storing data from various sources.

2. Data Searching: This part involves searching through the stored data to find relevant information when needed.

By blending these two processes, RAG allows an AI system to pull specific, relevant data on demand, significantly improving its response accuracy.

Subscribe now

The Blueprint: Building a Successful RAG System

Step 1: Understand the Requirements

The first step in building a successful RAG system is to thoroughly understand the project’s requirements. For Donna, this meant considering factors like:

• Number of Companies: How many businesses will Donna serve?

• File Types: What kind of documents and file extensions will she need to handle?

• Latency Constraints: How quickly does the data retrieval and search need to be?

These are just a few of the critical questions that need to be addressed upfront.

Step 2: Identify Integration Points

The next step was identifying where and how to integrate the RAG system with existing infrastructure. In the case of Donna:

• Data Source: We agreed to use Google Drive as the primary data source.

• Authentication: Implementing OAuth mechanisms was essential to securely access company data.

• Folder Selection: There needed to be a functionality for users to select specific folders for Donna to access.

• File Handling: The system had to manage file changes, additions, and deletions effectively.

• Chatbot Integration: Donna’s chatbot was already built using Langchain, so seamless integration with this framework was crucial.

• Communication: Since Donna interacts with users via WhatsApp, integrating with the WhatsApp server was also vital.

  Subscribe now

Step 3: Build the Population Pipeline

Once the integration points were identified, the next step was building the pipeline to eventually populate the database. This involved:

• Fetching the Data: Retrieving data from the identified sources, such as Google Drive, using secure access protocols like OAuth. This step is crucial for ensuring that the system has the necessary data to work with.

• Chunking and Embeddings: Splitting documents into manageable chunks and generating embeddings (vector representations) to make the data searchable.

• Vector Database Setup: Setting up a vector database to store these embeddings. A vector database is a specialized database designed to handle the complexities of storing and searching through vectorized data, which is crucial for efficient and accurate retrieval in a RAG system. I decided to use Pinecone as it is a fully managed, cloud-based Vector database with a reasonably large free tier. A tutorial I highly recommend where you can use Pinecone together with Langchain can be found here.

Step 4: Implement the Semantic Retriever

The semantic retriever is the core component that powers the search functionality within the RAG system. For Donna, I implemented a hybrid retriever that combines two powerful methods: similarity search and a BM25 retriever. This combination enhances the accuracy and relevance of the search results by addressing the limitations of each method when used independently.

Similarity Search

Similarity search is a technique that uses vector representations (embeddings) of documents to find the most similar pieces of information based on a query. In this context, each document is converted into a vector—a numerical representation that captures the semantic meaning of the text. When a query is made, the system searches for the vectors that are closest to the query vector in the high-dimensional space. This approach is highly effective for capturing the overall context and meaning of the query and the documents.

However, similarity search can sometimes miss specific keywords or exact phrases because it focuses on the broader semantic context rather than individual words. This is where the second component, BM25, comes into play.

BM25 Retriever

BM25 is a type of information retrieval model that is particularly good at keyword-based searches. It ranks documents based on the presence of query terms, considering factors like term frequency and document length. BM25 excels at finding documents that contain exact matches of the query terms, making it complementary to similarity search.

By combining BM25 with similarity search, we create a hybrid retriever that leverages the strengths of both methods. The hybrid retriever first performs a similarity search to identify documents that are semantically close to the query. Then, it uses the BM25 retriever to ensure that documents containing specific keywords or phrases are also considered. The results from both searches are merged, and duplicates are removed to provide a set of unique, highly relevant documents.

Subscribe now

Step 5: Test, Iterate, and Refine

Building a RAG system is as much an art as it is a science. After the initial implementation, we created a custom evaluation set based on real-world questions derived from documents provided by our building partners. Through continuous iteration and refinement, we significantly improved the system’s performance.

The Result: A Smarter, More Capable Donna

After numerous iterations and refinements, Donna became significantly smarter. Armed with the ability to access and retrieve company-specific data, she now offers more tailored, contextually relevant assistance to SMBs. The addition of RAG has transformed Donna into a more powerful tool, making her an invaluable asset to the businesses she serves.

Conclusion: The Power of RAG

Implementing RAG was a challenging but rewarding experience. It demonstrated the immense potential of combining retrieval and generation in AI, particularly for applications that require access to specific, contextually relevant data. As Donna continues to evolve, the lessons learned from this project will serve as a foundation for future advancements in AI-driven personal assistants.

For a basic implementation of populating a Pinecone database with Wikipedia articles and searching through it, please visit my repo.

For those looking to implement similar systems, remember that the key to success lies in understanding your requirements, identifying integration points, building a robust data pipeline, and being prepared to iterate continuously. The journey can be complex, but the results are well worth the effort.

Bart, Maarten en Elkan, good luck!

Michiel

Share

- E2.2: Welcome Agentic

- E3:2: Bringing Agentic to Life

Therefore, I’d like to share the current state of the project with you. Enjoy! Michiel

################################################################

What is Agentic Summaries?

Agentic Summaries is designed primarily for crypto enthusiasts to help you find your signals through the noise by providing tailored summaries from your favorite Telegram channels 🔵 and Subreddits 🔴. These summaries are filtered by your interests and delivered up to three times a day 🚀.

How to Get Started:

Go to our beloved Agentic Summaries Bot

Interact with the bot to configure your preferences and start receiving your personalized summaries.

Our Vision:

Our goal is to revolutionize how market players - traders and investors alike - access essential information, focusing on the most impactful insights to uncover real opportunities - alpha. We cut through the dense data in the crypto market to bring you targeted news summaries that highlight potential market moves before they become mainstream knowledge.

Stay Connected:

To contribute to the project, please join our Community Channel.

Welcome aboard, and let's find those signals together! 🚀

Introduction

In our previous blog, we explored the differences between production and non-production environments. Now, it's time to take that knowledge a step further with a practical implementation. In this post, we'll deploy a Reddit scraper that leverages Large Language Models (LLMs) and eventually posts its findings on Twitter in a fully automatic way, running once daily. Due to past experience, we'll use Microsoft Azure and VS Code for this deployment. Yet you can use a Cloud provider of choice as such deployments will be more or less similar among Clouds.

Why Learning Cloud Deployment is Crucial

Understanding cloud deployment is essential for modern software development. Moving your projects to a cloud environment brings numerous benefits, including scalability, reliability, and accessibility. It also equips you with skills that are highly valued in the industry. In this blog, we will take the Twitter scraper from blog E2.2 and deploy it, demonstrating the process step-by-step.

Recap of the System

Agentic consists of two agents: a Reddit researcher and an editor-in-chief. We scrape information of posts and comments on the subreddits specified in the variables.py file:

The researcher gathers information from Reddit, and the editor checks the work of the researcher. Once done, findings are posted to Twitter. We use open-source Agentic Workflow framework Crew AI for this. For more information, you can check the GitHub repository and the previous blog post.

Subscribe now

Functional Requirements for Deployment

Let’s use the framework as outlined in the previous blog to better understand how to approach the deployment.

Understanding the Production Environment

• Scalability Requirements:
  The system is designed to run locally with limited RAM and needs to operate only once per day, making extensive scalability unnecessary. A cron job will be used to schedule this daily task. A cron job is a way to set up automatic tasks on a computer to run at specific times, like setting an alarm clock for your software.

• Integration Points:
  As a new deployment, there are no existing systems to integrate with.

• Performance Constraints:
  Given the simplicity of this deployment, performance constraints are minimal.

Code, Versioning, and CI/CD Pipelines

• Version Control Systems (VCS):
  We'll use GitHub for version control, as it provides a free and robust platform for managing code.

• Model Versioning:
  Ensure the deployed version matches the most recent version on GitHub. The only exception is the pysqlite3-binary requirement and corresponding code in function_app.py, necessary for Linux deployment. Make sure to comment it out when running locally on Windows.

• CI/CD Tools:
  While CI/CD tools can be beneficial, they are overkill for this simple deployment.

• Pipeline Stages:
  For this project, automated pipelines are not required.

Containerisation and Orchestration

Containerisation and orchestration are not necessary for this small-scale deployment.

Monitoring and Logging

• Logging Practices:
  Implementing different logging levels is crucial for monitoring the bot's performance and debugging any issues. Azure's portal provides comprehensive tools for capturing and analyzing logs. Using the logging capabilities within the Azure function app suffices here.

Data and Model Governance

Not applicable for this hobby project.

Security Considerations

• API Keys:
  Store API keys in a .env file to follow best practices for security and configuration management.

Performance Optimisation

Performance optimisation is not a priority due to the deployment's simplicity.

Documentation and Knowledge Sharing

Sharing your deployment experience through a blog post is an excellent way to document the process and help others. :)

Subscribe now

The Deployment Process

1. Download and install the prerequisites

   1. Check out the Microsoft website for deployment of functions using VS code.

      1. Specifically, download the Core Tools and the Azure Functions extension.

2. Clone the repository from GitHub and set up your local environment.

3. Set up your .env file with your API keys.

4. If you don’t want to setup the Azure functions yourself and leverage my files, go to step 8. If you want to do it yourself, copy research_tweeting and only keep main_content.py, custom_functions.py, requirements.txt and variables.py and perform the steps below.

5. Create a Local Function Project:
   Use Azure Functions to create a Python-based function project with a time trigger.

6. Go through the steps and finally set the Cron Job:
   Configure the cron job to run the function daily at 1 PM:

The default setting is that the system also runs at startup. If not desired, change it in function_app.py.

7. Directory Structure:
   A lot of files are automatically being created now. Ensure your project directory is correctly structured. At this point, you Azure function files are at the same level as the research_tweeting folder. Unfortunately, your gitignore and requirements.txt function may be overwritten by Azure. Check this. Make sure to add the right lines back.

8. Local Function Testing:

   1. Use Azurite for local testing. Hit crl + Shift + P and Start Azurite.

   

   1. UTF Encoding:
      When you’re on a Windows, you will get an UTF encoding error. You can fix this by running Set $env:PYTHONIOENCODING = "utf-8" for proper encoding.

   2. Run the Function Locally:
      Execute func start --verbose to test your function locally (the verbose flag is for verbose logging).

9. Deploy to Azure:

   1. Create a resource group and storage account on Azure.

   2. Run your version of the following command with your resource group and storage account you just made:

   az functionapp create --resource-group agentic --consumption-plan-location westeurope --runtime python --os-type Linux --functions-version 4 --name agentic-twitter-app --storage-account agenticstorage

10. Publish to Azure

    1. Run the following command to publish your app to the app you just created. For any code changes, you can run this command as well.

func azure functionapp publish agentic-twitter-app

5. Check Logs:
   Monitor the logs in the agentic-twitter-app function app via the Azure portal which you can find under the section Monitoring within your function app. If you see you’re Crew AI logs, you’re up and running and your crew will start posting on Twitter soon.

Conclusion

Deploying a bot like Agentic to a cloud environment like Azure demonstrates the practical application of cloud deployment skills. This process not only enhances the functionality and reliability of your projects but also equips you with essential knowledge for real-world applications. By following this guide, you can confidently take your projects from local development to production, ensuring they deliver value effectively and efficiently.

Subscribe now

Call to Action

Ready to deploy your own projects? Leverage the approach in this blog to set up your own scraper that posts to Twitter. Embrace the power of cloud deployment and see how it transforms your workflow. The full code can be found in the GitHub repository.

Thanks and enjoy coding!

Michiel

Please reach out to chat about potential applications of the system.

Subscribe now

1. Understanding the Production Environment

Before deploying a system, it’s essential to understand the production environment where the system will operate. Key considerations include:

• Scalability Requirements: Assess if the system needs to handle large volumes of data or concurrent requests.

  For example, for an e-commerce platform predicting user behaviour, the model may need to scale to handle thousands of predictions per second, especially during peak shopping periods. This requires robust infrastructure and possibly load balancing to distribute the computational load.

• Integration Points: Identify how the system will integrate with existing systems and data sources. As an example, deploying a credit risk model often involves logistic regression.

  
  Big financial organisations generally make use of a SQL database (provided by those lovely SAP guys). Once the model is finalised, the model’s (transformed) weights, coefficients, and buckets must be translated into SQL code to be shared with the SQL team. Pretty manual unfortunately.

• Performance Constraints: Ensure the model meets performance benchmarks, such as response time and throughput.

  For a real-time fraud detection system, the model needs to process transactions within milliseconds to prevent fraudulent activities. This requires optimising the model’s inference time and ensuring the underlying infrastructure can support such low-latency requirements without compromising accuracy.

If there is no production environment yet, e.g. for your side project, choose the cloud of your preference. If not familiar with any cloud environment, make sure to learn one. If no appetite to learn or if it’s a simple deployment, stick with Heroku.

2. Code, Versioning, and CI/CD Pipelines

Effective version control and automated deployment processes are critical for maintaining reproducibility, collaboration, and efficiency in bringing systems and models to production.

• Version Control Systems (VCS): Use systems like Git to track changes in code and data. This ensures that every modification is documented, allowing teams to collaborate seamlessly and maintain a history of changes. As a tip, from now on never use the UI anymore. Teach yourself to work with the terminal.

• Model Versioning: Implement a strategy to version models, enabling rollback to previous versions if needed. This is crucial for tracing model performance over time and ensuring that any updates or changes can be easily managed and reverted if necessary. If you don’t use a CI/CD tool, make sure that your deployed version is always the same as the most recent version on your VCS.

• CI/CD Tools: Use tools like Jenkins, GitLab CI, or CircleCI to automate the testing and deployment processes. CI/CD pipelines help in maintaining code quality and facilitate continuous integration and delivery. Not really necessary for a small/moderate size project yet using them will save you time.

• Pipeline Stages: Include stages such as build, test, and deploy in your CI/CD pipeline. Automating these stages reduces manual errors and enhances efficiency, ensuring that the code and models are consistently tested and deployed in a reliable manner.

  • Build Stage: Compile and prepare the code for deployment.

  • Test Stage: Run automated tests to validate code and model performance.

  • Deploy Stage: Deploy the code and models to the production environment.

3. Containerisation and Orchestration

Containerisation simplifies deployment and ensures consistency across environments:

• Docker: Containerise the model and its dependencies using Docker. I highly recommend you familiarise yourself with using Docker.

• Kubernetes: Use Kubernetes for orchestrating and managing containerised applications at scale. For small projects, please don’t bother. Check this blog is you are in doubt.

5. Monitoring and Logging

Continuous monitoring and logging are essential for maintaining model performance:

• Logging Practices: Implement structured logging to capture relevant information for debugging and analysis. For Python, I’d recommend setting up a basic logger and replace all print statements with it:

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)  # It's good practice to use __name__ to get the correct logger

Most monitoring systems will recognise this type of logging perfectly. You can set different levels as well (warning/bug). Pretty sweet. In addition, make sure to catch as much errors as possible within your code. Leverage Try Except statements as much as you can.

• Monitoring Tools: Mostly depends on your choice of cloud. You can leverage stand alone tools like Prometheus, Grafana, or ELK stack for monitoring.

• Alerting: Set up alerts to notify the team of any anomalies or performance issues. Your cloud provider generally has some options in place.

6. Data and Model Governance

Governance ensures compliance and ethical use of data and models:

• Data Privacy: Adhere to data privacy regulations like GDPR/AVG (Dutch).

• Audit Trails: Maintain detailed records of data processing and model changes.

7. Security Considerations

Securing the model and data is important:

• Authentication and Authorization: Implement robust authentication and authorization mechanisms.

• Data Encryption: Encrypt data in transit and at rest.

• Vulnerability Scanning: Regularly scan for vulnerabilities in code and dependencies.

8. Performance Optimisation

Optimize the model and infrastructure to meet performance requirements:

• Profiling and Tuning: Profile the software system to identify bottlenecks and optimize performance. Use profiling tools to analyse CPU, memory, and I/O usage, and make necessary adjustments to improve efficiency and speed.

• Hardware Acceleration: Leverage hardware accelerators like TPUs for inference, if necessary. Consider using specialised hardware to enhance processing power and handle intensive computational tasks more efficiently.

9. Documentation and Knowledge Sharing

Not the nicest task but comprehensive documentation facilitates maintenance and collaboration:

• API Documentation: Document APIs using tools like Swagger or Postman.

• User Guides: Create detailed guides for users and stakeholders.

• Knowledge Sharing: Encourage knowledge sharing through workshops, seminars, or wikis.

Fortunately, you can leverage your gen AI counterparts to do at least 80 pct of the work for you. What a time to be alive.

Subscribe now

Conclusion

Bringing software systems, including data science models, to production involves more than just development. It requires careful planning, robust infrastructure, and continuous monitoring to ensure they deliver value in real-world settings. By following these best practices, developers and data scientists can bridge the gap between development and production, ensuring their systems are scalable, reliable, and secure.

Call to Action

Are you ready to take your software systems and models to production? Start by assessing your current workflow against these best practices and identify areas for improvement. Yet the most important thing here is to get your hands dirty as quickly as possible. Deploy something and get experience. Embrace automation, continuous monitoring, and robust governance to ensure your projects are production-ready and deliver sustained value to you and/or your organisation.

E3.2: Bringing Agentic to Life.

In the next edition, I’ll demonstrate how to deploy Agentic - a Telegram bot that scrapes Reddit and post its findings to Twitter - and use a Cron job to let it run once per day. We’re using Microsoft Azure for this. Stay tuned and thanks for reading.

Michiel

Share

Subscribe now

Welcome back to The Application Layer. Remember how we talked about a future where systems could do almost anything for us, just like in that movie “Her”? Well, today's not about dreaming anymore—it's about making some of that stuff happen for real. We're going to roll up our sleeves and get our hands dirty with a project that's as exciting as it is useful. Imagine having a little digital helper that can read stuff from the internet, make it look nice, and share customised content to your favorite medium, all by itself. Inspired by this tutorial, I'm here to show you how we can do this with scraping Reddit, formatting the content, and posting in on X.

Out of personal interest, I go down the rabbit hole of crypto. We specify the Agent to scrape the CryptoCurrency subreddit. This subreddit is highly active, contains valuable information but it takes time to split through all the nonsense. The Agent is doing this for me. The result of this is the creature that has come to life - Agentic (4GENTIC)

Subscribe now

For this adventure, we're teaming up with Crew AI, a AI Agent framework created by João Moura. It's open for everyone to use, which means you, me, and anyone with a curious mind can see how it works and even tweak it to do new things. Let's explore how to put together our own little digital assembly line using Crew AI, and remember, this is just the start. The real magic happens when you start playing with these ideas yourself. Let's get started!

Crew AI: An Open Source Agent Framework

Crew AI offers a framework specifically designed to orchestrate the creation and management of AI agents. At its core, Crew AI facilitates the development of autonomous agents capable of performing a wide range of tasks, from data scraping to content formatting and even social media management.

By leveraging Crew AI, users can assemble teams of digital agents, each with specialised roles or tasks. These could include an Explorer Crew tasked with gathering data from the internet, a Formatting Crew responsible for organising this data in a readable or presentable format, and a Social Media Crew designed to distribute content across various social media platforms. This modular approach allows for flexibility and customisation, catering to the specific needs or goals of any project.

The framework's design encourages experimentation and innovation, serving not just as a tool but as a platform for learning and exploration in the field of AI. Whether it's for educational purposes, personal projects, or professional development, Crew AI offers a solid foundation for building and deploying AI agents in the real world.

Subscribe now

Tutorial Overview: Scraping Reddit, Formatting, and Posting on X

The plan is clear. There are three tasks: Browse Reddit, format the output and post to X. I will create a single Crew to browse Reddit and format the output. It doesn’t require any creativity to post the result to X. Therefore, I apply a posting function to the formatted output instead of adding it to the workflow.

The steps to take are:

1. Browse Reddit: create Agent and Task

2. Format the output: create Agent and Task

3. Kickoff the Crew

4. Post to X

The project consists of four files:

• main.py which contains the workflow

• custom_functions.py which contains the X posting code

• .env which contains your environment API keys

• requirements.txt which contains all the necessary packages

1. Browse Reddit: create Agent and Task

• We need to setup a BrowseTool class to prepare the browsing:

class BrowserTool:
    @tool("Scrape reddit content")
    def scrape_reddit(max_comments_per_post=7):
        """Useful to scrape a reddit content"""
        reddit = praw.Reddit(
            client_id=client_id_re,
            client_secret=client_secret_re,
            user_agent="user-agent",
        )
        subreddit = reddit.subreddit("CryptoCurrency")
        scraped_data = []

        for post in subreddit.hot(limit=12):
            post_data = {"title": post.title, "url": post.url, "comments": []}

            try:
                post.comments.replace_more(limit=0)  # Load top-level comments only
                comments = post.comments.list()
                if max_comments_per_post is not None:
                    comments = comments[:7]

                for comment in comments:
                    post_data["comments"].append(comment.body)

                scraped_data.append(post_data)

            except praw.exceptions.APIException as e:
                print(f"API Exception: {e}")
                time.sleep(60)  # Sleep for 1 minute before retrying

        return scraped_data

• Define an Agent (explorer) and a Task (task_report):

explorer = Agent(
    role="Senior Researcher",
    goal="Find and explore the most exciting developments on CryptoCurrency subreddit today",
    backstory="""You are and Expert strategist that knows how to spot emerging trends and important events in Crypto, blockchain and web3 TODAY. 
    You're great at finding exciting projects on CryptoCurrency subreddit. You turned scraped data into detailed reports with titles
    of the most exciting developments in the crypto world. ONLY use scraped data from CryptoCurrency subreddit for the report. Make sure not to forget the links to the posts.
    """,
    verbose=True,
    allow_delegation=False,
    tools=[BrowserTool().scrape_reddit] + human_tools,
   llm=llm,  
)

task_report = Task(
    description="""Use and summarize scraped data from subreddit CryptoCurrency to make a detailed report on today's developments in crypto. Use ONLY 
    scraped data from CryptoCurrency to generate the report. Your final answer MUST be a full analysis report, text only, ignore any code or anything that 
    isn't text except for links. The report has to have bullet points and with 5-10 exciting crypto developments. 
    Each bullet point MUST contain 3 sentences that refer to one specific development you found on subreddit CryptoCurrency. Each bullet point must contain a link to the post.  
    """,
    agent=explorer,
    expected_output="""A detailed report with bullet points each containing 3 sentences about a specific crypto development found on the CryptoCurrency subreddit."""
)

2. Format the output: create Agent and Task

• Define an Agent (formatter) and a Task (task_formatter):

formatter = Agent(
    role="Expert Writing formatter",
    goal="Make sure the output is in the right format. Make sure that the tone and writing style is compelling, simple and concise",
    backstory="""You are an Expert at formatting text from technical writers. You can tell when a report text isn't concise,
    simple or engaging enough. You know how to make sure that text stays technical and insightful by using layman terms. You know how to format a report properly.
    """,
    verbose=True,
    allow_delegation=True,
    llm=llm,
)

task_formatter = Task(
    description="""
        Format the explorer's output for Twitter posting by structuring the information according to the following template:
        
        - For each news item, begin with the title.
        - Follow the title with a bullet point listing interesting facts about the news item.
        - Ensure each fact is compelling, engaging, and provides value to the reader.
        - Conclude each news item summary with a bullet point that includes a link to the original post.
        
        The output must be clearly structured, contain accurate and engaging information, and each news item must have a corresponding link. Your task is to verify the structure, assess the compelling nature of the text, and ensure all links are correctly included.
    """,
    agent=formatter,
    expected_output="""
        The formatted output should look like this for each news item:
        
        [Title of News Item]
        - An interesting and engaging fact about the news item. The fact should be compelling and present the news in an engaging way.
        - A direct link to the original post for readers to find more information.
        
        "Title of News Item" should be replaced with the actual title. This format should be repeated for each news item included in the explorer's output, ensuring a consistent and engaging reader experience.
    """,
)

3. Kickoff the Crew

# instantiate crew of agents
crew = Crew(
    agents=[explorer, formatter],
    tasks=[task_report, task_formatter],
    verbose=2,
    process=Process.sequential,  # Sequential process will have tasks executed one after the other and the outcome of the previous one is passed as extra content into this next.
)

# Get your crew to work!
result = crew.kickoff()

4. Post to X

• Split the result into sections to respect the limit rules of X:

def split_section_into_tweets(section, limit=280):
    """Splits a section of the message into parts that fit within the tweet limit, accounting for URL shortening."""
    url_placeholder = "https://t.co/xxxxxxxxxx"  # Placeholder for URLs, adjust length as needed
    urls = re.findall(r'(https?://\S+)', section)
    adjusted_section = re.sub(r'(https?://\S+)', url_placeholder, section)  # Replace URLs with placeholder

    if len(adjusted_section) <= limit:
        return [section]  # Return the original section if it fits
    
    # Adjust the splitting logic to account for item markers
    item_pattern = re.compile(r'\d+\.\s')  # Pattern to detect item numbering
    sentences = adjusted_section.split('. ')
    tweets = []
    current_tweet = ""
    
    for sentence in sentences:
        if len(current_tweet) + len(sentence) + 1 > limit or item_pattern.match(sentence):
            if current_tweet:
                # Replace placeholders with actual URLs when adding tweet to list
                for url in urls:
                    current_tweet = current_tweet.replace(url_placeholder, url, 1)
                    urls = [u for u in urls if u != url]  # Remove used URL
                    break
                tweets.append(current_tweet)
                current_tweet = sentence
            else:
                # Directly add if alone it exceeds the limit
                tweets.append(sentence)
        else:
            if current_tweet:
                current_tweet += ". "
            current_tweet += sentence
    
    # Replace placeholders with actual URLs for the last tweet
    for url in urls:
        current_tweet = current_tweet.replace(url_placeholder, url, 1)
    if current_tweet:
        tweets.append(current_tweet)
    
    return tweets

• Post the split Tweets

def post_to_twitter_callback(task_output):
    print("Callback function has been triggered.")
    # Initialize Tweepy client (assuming bearer_token, api_key, etc. are defined)
    client = tweepy.Client(bearer_token=bearer_token_tw,
                        consumer_key=api_key_tw,
                        consumer_secret=api_key_secret_tw,
                        access_token=access_token_tw,
                        access_token_secret=access_token_secret_tw)


    # Assuming 'result' contains the content to be tweeted
    tweets = split_section_into_tweets(task_output)
    total_tweets = len(tweets) + 1  # +1 for the initial header tweet
    sleep_duration = 0.4
    reply_to_tweet_id = None

    # Prepare the initial tweet
    today_date = datetime.datetime.now().strftime("%B %d, %Y")
    initial_tweet_text = f"Your Daily Crypto Updates of {today_date}  (1/{total_tweets})"

    try:
        # Post the initial header tweet
        response = client.create_tweet(text=initial_tweet_text)
        reply_to_tweet_id = response.data['id']
        print(f"Posted initial Tweet ID: {response.data['id']}")
        time.sleep(sleep_duration)
    except Exception as e:
        print(f"Error posting initial tweet: {e}")

    # Post the rest of the tweets
    for index, tweet in enumerate(tweets, start=2):  # Start from 2 since the first tweet is the header
        tweet_with_count = f"{tweet} ({index}/{total_tweets})"
        if len(tweet_with_count) > 270:
            allowed_length = 270 - len(f" ({index}/{total_tweets})") - 3
            tweet_with_count = tweet[:allowed_length] + "..." + f" ({index}/{total_tweets})"
            print(tweet_with_count)
        try:
            response = client.create_tweet(text=tweet_with_count, in_reply_to_tweet_id=reply_to_tweet_id)
            
            reply_to_tweet_id = response.data['id']
            
            print(f"Posted Tweet ID: {response.data['id']}")
            print(f'sleeping for {sleep_duration} sec...')
            time.sleep(sleep_duration)
        except Exception as e:
            print(f"Error posting tweet: {e}")

Subscribe now

Results and Future Development

It’s running for three days now and the results can be find here:

• Daily Crypto Update 9 April

• Daily Crypto Update 10 April

• Daily Crypto Update 11 April

As can be seen, formatting can be better. Nevertheless, interesting findings are posted over the last few days. One can imagine that such Agentic workflows open up opportunities for more complex tasks. In my Github, the full code can be found for the first iteration. My plan is to build this project out in the following way:

• Set up three Crews:

  • An Explorer Crew which consists of multiple Agents scraping the depths of the internet looking for Crypto developments (think about Telegram and Discord channels)

  • A Formatter Crew which takes responsibility for formatting the scraped content into the desired format for a specific social media platform (think about a blog for Substack and a bullet-point list for Discord)

  • A Social Media Crew which makes sure the formatted content is posted at a predetermined frequency to several social media channels.

In addition, currently, I run the main.py file once per day. Once I’m satisfied, I’ll place the code into a while loop and deploy it to a cloud service to let it run automatically.

Call to Action

I’ll encourage you to explore such Agentic Workflows yourself. These workflows start to get more and more traction. Try to automate something. Just play around with it.

Now, I invite you to dive into the world of Agentic Workflows on your own. As these powerful tools gain momentum in the tech world, there’s no better time than now to get your hands dirty. Challenge yourself to automate a task, big or small. Experiment, tinker, and see what you can create - it's still a playground for the curious. So, take the leap and start playing around with Agentic Workflows today. Who knows what you might discover or achieve?

Thanks,

Michiel & Agentic

Share

Leave a comment

Imagine a future where you're brainstorming for a new project at work, and you say, "Generate a project plan based on the latest trends in renewable energy technologies." Instantly, an AI Agent scours the internet, reviews the most recent academic papers, synthesizes market reports, and compiles a comprehensive plan complete with potential challenges, innovators in the field, and a timeline for development. This isn't a simple task automation; it's an AI Agent performing, arguably, creative thinking and strategic planning—a capability we're just starting to scratch the service of.

Subscribe now

Understanding AI Agents: The Basics

In the simplest terms, an AI Agent is a piece of software designed to perform tasks, some as simple as sending an email on your command, others as complex as navigating a car through city traffic. These Agents are a bit like invisible helpers, living in our smartphones, cars, home appliances, and even on websites, making decisions, and taking actions to assist us in our daily lives.

Unlike zeroshotting an LLM (prompting a model like GPT to generate final output without revising its work), an Agent can autonomously iterate over a series of tasks. It can learn from data, make decisions, validate those decisions and go back to an earlier stage based upon new information it got from experience.

Think about how you write a document like an essay. You make an outline, do research, write a draft, evaluate the draft, revise the draft, and repeat this process until you’re satisfied. This is how Agent workflows can work as well.

Yet these Agents didn’t come out of nowhere. Let’s move on to explore their development history and how they've gradually become more integrated into our (digital) lives.

The Evolution of AI Agents

AI Agents have come a long way from their inception, evolving through stages of development that reflect broader shifts in the field of AI. The journey from simple automated systems to the complex, autonomous Agents we see today is a testament to decades of research and technological advancement.

The early days of Agent evolution were marked by rule-based systems, where Agents operated under a set of predefined rules. These were straightforward but limited in scope, as their ability to handle tasks was strictly bound by the instructions programmed into them. For example, ELIZA, an early NLP computer program created in the mid-1960s, could simulate conversation but lacked any genuine understanding or ability to learn.

The introduction of machine learning algorithms in the latter part of the 20th century marked a significant leap forward. Agents began to learn from data, improving their performance over time without being explicitly programmed for every possible scenario. This shift from rule-based to learning-based Agents enabled more flexible and adaptive applications, from recommendation systems like those used by Netflix to the more sophisticated virtual assistants we interact with today.

In recent years, the emergence of neural networks and deep learning has further accelerated the capabilities of Agents. These technologies allow them to process and interpret vast amounts of unstructured data, leading to unprecedented levels of understanding and autonomy. Agents can now recognize speech, interpret complex patterns, and make decisions with a degree of nuance that closely mirrors, and often surpasses, human cognition (and latency, which enables real-time inference. See Groq).

As we look at the evolution of AI Agents, it's clear that their development parallels key advancements in AI itself, with each stage bringing us closer to creating Agents that can seamlessly integrate into our lives, providing assistance, and making decisions with a high level of sophistication. Let’s take a look at the industry nowadays.

Subscribe now

Current Landscape and Key Players

The autonomous AI and autonomous agents market, currently valued at approximately USD 4.2 billion, is forecasted to soar to USD 88.1 billion by 2032, thanks to advancements in AI itself.

Important to understand is that an Agent uses an LLM as its ‘brain’. Therefore, the companies that have there own LLM are likely to be in the center of the Agent industry as well.

Below a couple of these companies and their flagship models:

1. Microsoft/OpenAI - GPT4

2. Anthropic – Claude

3. Cohere - Command

4. AI21Labs – Jurassic-2

5. Google: Gemini

Nevertheless, there are plenty of decent Open Source ‘brains’ that can be used by third parties to build custom solutions on top of them. On top of this, due to the open source nature, newcomers can join the space with relative ease by forking such an open source model and start developing it based upon their specific preferences.

Below a couple of companies which opened up their code for the public:

1. Databricks: DBRX (just came out)

2. Mistral AI: Mistral Large

3. EleutherAI: GPT-Neo

4. Meta: Llama 2

5. X – Grok

Narrowing down to Agents, interesting projects are popping up from different companies from all sorts of sizes. While there is Microsoft’s AutoGen as framework for multi-agent conversations, start- and scaleups are entering the space rapidly. At website Easy With AI, there is a nice overview of Agents, both close and open sourced. A standout example of innovation in this space is Devin, an AI software engineer that exemplifies the complex decision-making and learning capabilities of modern AI agents (if Cognition can move the demo qualities to production).

Some other cool examples I came across:

• Relevance AI: Workforce Automation. I recommend the following tutorial where an entire team of Agents is working on creating the morning news for a company. No interns necessary anymore :)

• Hyper Write: from a writing assistent to a full PA. Booking flights, ordering a pizza, and many more possibilities.

• Crew AI: Open source framework for orchestrating role-playing, autonomous AI agents. In the next edition, we are going to use Crew AI to build an Agent Workflow ourselves.

One of the best business cases I’ve seen so far is the AI Assistant of Klarna (a shop now pay later fintech). Powered by OpenAI, this Assistent has had 2.3 million conversations (two-thirds of Klarna’s customer service chats), doing the work of 700 full-time human agents, and is on par with human agents in regard to customer satisfaction score. It is estimated that this Agent alone adds a 40 million USD profit to Klarna in 2024. Well done on you Sebastian and team!

Subscribe now

Potential and Pitfalls

I had the chance to chat with Daoud Abdel Hadi about his Ted Talk on AI Agents. Daoud is the Data Science lead at Eastnets, a payment solutions company for the financial services sector. At Eastnets, he is exploring the options to use Agents to create co-pilots that can help investigators investigate fraud and other financial crimes. In addition, he is the co-founder of an AI automation platform. He is using all sorts of things like retrieving information from documents, to researching competitors, to writing content.

We both hold the position that Agents have the potential to revolutionize how we interact with technology, automate repeating tasks, and analyze complex data sets with unprecedented speed and accuracy. By acting as personal assistants, data analysts, and decision-makers, they can significantly enhance productivity, provide personalized experiences, and open new avenues for innovation across industries.

But as much as AI Agents can represent a leap forward in technology, they also bring up important questions. How do they make decisions? What ensures they act in our best interest? And what does their presence mean for the future of work, privacy, and ethical considerations in technology?

With Shakespeares quote, he highlights the importance of personal judgement and skepticism towards intermediaries. This applies to outsourcing tasks to AI Agents as well. It’s important to start formulating answers to the aforementioned questions early on to prepare ourselves for the next generation of models to come. Similar to AI, there are significant ethical challenges, particularly concerning privacy, bias, and discrimination:

• Privacy Concerns: AI Agents' reliance on vast data pools raises questions about the security and protection of personal information. Issues like data misuse, surveillance, and unauthorized information sharing highlight the need for robust privacy protections​​.

• Bias and Discrimination: The risk of perpetuating existing societal biases through AI Agents is a pressing concern. Algorithms may display racial, gender, socioeconomic, and cultural biases, leading to unfair treatment of certain groups. These biases can affect everything from facial recognition technology to hiring processes and loan approvals, potentially reinforcing societal inequalities​​.

To mitigate these concerns, transparency in data collection, rigorous bias detection and mitigation, the development of ethical guidelines and regulations, and public awareness are crucial. Transparent data practices, diverse training datasets, and public education can help ensure AI Agents contribute positively to society without compromising ethical standards​​.

E2.2: Create your own Agent Workflow

As we've explored the evolving world of AI Agents, we've uncovered both the incredible potential and the significant challenges they present. Yet the single most important thing is that we should familiarize ourselves with the techniques and do the actual building. Therefore, in the next edition, we are going to create a project with true impact straight away. Using Crew AI, we are going to build an Agent Workflow that does research into a particular niche, writes a blog, summarizes the blog, and finally posts its findings on X on a daily basis. I’m pretty excited, I hope you as well. Thanks for reading, and I hope to see you back in two weeks.

Thanks!

Michiel

Share

Share The Application Layer

Subscribe now

In this edition, we'll focussing on what diffusion models are, a brief explanation of the training process and finally show how you can use a pre-trained model to create your own video from a text prompt in Python. This hands-on approach will show you how accessible this technology has become for creators and developers alike.

What are these models?

Diffusion models, a subset of generative models, are designed to create new data that resembles the training data, such as producing unique human faces from a dataset of faces. They achieve this by gradually evolving from a basic, often normal, distribution to the complex distribution of the training data, using reversible steps. The process involves starting from a simple noise distribution and progressively transforming it to emulate the detailed features of the target data distribution.

Though models often blend various components, we generally categorise them into four main types:

1. Denoising Diffusion Probabilistic Models (DDPMs): These are the foundational models that simulate the reverse process of gradually removing noise from a signal to generate samples. Their uniqueness lies in their straightforward approach to model the data generation as a reverse diffusion process, primarily applied in generating images and other data types from pure noise.

2. Conditional Diffusion Models: Characterised by their ability to generate data based on specific conditions or contexts (e.g., generating images from textual descriptions), these models differentiate themselves by using additional information to guide the generation process, making them suitable for tasks requiring targeted output.

3. Latent Diffusion Models: These models operate uniquely in a compressed, latent space instead of directly in the data space (like a ZIP file reduces file size without losing information). This approach makes them efficient for dealing with high-dimensional data, distinguishing them by their use of latent space transformations for the diffusion process.

4. Score-Based Generative Models (SBGMs): These models share the fundamental principle of generating data by transforming a noise distribution into a data distribution but focus specifically on modelling the score of the data distribution. The score here implies the direction in which to adjust noise to more closely resemble the target data. SGMs use this score, or gradient, in a continuous framework (instead of discrete steps) guided by stochastic differential equations (SDEs), distinguishing them within the broader category of diffusion models.

These models, or combinations of them, have the ability to generate highly realistic images and videos, each employing a unique approach to manage and manipulate the diffusion process.

Subscribe now

Training a diffusion model: A Quick Guide

Training diffusion models is a resource-intensive task often beyond the reach of individual devs. Yet, understanding the process from a high level provides valuable insights into how these models learn to transform text into video. Therefore, below the general steps of training such models:

• Data Gathering and Preparation: Begin by gathering a good dataset, ideally with diverse examples. For diffusion models focusing on text-to-video, ensure your dataset includes video clips along with corresponding textual descriptions. Preprocessing steps like normalisation and resizing may be necessary to align with model input requirements.

• Model Selection: Choose a diffusion model architecture that best fits your project, considering factors like the complexity of data and desired output. Setting up your development environment will involve libraries specific to your chosen model, such as PyTorch for computational graph management and model training.

• Model Training: Training diffusion models demands robust hardware, typically GPUs, due to their intensive computational requirements. The training phase encompasses initialising the model, defining a suitable loss function to measure how well the model's output matches the target data, and selecting an optimiser to adjust model parameters and minimise the loss. Regular checkpoints are crucial for monitoring progress and facilitating model recovery.

• Model Evaluation: Evaluate the model's effectiveness using a mix of quantitative metrics (such as loss reduction over time) and qualitative assessment (visual inspection of generated videos). Feedback from this stage is essential for refining model parameters and training approaches.

• Implementation: Once satisfied with the model's performance, prepare it for deployment. This step may involve converting the model for compatibility with different platforms or optimising it for faster inference. Continually update the model with new data and insights to maintain and improve its performance over time.

If you want to train a diffusion model from scratch (to generate images), I’d recommend this tutorial (which uses a Butterfly dataset). Yet, generally, the best results, and with less hardware requirements, come from finetuning a base model on your own data. An excellent tutorial can be found here. However, it could also be the case that there is already a model in place for your need. That’s where inferencing comes into play.

Inferencing a diffusion model: A Coding Example

Inferencing is the phase where a trained diffusion model applies what it has learned to generate new data from given inputs, such as turning text prompts into videos. This process uses the model's knowledge to predict or "infer" outputs without additional training, showcasing the model's ability to create realistic and coherent visuals based on the initial textual description. We are going to use a stable diffusion tex-to-video model (a combination of a latent and conditional diffusion model) from the Hugging Face website. The aim is to generate a short video based on our own textual input.

1. Create your virtual environment

For Mac, create a new folder, navigate to this folder via the terminal and run the following:

virtualenv venv
source venv/bin/activate

For Windows, create a new folder, navigate to this folder via Command Prompt and run the following:

virtualenv venv
venv\Scripts\activate

If virtualenv is not recognised, you can install it using pip3 install virtualenv. If it still doesn’t work, you probably need to add virtualenv to your path.

2. Install the libraries

pip3 install diffusers transformers opencv-python numpy torch

3. Do the imports

#imports
from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.utils import export_to_video
import cv2
import numpy as np
import torch

3. Initialise the pipeline

pipe = DiffusionPipeline.from_pretrained("damo-vilab/text-to-video-ms-1.7b", torch_dtype=torch.float32)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

This code initializes a pre-trained diffusion model for generating videos from textual descriptions. The first line line loads a specific model trained to create videos based on input text, setting it to operate in floating-point precision for computational efficiency. Following that, the second line configures the model's scheduler, which manages the diffusion steps during video generation, ensuring the process aligns with the model's training configuration for optimal performance.

We use model damo-vilab/text-to-video-ms-1.7b made by Alibaba / DAMO Vision Intelligence Lab. Yet you can take any other model of your choice (and make some minor code changes). Important to know here is that we use a CPU instead of a GPU implementation. Therefore, everyone can execute the code on its own device.

4. Define your prompt

num_inference_steps = 25
prompt = "A panda eating bamboo on a rock"
video_frames = pipe(prompt, num_inference_steps=num_inference_steps).frames

Define your input in the prompt line. The video_frames line line triggers the model to start the generation process, ultimately producing a sequence of video frames that visually interprets the given prompt. You can increase the number of inference steps to improve quality at the expense of processing time.

5. Define the video properties

fps = 16  # Frames per second
height, width, channels = video_frames[0].shape[1:]  # Get shape from the first batch
vid_name = f"{prompt[:50]}" + " nis" + f"{num_inference_steps}" + ".mp4"
video_writer = cv2.VideoWriter(vid_name, fourcc, fps, (width, height))

This code sets the frame rate (in frames per second) for the video, extracting the height, width, and number of color channels for each frame from the first batch of generated frames. It defines the video codec (that manage how videos are compressed) as 'mp4v', a common format for video files. The Video Writer object is then initialized to write the video file.

6. Write frames to video

for batch in video_frames:
    for frame in batch:
        # Ensure frame is normalized and in uint8
        frame_uint8 = (frame * 255).astype(np.uint8)
        video_writer.write(frame_uint8)

video_writer.release()
print(f"{vid_name} done")

This code snippet normalises each frame in the video by scaling the pixel values of each frame to span the full range of 0 to 255, a common scale for digital images, ensuring compatibility with video codecs. Converting to 8-bit unsigned integer format specifies that each pixel's color intensity is represented by a value between 0 and 255, fitting the standard format for image and video files. It then writes each frame to a video file using OpenCV, finalises the video writing process.

What we get out is the following video:

As can be seen, this video is short and not meeting production quality standards yet. You can increase the length by using a GPU implementation and play around with the number of inference steps to improve the quality. If not satisfied, try another model or consider finetuning a model on your own data. At the bottom of this post, other video examples can be found. For playing around with prompts without code (and, depending on your machine, probably higher processing speed), you can leverage the UI here.

To wrap up, this edition demonstrates how to leverage open-source diffusion models to generate video content, showcasing a practical application of such models. As the field evolves, we can expect the quality of these models to improve, potentially making open-source solutions a standard in the industry. This not only democratises access to powerful generative tools but also encourages innovation and collaboration in the development of high-quality video content.

The future is bright. See you in two weeks. Thanks for reading.

Share The Application Layer

In today's world, making videos from text is a game-changer. This tech lets us turn any written idea into a video quickly. You type something, and boom, you have a video of it. It's going to change how we work and play. Smart tech folks are all over this, making it better every day. And guess what? We're going to dive deep into how this all started and where it's going, especially looking at the most recent breakthrough of OpenAI. So, let's get into it and see how words turn into moving pictures.

Unlocking Personalised Video Content Creation

One of the most groundbreaking applications of text-to-video technology lies in its power to create highly personalised video content. This isn't just about choosing your adventure in movies or games; it's about crafting entirely unique experiences for each viewer or player. Here's how this tech is starting to change the game:

• Personalised Movies and Shows: Imagine watching a film where the storyline adapts based on your preferences, or where you can see yourself as a character in the plot. Text-to-video technology could make this possible by generating scenes on-the-fly, tailored to the viewer's choices or interests.

• Customised Gaming Experiences: In gaming, this technology can be used to create dynamic narratives where the game's world and story change based on the player's actions or decisions, providing a truly personalised adventure that's different each time you play.

• Educational Content: For education, it means lessons that adapt to each student's learning style or interests, making learning more engaging by incorporating elements that resonate personally with them.

• Marketing and Advertising: Brands can create customised video messages for their customers, making each interaction personal and increasing engagement by speaking directly to the individual's preferences and behaviours.

The potential for personalised video content is big, ranging from entertainment to education, and beyond. By harnessing text-to-video technology, creators and developers have a powerful tool at their disposal to craft experiences that are not just immersive but deeply personal. But why is this relevant now? Let’s explore the evolution of the models that make this all possible.

Subscribe now

The Evolution of Generative Models

Generative models have come a long way since their inception. Initially, computer graphics and simple algorithms were used to create digital images and animations. These early models required extensive manual input and were limited in their complexity and realism.

• Early Days (1970s-1980s): The journey began with basic computer graphics, where the creation of digital content was heavily manual and time-consuming. Artists and programmers had to painstakingly design each element, limiting the scope and scale of projects.

• Rise of Machine Learning (1990s-2000s): The introduction of machine learning models marked a significant turning point. Algorithms could now learn from huge datasets, creating more complex and realistic images. However, these models still struggled with understanding and generating coherent long-form content.

• Deep Learning Revolution (2010s): The advent of deep learning marked a significant transformation. Neural networks, particularly Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), started producing high-quality images that were increasingly realistic. These models learned to mimic the distribution of real images, generating new content that was often indistinguishable from real-world examples.

• From Images to Video (late 2010s): Transitioning from generating static images to dynamic videos was a natural progression but posed significant challenges. Videos require a temporal coherence and a much higher level of complexity, as they are essentially sequences of images that need to maintain consistency and tell a story.

• Recent Breakthroughs in text-to-video (2020s): The latest breakthroughs in text-to-video generation represent a culmination of years of research and development. Leveraging advancements in natural language processing (NLP) and deep learning, current models can now understand textual descriptions and generate corresponding video content. This involves not just creating visually appealing frames but ensuring that these frames are woven together into a coherent and compelling narrative.

Breakthroughs in Text-to-Video Generation

The journey from text to vivid, moving images represents an exciting frontier in AI. This transformation didn't happen overnight; it's the result of years of incremental breakthroughs. The evolution has been particularly rapid in recent years, with several key developments pushing the boundaries of what's possible.

The Foundation: GANs and Transformers

Generative Adversarial Networks (GANs), introduced in 2014, marked a significant milestone. GANs use two neural networks—the generator and the discriminator—in a competitive setup, where the generator learns to create data increasingly indistinguishable from real data, while the discriminator learns to tell the difference. Although initially focused on images, the concept proved pivotal for video generation, teaching machines not just to replicate but to imagine.

The advent of transformers in 2017, in the famous paper “Attention is All You Need”, initially for natural language processing tasks, revolutionised AI's capacity to understand and generate human-like text. Transformers' ability to handle sequential data made them prime candidates for adaptation in video generation, where both the narrative (text) and the sequence (video frames) are crucial.

The Role of Diffusion Models

As the field evolved, diffusion models emerged as a significant breakthrough, enhancing the realism and coherence of generated video content. Unlike GANs, diffusion models generate high-quality images and videos by gradually refining noise into structured visual data. This process, akin to reverse diffusion, allows for the creation of complex, dynamic scenes with unprecedented detail and fidelity. The adaptability of diffusion models to video generation marked a pivotal advancement, offering new possibilities for creating seamless, temporally coherent video sequences from textual prompts (like the one you see below).

OpenAI's DALL·E and Sora: Milestones in Generative AI

The path to sophisticated text-to-video generation was significantly paved by OpenAI's developments in text-to-image models, most notably with the DALL·E series. DALL·E, introduced in 2021, showcased the ability to generate highly detailed and contextually relevant images from textual descriptions. This breakthrough demonstrated the potential of combining natural language understanding with generative models to create visual content that closely aligns with human imagination and descriptions.

Building on the success and principles of DALL·E, OpenAI introduced the Sora model, a leap forward in text-to-video generation. Sora exemplifies how far the technology has come, capable of generating high-quality, contextually relevant video clips from textual prompts. This achievement highlights the potential of combining advanced natural language understanding with cutting-edge video generation techniques.

Challenges and future Directions

The advancement of text-to-video generation technology faces both technical challenges and ethical considerations, even as it opens new possibilities in media creation.

Key Challenges

• Temporal Coherence: Maintaining consistency over longer videos is difficult, as early mistakes can propagate, affecting overall quality.

• Computational Demands: High-quality video generation requires substantial computational resources, limiting accessibility.

• Ethical Concerns: The potential for misuse, such as creating misleading content or deepfakes, and questions around copyright and creative ownership are significant. As a potential solution, legal reforms could mandate disclosure of AI involvement in content creation, ensuring transparency and accountability. Technically, investing in detection tools could help differentiate between genuine and manipulated content, safeguarding against misinformation.

Looking Ahead

• Efficiency and Accessibility: Future efforts will likely focus on reducing computational requirements to make these technologies more accessible to a wider range of users.

• Enhanced Realism: Improving the emotional depth and realism of generated content will be crucial for more engaging and relatable videos.

• Interactive Generation: Advancements may enable real-time, interactive video generation, expanding possibilities in gaming, virtual reality, and live broadcasts.

As we navigate these challenges, the goal will be to leverage text-to-video technologies responsibly, enhancing creative expression while addressing ethical concerns.

P2: Create Your Own Video from a Text Prompt

In two weeks, in the second part of the article series, we'll dive deep into the practical application of diffusion models, guiding you through the process of transforming your imaginative text prompts into captivating video content using your own Python code. Stay tuned!

Subscribe now

The Application Layer is a bi-monthly series that takes you into the heart of technological progress, where the real magic isn't in the raw power of GPUs or complex infrastructure, but in the realm where ideas come to life: The Application Layer.

In the first edition of each month, we explore high-level concepts, industry trends, and groundbreaking ideas—perfect for everyone, regardless of your technical background. This is where we unpack how AI is influencing various sectors and what it may mean for the future.

The second edition each month is where we roll up our sleeves. Designed for those with a technical interest, it focuses on building and implementing the concepts discussed in the first edition. It's an in-depth look at the application of these ideas, ideal for tech enthusiasts keen on understanding the 'how' behind the 'what.' Ideally, one further builds on top of these implementations and creates value out of it.

Clarifying the Buzzword

In an attempt to prevent ambiguity along the way, I’d like to share the definition that Marc Andreessen, general partner of legendary Venture Capital firm a16z, holds about AI:

“A short description of what AI is: The application of mathematics and software code to teach computers how to understand, synthesize, and generate knowledge in ways similar to how people do it. AI is a computer program like any other – it runs, takes input, processes, and generates output. AI’s output is useful across a wide range of fields, ranging from coding to medicine to law to the creative arts. It is owned by people and controlled by people, like any other technology.

A shorter description of what AI isn’t: Killer software and robots that will spring to life and decide to murder the human race or otherwise ruin everything, like you see in the movies.

An even shorter description of what AI could be: A way to make everything we care about better.”

In my opinion, until proven wrong (which I don’t hope), by and large, this definition is valid.

Subscribe now

Rethinking Our Priorities

Think of how common software is in our lives now. That’s where AI is heading to. It’s going to be everywhere, helping us do all kinds of things. This is what's happening in the Application Layer. It's being used by businesses to reinvent how they work. For example, educators are leveraging AI to offer high-quality learning solutions to virtually everyone at nearly zero marginal costs, doctors are enhancing patient treatment quality with the support of AI assistants, and software engineers are building companies at a fraction of the costs of the incumbents by leveraging Copilot and ChatGPT (and soon automated unit testing using Meta's TestGen-LLM!).

All these different uses of AI are just as important as making it smarter. They're about finding new ways to improve things we do every day. This is how this technology will become a normal part of life, not just something tech bro’s talk about.

The Application Layer Pivot

Venture capitalist Chamath Palihapitiya made a point on The All In Podcast E167 (16:30 onwards) that's really about where the big opportunities are with AI. He mentioned how, back in the day, Cisco was a big deal because of the internet boom. But then, the real winners became the ones who used the internet to build new things—like all the apps and services we use today (Cisco was still doing great back then btw). He sees the same thing happening with AI. It's not just about the companies making the chips, like NVIDIA, but more about what people do with AI. One could thus argue that the true potential resides in the Application Layer—the myriad ways we can apply these tools to solve real-world problems.

The Application Layer isn't just about leveraging AI; it's about augmenting human potential. It's where entrepreneurs will find their next venture, where educators will discover new teaching tools, and where artists will birth new forms of expression. We are witnessing an increase in individual productivity that we have rarely seen historically. It's the playground for practical innovation, the bedrock of businesses that haven't been dreamed of yet, and the frontier for an inclusive technology revolution.

Share The Application Layer

A Collective Journey

Life's fast, and technology's even faster. One day, we're all about green energy, and the next, we're figuring out how AI fits into our lives. It's a lot, and I'm here to help us both keep track.

In this newsletter, I’m aiming to break down the big stuff. We'll look at where business is going and what that means for us. Plus, I'll share how I'm using code to turn ideas into something real.

I'm keeping an eye on the big picture and the small details. I’m not just following the latest trends; I’m getting my hands dirty understanding them. And I'm not just using new tech; I’m learning it inside and out to see how it can help us.

But I'm not doing this alone. While I'm the one putting this together, it's about all of us. I want to start conversations, share what I learn, and hear from you too. I’m reaching out to pros, hobbyists, and anyone who’s curious to join in.

Leave a comment

Every two weeks, I'll share what I've found. We’ll start with the broad view of what’s new in tech and business, and then get into the practical side of how it all works.

Let’s go on this tech adventure together.

A Personal Note

I’m an Amsterdam based Data Scientist that is gradually moving towards Software Development. After leaving Microsoft last year, I started my own company: Melchior Analytics. At MA, we're developing software solutions tailored for the financial sector, with our most notable clients being Booking.com and Maven11. On top of this, I’m co-developing an IOS application, which we aim to launch in H1, that let individuals keep track of their expenses in the easiest and most complete way. In my downtime, I enjoy the strategic game of chess and the endurance challenge of running marathons. Feel free to reach out if you want to chat about the newsletter or just have something fascinating to talk about.

Share The Application Layer