When people imagine multiplayer AI, they usually think of products like Claude Tag or PostHog in Slack.

These are fine for simple tasks, like fixing minor bugs and querying data, but most work happens outside Slack across dozens of apps in messy, nonlinear processes.

This creates a UI bottleneck that forces people to translate rich information into flat text, only to be transformed back into the original shape by agents at the end of the funnel.

We wanted to build a collaboration tool that’s designed for agents from the start, so we’ve been exploring alternatives beyond Slack for multiplayer AI these last few months.

Here are a few of the lessons we’ve learned so far about building multiplayer in PostHog Desktop.

1. You need to establish a shared reality

Context is important for any AI system, but the key word in multiplayer is shared context. Siloed context wastes tokens, duplicates work, and creates inconsistencies.

Say two teammates attend the same meeting but write down slightly different definitions of a goal metric. Their context now differs, which means their work may diverge without them even knowing. With agents, these minor differences can compound into completely different realities at scale.

A shared context system reduces that risk by being a source of truth. This also decreases the friction of information handoff, since teammates can just check a shared resource instead of asking and waiting on each other.

There’s no reason to not have a shared context system since it can be as simple as a team Notion page connected via MCP. The real challenges are in keeping it up to date, trustworthy, and complete.

How we’re building it

In chat-based systems, shared context is basically free since you can infer what’s in scope from the root conversation. When the PostHog in Slack bot gets tagged in a thread, it can just read what people have said in chat so far. It also has context from the relevant PostHog project for that organization via our MCP.

But our approach to multiplayer in PostHog Desktop is built on Spaces, not conversations. Spaces are like “rooms” that hold people, agents, and work objects in a shared container. They’re entirely user-defined, so there’s no way to automatically identify and initialize shared context.

We first tried to solve this by asking users to write down the purpose of a Space in a CONTEXT.md at creation time and update it regularly. As you’d expect, that never actually happened. Over 90 days, only 64 users ever started a shared context file, and only 14 users ever edited it.

We’ve since been exploring automatic context maintenance for Spaces through an AI-powered context layer. It starts out almost empty and grows by running a nightly “dreaming” task that records what happened that day in a version-controlled Markdown wiki.

This is similar to what others might describe as an implicit knowledge layer, or company brain. The actual work objects like docs, PRs, and tickets live elsewhere; the context layer just looks at them to extract information:

A key part of the design is that it only looks at what was actually shipped, merged, or decided in a day. If the context layer were to observe items like meeting notes or brainstorming docs, it would likely hallucinate and misrepresent reality and defeat the purpose of being a source of truth.

Here’s an excerpt of a decision log based on this PR, written in present-tense since the context layer’s job is to describe the current state of PostHog:

We’ve only been dogfooding this for a few weeks, but the need has been clear for a while. Almost every external user has requested it in our interviews without even being asked, and there are several startups building similar products like Unblocked, HumanLayer, and Factory.

The takeaway: If you’re building any sort of multiplayer AI, you need shared context to save your team time, energy, and mistakes. This can be as simple as a company handbook, but ideally your system can automatically update and ground itself in reality.

2. Most collaboration happens before and after writing the code

A core part of our vision for Spaces was live session streaming. We wanted teammates to be able to stop, steer, and prompt each others’ agents in real-time.

But so far in our early data, we’re seeing that coding in PostHog Desktop is mostly done solo. Shy, our main product engineer working on Spaces, explained his hypothesis:

Most collaboration in software engineering has always been after code is written, during the review process. And so far, in our internal tests, that still happens more in GitHub rather than our UI.

Something new we’re observing in other products, however, is agent transcripts used as part of the code review process. This is the main bet behind Delta, the recently-launched multiplayer coding app by Zed. In their UI, they place code diffs side-by-side with agent chats and let teammates comment on either in real-time. Copilot also added an agent commit trailer earlier this year that makes it easier for reviewers to understand its changes.

Research suggests code review is more about understanding the reasons behind changes, knowledge transfer, and team awareness.1 So as engineers produce more code than they can realistically review, it’s reasonable to think there will be more emphasis on gathering teammates’ intent, reasoning, and prompting by reading session transcripts, or high-level PR descriptions.

How we’re building it

We’re still talking to users to figure out what this means for how we design shared coding sessions in PostHog Desktop.

One area we are certain about is that collaborative tasks that happen before writing any code are ripe for better AI tooling.

We think artifacts are the answer here. Artifacts are agent-generated web apps that are rendered next to your agent conversation. You might know them as Claude Artifacts, ChatGPT Canvas, or PostHog Canvases. For simplicity, we’ll refer to them broadly as “artifacts.”

Most people think of artifacts as just another way for agents to display their answers to humans, like when you ask Claude to walk you through a codebase with illustrations rather than text. But it’s also useful as a medium for human-to-human communication.

For example, Shy has been rearchitecting some features within Spaces. Instead of sharing his brainstorming session transcripts, he generated this artifact that summarizes only the relevant pieces that he wanted his teammates to comment on:

The obvious benefit is that it’s easier to read than a giant wall of text. But, more importantly, artifacts make work portable. They’re like session state snapshots; if you pass your teammates a well-designed artifact, they don’t need access to your original agent, session, or sandbox.

This concept is the backbone of Linear, Asana, and many other software factory approaches. Persistent objects like tickets and specs hold context so that any agent (or human!) can pick up wherever the work has been left off – anywhere, any time – for future sessions and cycles:

We want to take that session-artifact feedback loop to the next level by turning them into real-time experiences. You can think of it like Figma, but every change a human makes also gets recorded as code. That way, agents can easily consume and work with the same object, making a truly multiplayer experience where every actor is speaking in the same language.

The takeaway: Coding is still primarily a solo activity in our early multiplayer data, and most collaboration is limited to GitHub PRs. The clearest opportunity we see today is to make pre-coding tasks like planning, research, and handoff more multiplayer-friendly with real-time editable artifacts.

3. Multiplayer AI scopes are a human problem

During the earliest stages of design, we couldn’t articulate what defines a Space. Should they be based on teams and org charts? Do we separate them by repos, projects, or tasks? How do people choose where and with whom to collaborate with anyway?

Rather than trying to speedrun the entire discipline of organizational psychology, we left the answer blank to just see what happens.

How we’re building it

Spaces initially had no prescribed structure – anyone could create one, and everything was visible to the whole company.

This actually worked well for us because PostHog operates on very little hierarchy. But most other companies have strict policies around role-based access control, which is why other approaches like YC’s QM place so much emphasis on governance, permissions, and admin settings. This is one of our biggest blind spots, so we’re interviewing lots of people to learn more.

Still, our dogfooding at least helped us identify how and why people set up different Spaces. Even among just ~200 PostHog employees, there were three clear patterns that emerged.

The first was expected. Most teams set up a Space for their team, which ends up looking a lot like our Slack channel directory:

But then people started creating Spaces based on criteria like product areas, specific incidents, or task types. For example, Adam Bowker cycles frequently between #posthog-desktop and a sub-team project called #desktop-onboarding, as well as #builder-relations to look at feedback from our Discord.

We were also wrong about how people would use public vs. private sessions. We thought PostHog employees would default to creating sessions in team Spaces to promote transparency, but they overwhelmingly prefer to start tasks in private. We often see behaviors like:

• Richard moving a pending task from #personal to the shared #gateway Space

• Eleftheria reviewing her own tasks in #personal, then switching to the #ai-infrastructure team-wide feed

• Phil reading shared agent tasks in #release, then going back to #personal to start a new session

Maybe this is just because we all got used to talking to agents informally and asking them dumb questions.

There’s still a lot left to learn about multiplayer scopes, but the hypothesis we’re landing on is that groups form around shared work and shared context – and those usually come from a shared goal.

To experiment with this, we just shipped some simple UI changes that make it easier for users to tell us about those goals. For example, they can select that the goal for the Space is to move a metric and, if so, we ask them about their targets, measurement intervals, and deadlines:

This sounds simple but, with enough data, we could leverage this information to suggest smart defaults for permissions, experiments, dashboards, and more in an AI wizard-like experience.

The takeaway: Organizational scopes are difficult because every company has their own culture with existing habits, tools, and systems. No one has ever fully figured that out for human-only collaboration, and now we’re adding agents to the mix.

Words by Jina Yoon, who wants multiplayer AI to support ranked competitive queues. Graphics by Lottie and Daniel.

⚔️ Grab these knowledge buffs

• The Multiplayer AI Manifesto – Sergey Karayev

• 6-7 loops we use every day to make PostHog self-driving – Andy Maguire

• Multiplayer AI: Competing Architectures for Human-Agent Collaboration – Josh Rosen

• To build or buy feature flags: Non-obvious things to know – Ian Vanagas

• How Anthropic built Artifacts – Gergely Orosz

• Are you data-driven, or are you just busy? – Natalia Amorim

🎮 Join our party

• Technical Customer Success Manager - Americas

• Technical Customer Success Manager - EMEA

• AI Research Engineer

• Finance Manager, Revenue Accounting

This breaks the definition of engineer as “someone who writes code.” But no one thinks engineers will go extinct as soon as this happens. Anthropic has claimed “coding is largely solved,” yet it has 208 open roles with “engineer” in the title.

So what will engineers be doing?

1. Monitoring the situation

Engineers spend a lot of time monitoring running agents, fixing errors, responding to customer requests, evaluating competitors, checking dashboards, strategizing on their role in company priorities, learning new workflows, and babysitting outstanding PRs.

I sum this up as “monitoring the situation.” The result of it is ideas and observations of how they can improve.

Engineers did a lot of this before, especially product engineers, but AI has amplified it. AI increased the amount of information an engineer can process, but it also increased expectations from managers and customers about productivity and polish.

This means engineers can’t be waiting for the right information to come to them. The best ones now create systems to get the information they need at the right time. This is part loop engineering and part context engineering. Engineers at PostHog have standup bots, repo summary scouts, custom setups for monitoring in-progress PRs, feedback surveys, changelog automations, anomaly alerts, and more.

Why is doing all this important? Because without this information, you will likely end up working on the wrong thing and not know it. Knowing all the possible things you could be working on enables you to prioritize and pick the right one.

2. Setting direction

Software might be increasingly driving itself, but you still need to say where you want it to go. This requires synthesizing and analyzing what you observed to figure out what is valuable and worth acting on.

Based on the input an engineer receives, there will be hundreds of potential paths. In the past, product managers or execs might have been responsible for picking the best ones, but it’s now increasingly up to engineers.

Our requests for comments repos are an example of what this looks like. They contain nearly 1,000 direction-setting decisions on topics like dashboard MCP improvements, environments, and picking programming languages. We can look at an RFC for a unified health page as a specific example. It filters information through:

• Personal experience and opinions. Rafa wrote it and the decision was ultimately his, with others adding their thoughts. As a specific example, he predicts: “We have 4 of them right now, but 3 of them were created in the last couple months, so they might start proliferating.”

• Importance to users, customers, and company. The problem statement details the many surfaces for diagnostic information and the problem this creates for users: discoverability, each health page providing an incomplete picture, and the developer overhead of maintaining each one.

• Technical feasibility. The design outlines the specific architecture using Dagster to run checks and store them in Postgres. It also includes how PostHog AI can use this data to provide suggestions.

• How it fits into the broader competitive landscape. Unifying the health page automates a problem the Onboarding team regularly runs into with customers when setting them up for long-term success. Healthier setups are better for our customers and for us.

Without a clear direction, you can feel productive without actually making progress. AI makes this worse by enabling you to add anything you can think of easily. The direction and experience of your entire product can become clouded by features that aren’t what users actually want or you can be quickly led down the wrong path that’s hard to reverse.

3. Deciding how to build and implement

Engineers might not be writing code, but they are still deciding what code gets written. A direction leads to many smaller decisions. Each decision is a hypothesis of what can be built to make progress towards the direction you're going.

What this often looks like is deciding on scope. This requires knowing the codebase, what’s possible, what agents are capable of, and their blind spots.

The importance of this is made clear when I compare myself against real engineers. As a marketer, I monitor the situation and think about the direction products should be going in. I make feature requests and can use agents to implement them, but just the ability to generate code, even in the right direction, is not necessarily good engineering.

For example, I wanted the ability to send AI user interviews as a link rather than an email. Although I could have asked an agent to build it, Paul’s reply shows the knowledge I lacked for making a good implementation decision.

I wouldn't have known to accept distinct_id and session_id as query parameters, to throttle requests, or to look at hosted surveys for inspiration like Paul did. At best, my agents would have taken many more tokens to get to the same solution. At worst, they would have introduced bugs and attack vectors.

It’s this understanding of the problem area, implementation details, and structure that enables engineers to ask the right questions and follow up to get a solution shipped.

4. Evaluating the work of your agents

This is the point where engineers would have been writing the code. Now, they just tell the agent to do it and that’s it, right?

Sadly, not yet. Engineers still need to evaluate whether it was built right.

This is done through code reviews and asking questions like:

• Did the agent actually follow directions?

• How is the code quality of this change?

• Did the tests pass? If not, how do I make them pass?

With the volume of changes coming from agents, this quickly becomes a bottleneck. If you are generating 20 PRs per day, reading every line and running them by hand is unrealistic. You need new systems, like review and testing agents, to keep up while ensuring the code you ship is good.

For example, many of our engineers have their own custom-built code review and PR management setups. Tom on our warehouse sources team built Talyn as a “mission control” for his PRs, our DevEx team built speedy virtual machines to preview changes, and we’ve already written about Paul’s code review setup.

Systems like these are necessary when engineers are seeing a dramatic increase in the amount of pull requests they need to review. At PostHog, we went from 1,441 PRs merged in January to 4,869 in August while only growing engineering headcount 10%.

5. Improving the entire loop

This process doesn't just end. It loops repeatedly. What you build and ship leads to new observations like “are people using what we built?”

The loop isn’t just linear either. Stages feed back on each other:

• The information you find valuable while setting direction informs how you monitor the situation: what information you pay more attention to and what you ignore.

• How well a decision gets built informs the scope of what you’ll decide for agents to build next. Seeing an agent flail and make mistakes leads you to reduce the scope of its future tasks.

• The functionality and usage of what you built informs whether it was the right direction, giving you experience and updating your knowledge on agent capabilities, technical feasibility, and customer demand.

It’s not just one big loop, but many smaller loops as well. The consequence of this is engineers doing more work on the system that builds the product than the product itself. They build the software factory rather than the software.

The growing capability of agents means there are cases where you can skip much of the loop too. Agents often only really need direction and the right tools to ship a valuable fix. With these, they can then figure out what to build, how to build it, and evaluate whether it actually worked. Here’s a real example from PostHog:

• Direction is encoded into a prompt for a Scout, a long-running agent that makes sense of all your data. In this case, we have one that aims to improve our MCP server, an important surface for our ICP of AI-pilled developers.

• Based on observations from mcp feedback submitted events and mcp_tool_call error rates, the Scout finds agents want to define catalog events before they have been ingested and use an ugly workaround to do so. Related observations are grouped and become a report.

• This report triggers an agent to write a fix, which is reviewed by another agent before escalating for human review. It wasn’t auto-approved because the review found 2 “must fix” issues.

• After the fix ships, another agent evaluates whether issues are actually solved. It does this by keeping notes in its scratchpad about what was reported, what shipped, and validates that the fix holds before considering a report closed.

In this case, the monitoring, deciding, and evaluation were all handled by the agent, the direction to improve the MCP server was implicitly guiding it, and our product improved without us needing to do anything. We call this making your product self-driving.

But even this case is reliant on your skills and knowledge as an engineer. Writing code might be cheap, but your system for building a product isn’t. All your sources of information, your way of setting direction, your implementation knowledge, the way you evaluate what was built, and the connections between these stages becomes what makes you and your product unique.

This is all still engineering work. It just doesn’t look like it once did.

Words by Ian Vanagas, OODA loop respecter. Graphics by Lottie and Daniel.

🧑‍💻 We’re hiring too

• Backend Engineer – Ingestion (Europe/UK)

• Technical Staff Writer (Blogs & Newsletters)

• Business Development Lead (Americas)

• Technical Customer Success Manager (Americas)

📖 More good reads

• From 1 to 100 IRL events in a year: The secret to getting engineers to demo – Daniel Zaltsman

• Multimodal models need video. We froze time to give it to them – Tue Haulund

• How to turn your data into content with PostHog – Natalia Amorim

• Boyd’s OODA Loop (It’s Not What You Think) – Chet Richards

• Why AI hasn’t replaced software engineers, and won’t – and

But then the models kept getting better. Now, the same context your agents couldn’t function without can make them perform worse.

Our AI onboarding wizard, for example, often landed on the wrong project in monorepos because our scripts pointed to root by default. Now that models are good at inferring repo structure, we updated it so headless runs take advantage of model recommendations.

This is the same reason why Anthropic removed 80% of Claude Code’s system prompt. Labs are going back on previous advice about rules and repetition; judgment, interfaces, and progressive disclosure are the new best practices instead.

The goal of context engineering has since shifted toward subtracting as much information as you can to get out of the models’ way:

But figuring out exactly what to subtract isn’t easy when dealing with unpredictable behavior.

When asked about his approach, Boris Cherny, the creator of Claude Code, recommended deleting your CLAUDE.md every six months to stay on the bleeding edge. Theo recently reported it was worth rewriting his AGENTS.md by hand:

If you'd rather not start from scratch every time, here are three practical ways to maintain your context while "unhobbling" AI, from most hands-on to most autonomous.

1. Run /doctor, but follow up with real eyes

The simplest way to keep your context healthy is to read and edit it yourself.

Most coding agents have a built-in /doctor command to help with this. Anthropic’s version of it runs basic health checks, deletes redundant prompts, catches broken settings, finds unused plugins, and optimizes for lazy loading.

The report also prints a summary of how often you’ve used each file, and how much it can trim. Here’s what that looks like for our website’s main AGENTS.md:

 Component: posthog.com/AGENTS.md (= CLAUDE.md symlink)
  Type: memory file
  Scope: project (checked in)
  Uses (total since install): always loaded
  Used in window?: yes
  Est. resident tokens: ~1,780
  Verdict: trim ~350

Across the posthog.com repo, it suggested turning off 3 unused plugins and 3 skills across our website repo and estimated saving an average of 6K tokens per session.

This is decent, but you can’t rely on it to catch everything because it doesn’t check for correctness; it acts only based on what it can derive from your code.

For example, last month, we started using merge queues and added this line to our AGENTS.md:

A couple days later, we paused the queue to fix some failing tests and forgot to update that line. Our agents therefore had wrong instructions for 21 hours. During that time, one engineer’s PR got stuck for 10 hours; another lost 45 minutes investigating the cause before updating the instructions.

We didn’t use claude doctor in the posthog monorepo to fix this, but it couldn’t have caught it anyway since merge queue state lives in a GitHub setting rather than in code. Agents don’t know what they don’t know.

Try this: Run claude doctor after each upgrade and follow up with a manual pass on your AGENTS.md. For each line, if you can’t name the failure it prevents, delete it.

2. Test your context like it’s code

Another tool for maintaining and unhobbling your context is evals.

It’s just like adding regression tests whenever you fix a bug in code. Every time you update your context to address an agent’s mistake, capture what caused it in the first place as an eval.

For example, we record technical “gotchas” we see from real PostHog Wizard runs as framework-specific commandments such as:

• For versions 15.3+, initialize PostHog in instrumentation-client.ts for the simplest setup.

• For Phoenix or Plug apps, add PostHog.Integrations.Plug before the router so request context is attached to captured events and errors

• posthog-rs is the Rust SDK crate; add it with cargo add posthog-rs and construct the client with posthog_rs::client(options).await.

These get combined with instructions from the context-mill (e.g., “reuse event names the project already uses”) and a set of ~40 clean sample apps to create the evals we use in the wizard-ci. Here’s how it works:

1. wizard-ci runs the PostHog Wizard on all ~40 sample apps and creates one PR for each.

2. Those PRs don’t get merged. Instead, a second agent called the pr-evaluator grades each of them based on the diffs and session logs.

3. The pr-evaluator leaves metrics and reports on the trigger PR and the wizard-ci PR.

These reports help us catch issues like the wizard skipping installing PostHog entirely because the model determined it was already complete:

wizard-ci is just one of many context-as-code tools we’ve built in our wizard-workbench. The setup is complex since it maintains our biggest conversion path, but the core concept is easy to apply: test your context by saving prompts that check if your agents are doing what you want them to do.

Try this: The next time your agent makes a mistake, paste the prompt that caused it into a failures.md. Re-run those prompts the next time you edit or delete parts of your AGENTS.md as a quick test suite for your highest-cost piece of context.

3. Ask agents for feedback directly

Another option for fixing your context is to just ask. Agents are good at providing feedback about how they would improve your context while they’re using it.

Many developers already do this to update their skills, but you can take it further by putting it in your prompts by default.

For example, the final instruction for the PostHog Wizard says to send back a quick remark about any errors they encountered in production:

This gives us a rich feed of live bug reports for cheap – a basic form of AI observability:

We took this even further by feeding them back into our context-mill with a loop, making it self-driving. But since we can't trust agents at face value, we cluster and verify the underlying issues first. Here's what the feedback clusters looked like for the last month:

Half of the other category is just confirmation messages like “succeeded on the first attempt,” or “posthog-js already installed”. The other half is a long tail of issues that weren’t common enough to cluster.

Once the loop identifies a meaningful cluster, it deploys subagents to verify the issue before attempting a fix. For example, to check the notebook create schema/tool issue, a subagent followed the legacy instructions to reproduce the hand-splicing text failure that was addressed in this PR.

This approach works best at scale to cluster remarks, but you can still apply the concept to your workflows by simply asking agents to log what context files they used.

We do this in our posthog monorepo PR template, which prompts agents to name any skills invoked. We’ve seen other agents use this information to catch and fix skill inconsistencies, like in this PR where Claude found an issue while unblocking stalled ClickHouse cleanup PRs.

Try this: Agents face issues all the time. Ask them to provide feedback about your context with a structured prompt about any errors, inconsistencies, or failures they ran into. Over time, you can turn verified reports into a self-driving context system.

Written by Jina Yoon, who would like a word with whoever coined the term “unhobbling.”

🧠 Optimize your HUMANS.md

• A field guide to Claude Fable 5: Finding your unknowns – Thariq Shihipar

• Our context warehouse knows your data, the semantic layer tells it what to trust – Thiago Rocha Salvatore, Lizzie Epton

• The most common questions about developer marketing, answered – Joe Martin

• Nobody watches their session replays: A study of 7.7 million views – Natalia Amorim

• Treat prompt changes like code deploys – Luke Murray

🦔 Join the PostHog crew

• Backend Engineer – Ingestion (Europe/UK)

• Technical Content Writer

• Business Development Lead (Americas)

• Technical Customer Success Manager (Americas)

It came from a simple reality no company can escape: the product-market fit game never really ends. Whether it’s frontier labs attacking your territory, or the hundreds of scrappy two-person teams determined to build “[Your App] for 2026”, now is the time disrupt yourself.

I’ve had the job of marketing our in-house disruption through every shape it’s taken. It started as PostHog Code, an agentic coding tool and product editor. Then the most valuable thing that grew inside it, self-driving, got de-bundled and shipped into PostHog Web.

The container it left behind became PostHog Desktop: a different product shape entirely, built around multiplayer spaces, generative UI artifacts, and the thesis that shared business context is key to a product that builds itself.

This isn’t advice from someone who’s run the PMF playbook and won – PostHog Desktop is now in open beta and doesn’t have product-market fit (yet). It’s mid-game observations that might help if you’re post product-market fit, and pre-whatever comes next.

1. Write down how you die

Disrupting yourself is about playing a good offense, but you still need to understand what you’re defending against. Write down the threats as three or four one-liners: who or what makes your product unnecessary? (we’ll counterattack later, so just speculate the future for now).

For example, Vercel built its agent tooling by running its own agents in production and shipping the gaps as products: AI Gateway, Chat SDK, and the Agent Stack. Your cause of “death” doesn’t have to be a villain. AI models getting so good that everyone can ship software is arguably utopia, but you don’t need me to tell you how disruptive that’s been.

2. Fund the attacker yourself

Give the “death-page” to a small team with big ideas – plus a blank slate to build something where your existing users won’t trip over it. 3-5 people, reporting to a founder rather than to the department they’re disrupting. You get to be the incumbent and the seed investor at once! (startups attacking you would kill for this setup).

3. Score it like a seed bet, not a business unit

In all likelihood, the new thing you build will have lower margins, worse retention, and more pivots than you expect. Don’t compare the metrics of your pre-PMF bet to those of your current PMF product (not yet, at least).

Write down, in advance, what the experiment will be judged on, the resources it’s allocated, and the runway it has before you even think about killing it. This is your defense against pulling the plug too early, and subsidized infinity.

4. Look for customers doing cool things with your product

With the PostHog MCP, you never have to open the web app at all – and that’s fine by us. Whether people reach PostHog through the MCP, the Slack app, Desktop, or the CLI, our job stays the same: give them tools they need to build successful products.

One of our largest customers took us up on that in a way we didn’t expect. They wired our MCP to Claude (plus a few other tools), and built their own working prototype of PostHog 2.0. This is the yellow flag every founder should watch for: when your users start assembling the next version of your product themselves, the demand is proven.

5. Build first, validate later

Your users are a great source of insight about how to improve your existing product. A good product team is obsessed with pleasing them, and every signal they send you compounds to growth.

The opposite is true when taking a bet on something new. Ask this amplification machine whether you should build a strange new thing for a strange new market (maybe even a whole new category) and it’ll probably say: “uh… no? Why risk what’s working?”

You can win these people round, but only after you’ve built something real, so don’t get hung up on validating your idea with them beforehand. We launched PostHog Code as a closed beta and validated it with real users, which led us to integrating its core ideas into PostHog Web.

6. Ask the disappointment question early and often

Once you have a prototype, dogfood it internally. The classic Sean Ellis PMF survey: “how disappointed would you be if this disappeared?” works on your own team (especially if they’re your ICP).

We did this before a recent product launch and scored 23%. The industry PMF benchmark is ~40%, so obviously this was painful. Your team will tell you the truth before your churn does, but only if you ask them.

7. Let usage overrule everyone, including you

Building a product runs on a mix of data (user interviews, session replays, product analytics) and, honestly, some plain vibes. Vibes are fine for deciding what to try – not what to keep on building. We set up a Replay Vision agent that watches session recordings of PostHog Desktop users and posts a daily summary in Slack. It contradicts our vibes constantly.

One thing to keep in mind: people who love your current product will land in the new version carrying assumptions. They’ll judge it against the workflow they already have, and read every change as something taken away. Net-new users have no map, and will show you what the new thing actually invites people to do. Both signals matter, but don’t try to average them.

8. Build the new thing on the asset your attackers can’t copy

Every incumbent has something a newcomer can’t copy – usually data, distribution, or both. Find a way to give your pre-PMF team that family advantage without the generational baggage.

In our case, Desktop shares almost no interface code with PostHog Web. What it shares is the data, including the same MCP we hand to customers. And because it’s a new app, long-time users keep the interface that works for them, while new users get to choose their own PostHog adventure.

GitHub did the same with Copilot: a brand-new product built on the two things only it and Microsoft had: the world’s largest pile of code, and distribution to millions of developers.

9. Ship before you’re ready

When you hop back on the PMF treadmill, it will feel bad in very specific ways. The new thing will lose in almost every comparison your company knows how to make. You’ll change the name too many times. You’ll kill features you were proud of two weeks after shipping them. And your own people will tell you they don’t “get” what you’re doing.

If none of that is happening, you’re probably not disrupting yourself.

But you’ve done this once before, so you’re better armed than any first-timer. You have revenue funding the experiment, real users telling you the truth, years of data to build on – and now agents doing half the building.

PostHog Desktop is our version of this, and it’s out now for you to try (no waitlist). Come use it in ways we never imagined, and tell us what you’d fight to keep. If it all goes sideways, we’ll write the post-mortem post too (those are usually pretty popular).

Words by Cleo Lant, who thinks a sandwich cut into triangles tastes better.

💼 Join us on our journey to PMF (again)

• Technical Customer Success Manager

• Technical Account Executive (SF/U.S. West Coast)

• Technical Account Executive (EMEA)

• BizOps Manager (Sales Focused)

• Backend Engineer – Ingestion

Fable limits made the average developer increasingly aware of their spend, and giants like Facebook, Microsoft, and Shopify turned away and curbed the practice.

Now everyone is (rightly) obsessed with tokenminning: optimizing token spend and limits for fun and profit. This post is your guide to squeezing the most out of your dollars and context.

1. Learn how to track token spend

The first step in tokenminning is knowing how much you spend. If we didn’t track this, we never would have known:

• A product’s LLM costs doubling from $5k to $10k in a day wasn’t a regression. A launch caused a workflow to run 4x more, but the cost per run actually fell from $1.8 to $1.4.

• The Claude Code SDK in PostHog Desktop uses Haiku extensively (3-5 calls per Opus call) revealing an opportunity for us to swap it with a cheaper, self-hosted model in the future.

• RTK wasn’t saving tokens on bash commands as expected because it wasn’t being used properly by Claude. A fix cut 13% in bash token usage and a PR is open to ship it to everyone.

To catch and optimize spend like this, you need to monitor:

1. How much you use on flat-rate subscriptions like Claude Code which are constrained by context windows and rate limits, rather than costs. The /usage command is the big one here, but so are ccusage, quota widgets like ccseva, and monitoring of LLM spend through gateways.

2. How much your team uses automations, Slack apps, and scheduled agents. Many have monitoring built-in, but you need to consolidate them to prevent blind spots.

3. How much customers use AI-powered features. Being efficient here impacts what products you can build, how much you can charge for them, and how much you can grow. AI Observability is a big help.

Teams almost always have a gap in at least one of these places. This limits the details you have about what workflows, features, and use cases are using the most tokens. Your monthly bill from Anthropic won’t tell you this.

However you find your baseline costs, it needs to go hand-in-hand with tracking accuracy, success rates, and usage because it defeats the purpose of tokenminning if the changes make your workflows or products worse. Evals and product analytics are critical to monitoring this.

Tracking quality also reveals failures like bad queries, repeated retries, and agent runs that produce nothing. Fixing errors like these prevents token spend that never should have been spent in the first place.

2. Optimize your AGENTS.md

Your AGENTS.md is like a backpack you carry wherever you go. Its goal is to save your agent tokens when they read it. To best do that, it should include:

1. Discovery shortcuts. Facts an agent will find eventually, expensively. For example, details on our bespoke, unguessable hogli CLI for testing, linting, and building.

2. Undiscoverable intent. Direction of travel, taste, and policy. Like preferring SeaweedFS over MinIO (because the latter is being removed) or that we ban hooks.

3. Landmines. Things agents get confidently wrong and only learn from CI or prod like workflow backward compatibility breaking CI repeatedly or dispatch caps on stacked PRs.

It shouldn't include:

• Lint-enforced content, like using camelCase for TypeScript or American English

• Obvious model-defaults, like following existing patterns in the codebase and writing tests

• One ls or grep answers, like directory trees, the list of packages, 200 line reference components

• Docs meant for humans, like mission statements or contributing etiquette

An AGENTS.md file is definitely worth having though. One study showed “the presence of AGENTS.md is associated with a lower median runtime (Δ 28.64%) and reduced output token consumption (Δ 16.58%), while maintaining a comparable task completion behavior.” That’s tokenminning if I ever heard it.

3. Audit your MCP servers

Although MCP tool definitions and schemas don't load into the system prompt anymore, fetching one can still blow up a session's context window. For example:

• The official Atlassian MCP server consumes roughly 10k tokens for Jira and Confluence tools alone.

• The official GitHub MCP server exposes 94 tools and consumes roughly 17.6k tokens.

• Cloudflare’s native MCP with full schemas include 2,594 tools with a cost of 1,170,523 tokens (but Code Mode fits it in 1,069).

• With every tool exposed, PostHog’s MCP server used to have 183 tools and 113,843 tokens (it doesn’t anymore).

For developers, you need to know what MCP servers are costing you. A CLI like the gh for GitHub or posthog-cli can often be more efficient and there are tools like RTK that further optimize CLI commands.

For MCP server providers, you cannot just expose every endpoint as a tool. Tool descriptions can occupy a huge amount of the context window. Platforms like Cursor and OpenAI limit tool count anyways.

Let us be a lesson. As a company with a formerly hefty MCP server, here’s what we’ve done to optimize ours:

• A single exec tool. This brought our MCP server down from 113,843 tokens to 5,000.

• Progressive schema exploration. Tools like query-trends, dashboards, and execute-sql had input schemas that exceeded the 16k token limits individually, so we summarize them instead.

• Connect time scoping. mcp.posthog.com/mcp?features=flags,dashboards mounts two features instead of forty. ?tools=dashboard-get pins one tool. Most users only use a handful of our tools, our MCP server can match this behavior.

4. Learn what caching is and why it matters

Vincent on our Wizard & Docs team once audited our AI install wizard and found it embezzling tokens. The trivial conclude step cost $1.47 and he thought it was the agent holding onto context up until that point. A fresh query() call cut accumulated input 89%. You’d expect this to cut costs, right?

Wrong. Every new call has to rebuild the whole cache, which costs more. Cache writes cost much more than cache reads, so clearing context only pays off if you save 12x tokens for every one it forces you to rewrite. Cached tokens are 10x cheaper in dollars per token than regular ones, and sometimes even faster too.

This is why understanding caching is so important. ngrok and earendil have great guides if you want all the details, but the most relevant parts in practice are:

1. Caches are per-model, so beware of fallbacks. If a conversation falls back to another model in a session, it will instantly double charge you.

2. The cache is keyed on the literal bytes of your prompt in render order: tools, system, messages. Put stable tokens first and volatile content last. Anything that changes per request belongs after the last cache breakpoint, not interpolated into the system prompt.

3. Subagents are great for context isolation, but don’t share its parent’s prefix, so every one you spawn pays the full cache-write price. The parent’s cache can also run out waiting for children to return.

4. Anthropic cached writes cost 1.25x normal input on the 5-minute cache and 2x on the hour. Reads cost 0.1x. This means you need one read to earn back the 5-minute write, but the hour cache write needs two reads.

5. Check cache_read_input_tokens in the response. If it’s zero across requests that should share a prefix, something is silently invalidating.

5. Choose the best model for the job

First, do you need a model at all? The most token-efficient call is the one you don't make. Linting, regex, a SQL query, or a script are free, fast, and can't hallucinate.

If you do, start by understanding the model landscape: which ones are available and what do they excel at? This often isn’t obvious.

For example, on paper, Sonnet is cheaper than Opus, but in a test on our self-driving capabilities, we found that Sonnet 5 was 20% cheaper than Opus 4.5, not 2.5x as advertised, because Sonnet spends 1.7x turns/tokens.

We used Sonnet 4.5 on early agents not because it was the smartest but because it was the best blend of quality, speed, and cost. The blend is still the primary way we evaluate models, iterating by giving a variety of models the latest data and using LLM-as-judge to pick what model makes sense.

Typically this means:

• Cheap models for one-shot jobs with simple outputs. We use claude-haiku-4-5 for titles and Slack app classifiers, gpt-4.1-mini for MCP intent generation and survey summaries.

• Frontier models where the reasoning is the product. claude-opus-4-8 is the default for PostHog Desktop and query materialization fixes. gpt-5.6-luna classifies Slack app tasks and reasons about which model to use for them.

• Gemini where it’s structurally the right tool, not just cheaper. We use gemini-3-flash-preview for Replay Vision because it actually watches the video (others don’t) and gemini-2.5-flash-lite-preview for summarizing huge volumes of LLM traces because it needs a cheap model with a huge context window.

A router can provide a gateway to many models, which is helpful for testing and optimizing. Using multiple models, and routing to different ones per request, is the current meta with some companies saving 60-75% on workloads.

The final boss of “choosing the best model for the job” is training your own. We’re doing this for session replays as it’s a specific format where even a basic custom model can better give signal to whether a replay is interesting or not, for cheaper.

6. Separate working memory from long-term memory

Agents don’t need to remember everything you’ve ever done, like what you had for breakfast this morning.

Even though Claude and ChatGPT have auto memory features, it is still useful to build a mental model of how and when agents make memories. This way you aren’t stuck with wasteful memories and you can context engineer a token-saving memory system when using models directly if you need.

There are two types of memories:

1. Working memory: Session-limited, in-context, compressible.

2. Long-term memory: Persisted, project-level, selectively retrieved.

All memories start in working memory. It is only with some rule that they get promoted to long-term. For us, that is: Is it actionable for a future run?

Both the “future” and “actionable” parts of this rule are relevant. We follow these in our Scout scratchpad by creating memories that are “dated, names the entity ID, gives a clear conditional, bounded by a precise time anchor.” An example looks like this:

The key dedupe:error_tracking:19f… is structured so when agents are researching another error, they only need a single text search to find error_tracking related memories. The content then gives instructions on what to do if it’s the same row-tracking Redis failure again: skip authoring a new report.

A bad memory might have a key of note-1 and content of all the context. This requires the agent to scan much more content to find the relevant memory and then interpret the content to decide if it’s actionable.

7. Know when not to tokenmin

There is a lot of stupid advice about tokenminning, like spending your time optimizing over single words. I've seen influencers say you shouldn't say "please" or "thank you" because those are wasted tokens.

Optimizing a prompt you use a handful of times isn’t worth it. Neither is a workflow that doesn’t use many tokens. Our AI installation wizard is an example of the shape that’s worth optimizing for. It runs hundreds of times per day and costs ~$7 each time. Look at per-run cost to find waste and run count to decide if it’s worth changing.

Don’t optimize something whose shape isn’t settled. Prototypes get rewritten. Tuning caching and model selection on an in-progress workflow just means doing it again later.

If you’re an individual developer on a flat-rate plan, dollars are probably not your constraint. Focus on your context window and what’s loaded every session (AGENTS.md, MCP), not shaving prompts.

Done right, tokenminning helps you tokenmaxx on the right things.

Words by Ian Vanagas, who will still use his tokens to say "thank you" no matter what the haters say.

💼 Jobs at PostHog

• Technical Customer Success Manager

• Technical Account Executive (Americas)

• Business Development Lead (Americas)

• BizOps Manager (Sales Focused)

• Backend Engineer – Ingestion

📚 Good reads worth spending time with

• I bribe customers to talk to me - Jake Sciotto

• How to understand AI agent performance in your product - Will Wearing

• The Tokenminning Manifesto - Rob May

• AI usage patterns in software teams - Tim Qi

• The Shape of Things to Come - Steve Yegge

To keep from becoming the bottleneck, engineers everywhere have been building software factories: automated systems and guardrails that let agents ship and test on their own.

But the hype came to a sudden halt when HumanLayer CEO Dex Horthy went viral with a three-part series on Why Software Factories Fail last month. He argues that no amount of harness engineering can ever make them work at scale.

So, will we always be stuck reading the code, or can software factories actually work?

What is a software factory?

The idea is simple. With the right pipeline in place, you can automate most (if not all) of software development, much like an industrial assembly line.

It involves several loops scaffolded with tools like version control, CI/CD, and monitoring that make it easy for anyone – human or agent – to ship code in a standardized, repeatable process.

Current software factories that incorporate agents look something like this:

1. Humans decide what to build and put it into a ticketing system.

2. Agents pull tickets, build and test PRs that satisfy the requirements, then submit them for CI and review.

3. If the PR is approved, the code is deployed to production via CD.

4. If it’s rejected, it goes back to an agent for further iteration until it gets approved.

5. Once the code is in production, users engage with it and emit data, feedback, and signals that inform what gets built next.

Since it’s just a metaphor, there’s no actual criteria for what counts as a software factory. It’s easier to think about it as a spectrum based on how much of the code is produced by agents, and how closely humans still read the code:

Ramp, Cursor, and Uber land near the middle today. They’ve posted about using autonomous background agents without explicitly calling them software factories. PostHog is just a little further than that with agents writing about 70% of our PRs, but humans still skim at least 80% of those.

The top right corner is where it gets controversial with “lights-off” software factories, or “dark” software factories. In these, agents write all of the code while humans read none. This requires:

• A high degree of AI adoption in a team. Just having access to coding agents doesn’t mean anything if no one is using them.

• A system where agent output gets validated to the point of mergeability, such as with peer reviews, local testing, or browser clickthroughs; writing unit tests or reasoning about their own code is not enough.

• Guardrails that assume failure, such as sandboxes, permissions, and deterministic checks. Humans only act as decision makers on high risk PRs.

Only a few companies like StrongDM have publicly written about going fully lights-off, and many developers are skeptical about the approach.

Why is everyone talking about them now?

As humans shifted from writing code to orchestrating agents that write code for them, code review has become the new bottleneck.

At the same time, models keep getting better, so people are building loops, systems, and tools where agents review and/or merge PRs instead of humans – a.k.a., software factories.

But the discussion really heated up when Dex Horthy went viral with his three-part piece on Why Software Factories Fail a few weeks ago.

He argues that no amount of harness engineering makes lights-off software factories viable because the problem lies in the models.

He specifically highlights how coding agents are rewarded for one-off code correctness during RL training:

This happens because there simply aren’t many benchmarks that can detect quality and maintainability in the first place. The paper behind one recently published benchmark, SlopCodeBench, describes the challenge well:

In other words, whether a design is good or not depends on what an agent will get asked to build next. That rubric is unique to every codebase and impossible to know in advance.

So, until there are better benchmarks in place, Dex’s recommendation is to either keep reading the code, or front-load human effort in the planning phase.

What software factories are missing

The problem with most software factories today is that they enforce a clean separation between “deciding what to build” and “building it.”

This is evident in every blog about software factories we’ve seen so far. The pipeline always begins with a step where a human does all the planning and decision-making first, stuffs the requirements in a ticket, then hands it off to an agent with zero prior context on the problem.

Dex’s argument follows this pattern, too. In part 3, he says a better oracle for code quality would be whether the output “passes all verifiers for an incrementally-divulged spec.” Again, it’s assumed that the agent that writes the code must receive a spec written by a human.

In the real world, that’s not how good product engineering works.

Every engineer knows that to design a system well, you need both the technical constraints as well as a deep understanding of the problem. This is why product engineers need to talk to users, measure product-market fit, and validate product ideas. It’s also why our engineers decide what to build, not PMs.

The same applies to coding agents. An agent makes better architectural choices if they have access to relevant information, such as how your product is currently being used, your ideal customer profile, what users are complaining about, or what else is going on in your company:

Looking at signals from prod is especially valuable for enabling code maintainability since most software changes happen in response to actual usage. At PostHog, for example, 63% of changed lines are in files that already exist, and fix is our largest commit type at 40%.

And if you think about the last time you addressed an incident in prod or a piece of user feedback, you probably relied more on logs and traces to figure out a fix instead of trusting someone else’s secondhand account of the problem.

These are the kinds of problems we’re working on at PostHog to make self-driving products a reality: giving agents the same context that human engineers would use to make better design decisions.

Once you have that, turning off the lights might not be so crazy after all.

Words by Jina Yoon, who still can’t believe we got software factories before GTA 6.

📕 Long-term quality reads

• Are agents really killing UI? – Cleo Lant

• Own the Outer Loop – Addy Osmani

• The Session You Cannot Take With You – EARENDIL

• Stateless MCP has recaptured my interest – Simon Willison

• We spend $45,000 on doing more weird every month – Charles Cook

🏭 Join our production line

• Technical Account Executive (Americas)

• Business Development Lead (Americas)

• BizOps Manager (Sales Focused)

• Backend Engineer – Ingestion

• Site Reliability Engineer – ClickHouse

They’re essential because agents have amnesia. Every conversation, they rediscover your codebase and workflows from scratch. They pick the wrong tool, fall for the same gotchas, and repeat old mistakes. They figure it out eventually, but only after wasting time and tokens.

This is irritating for engineers who hate repeating themselves. That’s why they automate, write abstractions, and keep handy scripts around. Skills are this instinct applied to agents. Write them once and never explain to agents again.

Here’s everything we’ve learned about writing skills that actually work, for both you and your product.

1. You need to master progressive disclosure

Agents can learn entire programming languages, figure out how to use nearly any tool you throw at them, and improve themselves in a loop, but only if they have context to do it in.

Context is usually dumped in all at once at the beginning, which limits the amount agents have to work with. Skills flip this. Instead of loading everything up front, they disclose information progressively. The skill’s information is only loaded when the agent decides it’s relevant, whether that is the main SKILL.md file or references and scripts within it. This means good skills act as a router.

First, the agent needs to choose to load the skill. The skill’s name and description should describe when to reach for it, not what the skill is, because they’re only part always in context.

Here’s our SQL skill as an example:

---
name: querying-posthog-data
description: ‘Required reading before writing any HogQL/SQL or calling execute-sql against PostHog. Use whenever the user wants to search, find, or do complex aggregations PostHog entities (insights, dashboards, cohorts, feature flags, experiments, surveys, hog flows, data warehouse, persons, etc.) and query analytics data (trends, funnels, retention, lifecycle, paths, stickiness, web analytics, error tracking, logs, sessions, LLM traces). Covers HogQL syntax differences from ClickHouse SQL, system table schemas (system.*), available functions, query examples, and the schema-discovery workflow.’
---

• The name and first sentence define when to read the skill.

• The middle sentence defines when to use it.

• The last sentence defines what the skill does.

Having too many skills or too large descriptions causes problems. We learned this the hard way when building (too many) skills for PostHog AI. The agent’s effectiveness declined as skill descriptions filled the agent’s context window. Databricks also found that agents increasingly pick the wrong skill as more are added.

Sections within the skill also need to work as routers. The rest of our SQL skill is a thin workflow that links to 26 schema files, 22 example query patterns, and a function index. This enables further progressive disclosure.

If done correctly, skills help agents succeed faster with fewer tokens, but only because the context isn’t loaded all at once.

Try this: Read your skills through the lens of progressive disclosure. Do the names and descriptions help them get discovered? Are they split in a way that agents can only load what they need?

2. Skills aren’t just code

Look at these two mini-skills:

1. Run git log --oneline -20.

2. If you see a commit with “fix” in it, check the diff.

3. If the diff touches auth.py, read lines 40–120.

4. If the function is over 50 lines, flag it.

And

Find recent changes that look risky and explain why. Start with the commit history.

They have the same spirit, but the first is over-specified. It breaks the moment the repo doesn’t match what the skill expects. The second survives because it embraces ambiguity, the agent can actually look at what’s in front of it.

This is how treating skills like code goes wrong. Skills are a superset of code. They can do everything code can (because agents can write code) as well as a lot of things it can’t (because they are smart). Agents can handle cases in your skills you can’t foresee, like errors or runtime context, but only if you let them do so.

This means skills should be precise about:

1. The goal. What does “done” look like? If you don’t say, the agent will decide for you. Give it examples of what good looks like or ways to self-verify. For example, the creating-an-endpoint skill asks the agent to “confirm by calling endpoint-run with a sample payload to verify the response shape.”

2. Constraints. The boundaries and guardrails that prevent disaster.

3. Context it can’t derive. Where your data lives, which tool to use, the schema it would otherwise need to guess.

But ambiguous about:

1. The steps. You don’t need to be prescriptive about what files to check. Tell it to understand how billing works and it will find them.

2. Failures. You can’t know everything that will go wrong. The agent can read the error and you’re paying it to figure it out.

3. Runtime specifics. What the data actually looks like. Line numbers, file lists, counts, versions, and outputs all change more often than you think.

Over-specification turns a skill into a workflow and strips the intelligence you are paying for. Leaving the path ambiguous gives agents the flexibility they need to do what you want them to do.

Try this: Make your skills less prescriptive. Give the agent the tools they need and the goal you want them to achieve then let them find a way to do it.

3. Skills rot (but you can prevent it)

Products, APIs, CLIs, MCPs, models, behaviors, and best practices all change. When these changes break skills, most people just make fixes ad hoc, but this wastes time and tokens whenever it happens.

You can be more proactive than that. Preventing rot comes down to three principles:

1. Split durable structure from volatile content. Hand-write parts that rarely change, reference parts that drift.

2. Point to a single source of truth. For us, that’s docs. Skills that point to specific URLs are less likely to go out of date than ones that include all the content separately.

3. Regenerate, don’t patch. Rewrite and refine your skill from a stable base rather than piling on fixes. Repeated fixes cloud a skill’s focus until it rots into something less effective.

These principles work at the individual level, but can also scale to a robust pipeline for skill generation. Our Wizard & Docs team do this to publish 120+ skills on how to integrate and use PostHog correctly.

The main part of this pipeline is the context mill (above) which consists of three parts:

1. Sourcing: Pulls all our docs plus curated prompts and working example apps.

2. Assembly: Transforms and packages these into a portable, self-contained zip manifest.

3. Delivery: Creates a versioned release that is used by both the PostHog MCP (as resources and slash commands) and the install wizard.

You don’t need a pipeline like this to prevent your skills from rotting. The three principles work by hand. Done right, a skill doesn’t just resist rot, it improves on its own. Point it at the docs and it tracks their changes. Regenerate it from a prompt and it inherits model and implementation upgrades.

Try this: Next time you need to do an ad hoc fix for a skill, think of and implement a way to prevent future rot like pointing to a URL or generating a part of the skill instead.

4. Asking questions > making demands

Agents, like people, work better when you understand them and what they need. So rather than jumping straight to making demands, try asking questions like:

1. “What can you do to help?” You need to know what tools and capabilities an agent has to help you accomplish your task. You should care about agent ergonomics and let it shape the skills you write.

2. “What do you need to do better?” Beyond what’s built-in, there might be other tools, capabilities, context access, or evaluations that might help it improve the skill.

3. “Based on the last run, how can this skill improve?” Runs often uncover errors or inefficiencies that can be fixed by tweaking the skill.

This works because the agent holds information you don’t: which tools it has, the context it can reach, and what broke on previous runs. Guessing that is how you burn tokens and time; asking is how you stop. As Charles said “The single most useful hour I spent with Cowork was asking it to help me figure out what to use Cowork for.”

Try this: Before building your next skill, build some empathy for the agent by asking the questions above. This is high leverage work, so use a more powerful model, like Fable, even if a less powerful model will run the skill.

5. Not everything deserves a skill

Reading all this might inspire you to write skills for everything, but remember there are context and maintenance costs for each one. Picking the right ones matters, so you should write skills for work that…

• You do repeatedly. Point your agent at your last 30 days of work. Your Slack, emails, code, writing. Have it identify manual workflows worth packaging. Kristopher Dunham suggests asking “Have you done it three times and will you do it three more times?”

• Agents do badly by default. Ask your agent to do the skill you’re planning to write. If it can accomplish it efficiently, it doesn’t need a skill. For example, it doesn’t really need a skill to write SQL, but it might need a skill to write ClickHouse-flavored SQL specific to PostHog.

• Needs context models don’t have out-of-the-box. For example, a bunch of data like AI observability and logs in PostHog have nested or non-obvious payload shapes. A skill ensures agents don’t need to rediscover this shape every conversation.

• Can run on autopilot. You can create loops to handle this. Add a goal, context, way to evaluate work, and put it on a schedule. Examples include PR babysitting, flaky test hunting, and performance improvement autoresearching. Our scouts are long-running agents guided by a skill.

Try this: Run this prompt to create some new skill candidates.

Look over my work from the last 30 days and identify repeated manual workflows worth packaging into skills.

Use available evidence in this order:
- Recent sessions and task summaries.
- Memories and outputs to find patterns repeated across sessions.
- Existing skills, custom agents, and automations, so you reuse or extend what already exists instead of duplicating it.

Look for work that is repeated, time-consuming, error-prone, context-heavy, or benefits from a consistent process. Include workflows across coding, research, writing, planning, communication, operations, analysis, and personal administration.

Only act on a candidate when it:
- Occurred at least twice, or is clearly likely to recur and costly to repeat
- Has stable inputs, a repeatable procedure, and a clear output or stopping condition
- Would materially improve speed, quality, consistency, or reliability
- Is not already adequately covered.

Produce a compact shortlist with:
- Repeated workflow
- Supporting evidence and dates
- Frequency/confidence
- Why it is or is not worth creating

Words by Ian Vanagas, who’s definitely skill-pilled.

🦔 Jobs at PostHog

• Technical Customer Success Manager – Americas

• Technical Customer Success Manager – EMEA

• BDR Team Lead

• Revenue Ops Manager (sales focused)

📚 More good reads

• 10,000 PRs a month is easy: How devex is evolving at PostHog – Paul D’Ambra

• Equipping agents for the real world with Agent Skills – Anthropic

• Micromanaging the Model Backfires – Tim Ariyeh

• How to get hired to PostHog’s marketing teams – Joe Martin

Some think the answer depends on how good the model is – i.e. when models get better, you can trust them to do more. But trusting your agents just because the models got smarter is like skipping your seatbelt because you got a nicer car.

The real answer has nothing to do with the model, and everything to do with the task. You need to build a mental model for when to delegate and how much, which you can then adapt.

Here’s a simple guide to maximizing agent autonomy, so you can ship fast without breaking things.

The two factors that set the ceiling

You need to ask yourself two questions first:

1. Is it easy to check the agent’s work?

To run safely on their own, agents need immediate feedback when they’re wrong. This is possible for most code with deterministic checks, like unit and integration tests. But subjective tasks, like renaming a parameter for clarity, are harder without human taste and judgment.

2. Is it cheap to undo an agent’s mistake?

Just like in traditional software engineering, if you want to trust an agent run without supervision, you need a guaranteed Ctrl+Z for worst case scenarios. That’s why StampHog, our PR approval agent, routes anything that contains deny-list keywords to a human.

Together, these two factors1 point you to one of four levels for any task:

• Level 0: Agent as assistant. For tasks that are hard to check and hard to undo. Necessary when dealing with tricky and sensitive code.

• Level 1: Human-in-the-loop. For tasks that are hard to check and easy to undo. Usually when you need subjective evaluation.

• Level 2: Agent delegation. For tasks that are easy to check and hard to undo. The default ceiling for most dev work today.

• Level 3: Self-driving mode. For tasks that are easy to check and easy to undo. Everything is accelerating towards this.

You can map these four levels to a very simple decision tree, which you can apply to any task:

This helps you make decisions about how much to delegate to your agents, but you still can engineer your pipeline to enable higher autonomy for any specific task.

Level 0: Agent as assistant

When the work is hard to check + costly to undo

This is the lowest level of agent autonomy. Think asking ChatGPT for advice, or auto-complete in Cursor, just like the good ol’ days of 2024.

But just because it’s old-fashioned doesn’t mean it’s bad; this mode is ideal when dealing with tricky problems in sensitive code surfaces.

For example, when Dylan updated our feature flag engine to support generic property targeting last year, he had to migrate an assumption that was indirectly baked into every feature flag at PostHog.

This would have been hard for agents to check deterministically since it wasn’t possible to grep for. The update also had a huge blast radius since it touched live customer flags, API response shapes, and reason-scoring functions.

How to level up from here

Break the task down. Small pieces make it obvious where delegation is safe or not. Dylan handed off less critical work, like propagating the new targeting logic across our JavaScript, PHP, Ruby, and Flutter SDKs, to agents while doing the riskier core migration by hand.

Level 1: Human-in-the-loop

When the work is hard to check + cheap to undo

This level of agent autonomy is common for tasks that need subjective evaluation since it’s hard to teach agents taste and judgment (for now, at least).

Human-in-the-loop tasks are considered cheap to undo since the code stays in draft mode and won’t get merged until it’s verified by a human. An undo just means kicking off another iteration.

This code readability refactor by Thomas was mostly done by hand, but it's a good example of something an agent wouldn't know how to grade. It was just a few lines that made it easier for humans to understand – adding comments, grouping actions, swapping strings for an enum – and didn't introduce any breaking changes.

How to level up from here

• Use LLM-as-judge. This is how most people are building their agentic code review systems. As models continue to improve, more tasks that require human judgment can be checked with LLMs.

• Define a scoped, measurable goal. Success metrics or contracts can work as a proxy for a subjective evaluation. For example, you could instruct a system to experiment with landing page copy until a variant reaches 3% conversion.

• Write custom skills. This helps agents produce work that reliably fits your standards, conventions, and taste with less steering required. For example, many developers write custom code review skills to follow team-specific standards.

Level 2: Agent delegation

When the work is easy to check + costly to undo

This is the level that most developer tasks are today. An agent writes code that can be tested deterministically, but the final act of merging it is gated behind a final safety check.

When Robbie rewrote our SQL parser in Rust from scratch, he barely read the code since he had a machine oracle to check the work. But since the parser touches every query at PostHog, the agent’s work was gated behind multiple safety checks: shadow mode in production, then a staged cutover.

How to level up from here

Enforce policies and guardrails with code. Most people default to gating Level 2 tasks behind a human (a.k.a., themselves), but this habit turns you into your own bottleneck. Instead, encode as many of your guardrails directly into your pipeline with policies like dry-running by default, scoping credentials, and putting changes behind feature flags.

Level 3: Self-driving mode

When the work is easy to check + cheap to undo

There aren’t very many tasks in this category yet – just smaller ones like dependency bumps, lint fixes, adding test coverage to existing code. But the category is growing quickly, especially with long-running agents, goal-driven loops, and more complex orchestration.

We’re going all-in at PostHog on making self-driving mode a reality for builders. Last month, for example, we launched Scouts – agents that run on a schedule, investigate signals from product data, and draft a PR based on what they find.

How to level up from here

• Train domain-specific models. Much of the next wave of tooling will be about improving verification tasks that are difficult for LLMs today. One obvious path is purpose-trained models that know what “good” looks like in a specific domain, which is why we’re training our own AI models at PostHog.

• Build expert-level context banks. A lack of agent autonomy is often just due to a context deficit. We learned firsthand from engineering our context layer for the PostHog Wizard that closing this gap with structured and fresh knowledge is one of the highest-leverage things you can build for reliable agents.

• Design clear signals for scouts. The bottleneck for long-running agents is going to be whether they know when there’s work worth doing, and how to distinguish valid signals from random noise.

Words by Jina Yoon, who writes these newsletters at Level 0.

🕵️ Have your agents talk to our agents

• Technical Customer Success Manager - Americas

• Technical Customer Success Manager - EMEA

• BDR Team Lead

• Revenue Ops Manager (sales focused)

🔺 More ways to level up

• Agentic Autonomy Levels – Addy Osmani

• Repositioning PostHog – Charles Cook

• How Cerebras Built Its Enterprise Knowledge Base – Isaac Tai, Daniel Kim, Mike Gao

• What is a context warehouse? – Lizzie Epton

• The new rules of context engineering for Claude 5 models – Thariq Shihipar

Many of the conversations and decisions about what we’re building now are impacted by this. More and more, we’re aiming towards building “2030-shaped software.”

This is a vague aspiration and “agent-first software” isn’t much better. But, while we can’t predict the future, we can share how we’re thinking about the problem.

Human and agent work is split

The current era of software is designed for humans to do all the work, but in 2030, agents will be the ones doing. Agents have shown that they can do the obvious stuff themselves. What does this leave humans to do?

1. Judging and deciding. Humans will approve, prioritize, choose direction, and resolve ambiguity.

2. Understanding and trusting. Humans will evaluate whether it worked, if it’s safe, and what’s changed.

“Doing” UI like buttons and forms are going away. They will be replaced with UI that supports the two types of work above. This helps optimize humans’ limited time and energy by pointing them where their input actually matters.

Product infrastructure is agent-first

If a human can do something with your product, agents should be able to do it too. This means complete APIs, an MCP server, tools, and permissions. Build for agents as a primary surface, not an afterthought.

Missing capabilities are a failure mode. Say you ask an agent to set up an A/B test. It creates the feature flag, builds the insight, wires up the events, and stops because you never exposed a tool to create the experiment. It now has to ask you to do that part which defeats the purpose of using agents.

This doesn’t mean that tools are all you need. It won’t be enough to just let Claude use your product on its own. Relying entirely on a model provider is too much risk. Building your own harness lets you guarantee quality, protect your brand, and make continuous improvements. Even OpenAI notes that agents struggle not because they are incapable, but because they lack “the tools, abstractions, and internal structure required to make progress toward high-level goals.”

Context is what makes your agent special. The model is the same for everyone, your moat is the source code, usage data, customer data, and product skills you feed it. This means context engineering, creating pipelines and flows to get fresh, valuable context to the agents when they need it.

Chat is the front door, generative UI inside

People like to complain about chat as an interface, but we think it's a good one. There's a reason why it's still the primary interface to work with agents and companies are doubling down with products like Slack apps – have you tried the PostHog Slack app yet? Telling an agent what's needed and letting it go do it is a powerful interaction method.

But chat isn’t enough because sometimes reading a paragraph is worse than glancing at a chart or flicking a toggle. Text can be too low bandwidth as it’s flexible but linear; a UI is legible but rigid. Generative UI is in between.

Most generative UI will be disposable, but some will become artifacts: outputs worth keeping, sharing, and forking. Examples include reports, documents, code, pull requests, configuration, and even entire apps. All of them are backed by your app's data. Generative UI is how you work with the agent right now; artifacts are what you walk away with.

Build UI when text isn’t enough

Text and generative UI cover much of what a product needs, but not everything. Some jobs will continue to need persistent, purpose-built UIs, like:

• Trust and verification surfaces confirming the agent did what you wanted. Checking work, not doing it.

• Triage and inbox surfaces listing decisions that need a human. Approvals, ambiguous calls, unclosed support tickets.

It's tempting to make your UI agent-first by removing the buttons, forms, and other parts of the UI that power the doing, but this is a half-measure. Most of this new "read-only" UI can be delivered as text by an agent, making it redundant and misaligned with what humans need the UI for.

Take our feature flags page. The jobs to be done are:

• Is my flag rolled out?

• To whom?

• Does it have a payload?

• What type is it?

An agent can answer all these questions in a sentence. You're not verifying the agent's work and there's no decision to be made. You're just checking the flag's state. A 2030 flags page shows how agents change rollouts and previews a pull request to clear up stale flags.

The question to ask is: is a human looking at this for trust and judgement, or to act? If it’s an action, give the agent the capability. If it’s to trust or judgement, build for that specifically.

Be everywhere

Agents don't work like apps. They run on their own, often for a long time. Nobody should have to sit in a browser tab watching them. 2030 products come to their users, no matter if that means desktop, Slack, mobile, email, API, or voice.

These are a shared control plane between humans and agents. Thin clients over a shared backend, so the experience is the same anywhere and the handoff between surfaces is seamless.

This pushes you towards cloud-first. The real bottleneck when working with agents is concurrency and babysitting. Builders want to run many agents at once, point them at a backlog, and refine the output from anywhere. On-device execution can’t do that.

Cloud-first only works if all that infrastructure is excellent. Sandboxes need to be rock solid: fast repo imaging, snappy execution, no crashing or freezing. Your app is ubiquitous, your infrastructure invisible.

Building 2030-shaped software now

2030 software requires a lot of changes, and 2026 is not 2030. As a company with multiple successful products and thousands of users relying on us, the hard part is getting there while having a real product and revenue to support. Do we retrofit or start fresh?

Retrofitting maximizes reach, that’s where our users already are, but we inherit all our 2026 patterns. Every change draws complaints from users attached to the way things work today (not to mention the tech debt).

Starting fresh means a clean break with years of cumulative UI built on assumptions that might not be valid or relevant. The people who opt in are early adopters who want to co-create the future with us. They don’t complain about what’s missing, they tell us what they need. The cost of this is convincing existing users to switch (eventually) and splitting focus with our existing products while we do.

We’re cheating a bit and going semi-fresh, using our existing data and infra on a new surface. We’re expanding our existing PostHog Desktop app to fit our 2030 vision first before migrating other surfaces. It’s already where the best agents live, it draws exactly the early adopters we want, and it’s simple enough to be a genuine blank slate.

Without this change, we risk building 2026-shaped software with AI bolted on. We answer “how do we add AI to this” rather than “what is the agent-first version of this?” We can’t predict 2030, but don’t need to be right about the details to build in the right direction. The companies that matter in 2030 will be the ones who started building for it well before they had to.

Words by Ian Vanagas who’s looking forward to 2030 software almost as much as a 2030 World Cup.

📅 2026 blog posts you should read now

• Our session replays are opening their own pull requests - Sara Miteva

• The Future of SaaS Is Agentic - Arjun Kashyap

• Riding Technology Waves - Stay SaaSy

• Own the Outer Loop - Addy Osmani

• Why AI hasn’t replaced software engineers, and won’t - Arvind Narayanan and Sayash Kapoor

💼 Join us on our journey

• AI Research Engineer

• Site Reliability Engineer (Pacific timezone)

• BDR Team Lead

• Context Engineer

• Backend Engineer

We accomplished our original goal

We started Product for Engineers to challenge the idea that engineers only care about code.

We showed that engineers can flex their product muscle, talk to users directly, be opinionated about what to work on, work autonomously to ship features, and figure out if they are successful. The tens of thousands of subscribers we amassed, the rise in popularity of the product engineer role, and 100+ YC startups hiring them now are proof we accomplished this.

Along the way, we learned product engineers care about more than just product. They care about building a successful company and career more broadly. After all, great products come from the people and companies who build them.

Following the advice to “build write something people want,” we wrote about everything from finding a job to scaling a startup to marketing to collaboration.

But every post like this pulled us away from the “product” framing in our name. Helping people build more successful products meant a lot more than just helping engineers do product.

AI changes engineering and the meaning of “engineer”

The other part of our name has also seen huge changes because of AI.

First, people are building AI-powered products and using AI to do so. Last week, PostHog saw 3.5M MCP tool calls, nearly 100K PostHog AI chats, and 40% of experiments created through our MCP. Our AI observability product grew 17x in the last year. Building the best products now requires building for agents, writing loops, and making full use of AI coding tools.

Second, AI has made it possible for anyone to build. The engineeringification of everything is real. Salespeople ship features, product managers fix bugs, marketers build video games. Our posts are relevant to a new group who don’t exclusively label themselves as engineers.

PostHog as a company is changing to reflect this. Beyond building more for AI, our ICP has changed from engineers at high growth startups to the people building products in AI-pilled software teams at any scale. Our newsletter should reflect this change too.

The Product for Engineers brand wasn’t resonating for us or our readers, so we’re changing it and going build mode.

What is build mode?

The tools, tactics, and taste builders need to build successful products, companies, and careers like:

1. How to build AI-native products (and agents) successfully

2. How to leverage AI to build more successful products and careers

3. How to build a successful company, culture, and brand

4. Lessons from the frontlines of doing all of the above at PostHog

build mode is a reflection of what's happening now and what we think matters to builders going forward. Our upcoming posts will cover what we think the future of software looks like, writing agent skills, and an explainer on effort levels.

It turns out that when code is cheap, the product skills we wrote about become even more important. AI enables more people to build and takes them further, faster, but it still needs the guidance, judgement, and taste only you can provide. Build mode aims to help you with this as best as we can.

Go build mode.

Words by Ian Vanagas who will miss our mascot Swolehog.

The naive solution would be for developers to review code faster. The 500 IQ take is for developers to review as little code as possible.

If you need to be involved in every code review, you will always be the bottleneck. Instead, put yourself outside of the code review loop by building a pipeline that delegates tasks to agents.

We asked engineers at PostHog how they’ve been reviewing AI-generated code to keep shipping fast without losing quality.

Here are four workflow changes you can steal (with prompts included) to make your life easier.

1. Make agents review code for you

The number one thing to add, if you haven’t yet, is a way for agents to review code for you.

The goal is to offload the simpler reviews to agents, and flag if something genuinely needs a human.

The key is that the agent that wrote the code can’t be the one that reviews it. Agents are bad at checking their own work since they’re often unaware of their own blind spots.1

For the same reason, it’s better to have multiple agents with different instructions and goals to cover more gaps,2 as well as different models and providers for different reviewers.3

Here’s how one of our engineers, Paul D’Ambra, makes his own custom agent review system work:

1. First, qa-swarm spawns four reviewer agents, each with their own special instructions:

   • qa-team – spawns technical subagents that hunt for security, database, performance, etc.

   • security-audit – probes for vulnerabilities like SQL or prompt injections

   • paul-reviewer – uses Paul’s voice and focuses on observability, rollouts, naming

   • xp-reviewer – applies an Extreme Programming lens to review

2. Then, review-triage sorts those reviews to classify threads into three categories:

   • actionable → gets fixed and pushed

   • nits → get resolved and replied to with a comment

   • ambiguous → escalated and sorted for Paul to work through with the agent later

3. An outer loop iterates up to three times or until no new actionable threads appear.

From there, you can connect this to another loop that shepherds the PR until it’s ready to merge – more on that in the next section.

The takeaway: Save time reviewing code by making agents review each other. This knocks out easier reviews so that only the PRs that really need human attention get flagged.

Steal this

You can check out and copy Paul’s qa-swarm and review-triage skills, or use this prompt to design your own review loop based on his:

Read Paul D'Ambra's qa-swarm skill, plus its sibling review-triage in the same folder, then help me design my own version: https://github.com/pauldambra/dotfiles/blob/main/ai/skills/qa-swarm/SKILL.md

It should take in a single PR, spawn a reviewer panel, triage every finding and existing PR thread into actionable / nit / ambiguous, and keep going until nothing's left but the ambiguous ones flagged for me.  

Interview me about my stack, tooling, available models, and how autonomous it should be — what gets auto-fixed vs. only reported, and what it may post to GitHub — before writing the final SKILL.md, then install it.

That said, these systems can get token expensive:

So if running multiple agents or loops isn’t an option for your team, look for single agent designs like this one by Kun Chen.

2. Delegate PR babysitting to loops

The context switching that comes with agentic coding is exhausting. One easy way to reduce that fatigue is by automating code review-adjacent tasks that don’t need your attention.

For example, babysitting a single PR can involve tedious tasks like monitoring CI, re-running flaky tests, checking notifications for comments, and keeping the branch up to date.

Why waste your most precious resource – your energy – when you can just delegate all of it to a loop?

The takeaway: Reduce context switching and fatigue by delegating simple tasks like PR babysitting to a loop.

Steal this

You can implement your own PR babysitter skill, based on babysit-prs by Phil Haack with the prompt below. (It works best if you run it after creating a review loop skill from the previous section.)

Read https://github.com/haacked/dotfiles/blob/main/ai/skills/babysit-prs/SKILL.md and adapt it for me: same sweep/state design, but it dispatches my own single-PR review skill via a spawned agent per unreviewed PR. 

Before writing SKILL.md, interview me on: which skill it dispatches and where my skills live, my stack/tooling/models, and which extra tasks to include — CI monitoring, branch freshness, flaky-test reruns, lint/format autofix, regenerating drifted artifacts, description sync. 

Ground the interview in  facts you can discover yourself (my open PRs, gh auth, clone layout) rather than asking about them.

3. Add a PR auto-stamper

Fast-moving teams generate a lot of small, low-risk PRs, and every one still needs approval on GitHub (a.k.a., a stamp).

At PostHog, we used to handle this in Slack where you drop your PR in #dev-stamp-exchange and wait for someone to give it a quick approval and react with a stamp emoji. We even built a leaderboard for it.

It worked, but each stamp required another engineer to take themselves out of their flow to approve a change they had little to no context on.

Now, most of those are done by our StampHog agent instead. And in just one quarter, it gives the final stamp on roughly 1 in 3 PRs merged into our main repo.

Our engineers add a stamphog label on their PR in GitHub, and it runs a few safety checks based on:

• PR state. No merge conflicts or changes requested

• Blast radius. Deny-list keywords (auth, secrets, billing, public APIs, etc.)

• Diff size. Under 500 lines and 20 files

• A simple LLM check. For basic showstoppers

If the agent approves, it’ll leave a bare GitHub approval with no line comments.

Otherwise, it refuses or escalates with a 1-2 sentence reason, risk level rating, and next steps. Usually that means routing to a subject matter expert based on CODEOWNERS-soft and git-blame familiarity.

We still use #dev-stamp-exchange when the agent can’t auto-accept or route, but it’s way less active now. Last month, the StampHog agent took care of 1.6K PRs on its own – that’s 1.6K fewer Slack interruptions for our engineers.

The takeaway: Let an agent can take care of low-context PR approvals and routing to reduce distractions. Use deterministic checks to route sensitive code to humans.

Steal this

The code for StampHog is available here. Many of its inner workings are specific to PostHog, so instead of copying it, here's a prompt to start customizing one for your repo based on ours:

Read https://github.com/PostHog/posthog/blob/master/tools/pr-approval-agent/README.md and build the equivalent for the repo at <path>. 

Copy the architecture; preserve its safety invariants exactly (fail closed, never request changes or merge, LLM can tighten gates but never loosen). 

Their deny-list and thresholds are calibrated to their codebase — re-derive for me: mine my git history for high-blast-radius deny candidates and calibrate size/tier ceilings from my merged PRs, then propose the full gate config for my sign-off before writing any code. 

At the same time, ask me whatever you can't derive from the repo — at minimum the CI system and trigger label, escalation routing if there's no CODEOWNERS, and which LLM/SDK to use and how CI gets its credentials. 

Leave the result as uncommitted files on my working tree.

4. Verify by observation, not reasoning

Agents are good at explaining why their code works. The explanation is often convincing... but also wrong.

If you run the code end to end, you’ll frequently find errors the agent never reasoned about, or output that’s just slightly not what you asked for.

That’s why Daniel Visca‘s rule of thumb is observability over reasoning. Don’t accept an argument that the code works when you can watch it work.

The gold standard is something you can observe directly, like sending a real API request and reading the response. If the behavior is in front of you, you don’t have to trust the agent’s rationale at all. But this has a scaling problem since a 3,000-line PR would be challenging to trust and observe.

His approach is to make agents decompose the work. For example, for a large change (like building a metrics pipeline end to end), he instructs the agent to produce a stack of small, single-purpose PRs and then uses Graphite for its “stacking” functionality. This makes each diff independently runnable and observable:

At each step of the stack, you can run a real check and confirm the output matches what you expect. Then, as you merge bottom-up, each layer only builds on behavior that’s already been verified.

This way, early mistakes can’t compound, and when something does break, you’re debugging one small diff instead of the whole change.

As a bonus, this lets StampHog from #3 auto-approve the small and focused PRs. You end up with two different checks: the agent reasoning about the code first, and a human observing its actual behavior.

The takeaway: When you can't trust an agent's reasoning, don't read more code; decompose the change until you can watch each piece run. Observation scales better than review.

Steal this

You can set this up by using Graphite to stack smaller PRs produced by your agents with these instructions:

Split work into a stack of small PRs, each under 400 changed lines and focused on a single change, building only on the PRs below it. 

Every PR must ship with its own tests and end with a way to observe it working directly — a command to run and the output I should expect.

This approach is especially valuable for frontend work since deterministic tests don’t always capture the visual or behavioral functionality you’re looking for.

You can use this prompt to copy the qa-frontend skill Pawel Cebula created for this. It’s a huge timesaver to have an agent run the code and take screenshots and GIFs for you:

Read https://github.com/PostHog/posthog/blob/master/.agents/skills/qa-frontend/SKILL.md and build the equivalent for the repo at <path>. Copy the architecture and preserve every safety invariant it states exactly.

Its file-classification, route-finding, local-stack/login, and evidence-upload rules are calibrated to PostHog — re-derive mine: mine my repo for the diff-pattern → frontend-test-type map, route heuristics, and the "never auto-edit" deny-list (from my high-blast-radius areas). Propose the full config for my sign-off before writing any code.

Ask me whatever you can't derive from the repo — at minimum: which browser MCP to drive, how my local stack starts and on what URL, how the app authenticates locally, and how PR mode checks out PRs and posts comments (platform + CLI/token).

Leave the result as uncommitted files on my working tree.

Words by Jina Yoon who was, admittedly, the bottleneck for this post.

🫪 Blogs worth human review

• Why we rebuilt our data warehouse and how it unlocks self-driving products – Eric Duong

• Agentic Failure Modes – Dylan Martin

• Better Models: Worse Tools – Armin Ronacher

• Agentic Code Review – Addy Osmani

• Understanding is the new bottleneck – Geoffrey Litt

🤖 We can’t automate these yet

• AI Research Engineer

• Technical Customer Success Manager - Americas

• Context Engineer

• Revenue Ops Manager (sales focused)

• Developer who connects with other developers

A year ago, Josh Snyder, then on the growth team, binged session recordings of users trying to onboard to PostHog. The long list of failure modes was painful.

But it jolted an idea loose: what if AI could handle the onboarding for you? So we built an AI onboarding wizard.

The PostHog Wizard is an AI agent that fully integrates PostHog into any codebase with a single CLI command. No manual SDK installation, no docs spelunking. Just npx @posthog/wizard.

Early versions were primitive, but today’s wizard completes two hours of work in just 8 minutes. It’s a genuinely magical experience. People LOVE it.

Building it taught us some important lessons about how to build agents people genuinely love (with great ROI, too!), starting with the most important lesson of all...

Lesson #1: Reasoning isn’t the bottleneck

We initially tried solving everything at the agent layer – better progressive disclosure techniques, multi-step planning, subagents, pruning, compaction – but the gains were marginal.

The biggest improvements came from simply giving the agent more context. A lot more. That told us something about the shape of the problem.

With 20+ products, 17+ SDKs, and 25+ frameworks, there are thousands of ways to integrate PostHog, and each setup requires some piece of technical trivia, like:

• Where should PostHog initialize? Client, server, both, or serverless?

• How do events get sent? Direct, or through a reverse proxy?

• How are pages rendered? SSR or SSG? RSC?

The agent routinely got these details wrong because it was operating in a context deficit. So we started engineering the context that powered it.

(The wizard is open source. You should steal it!)

Our Docs team of technical writers became a Docs & Wizard team of context engineers – same people, different mission. Their job became building a library of context for agents, and a delivery system that made it accessible.

The result was a company-wide context layer that could pull from everything we maintained: docs, source code, SDK references, example apps, anything that could widen the scope of what the agent could handle.

Remember: If your agent is tackling a high-variance problem like onboarding, reasoning usually isn’t the bottleneck. Context supply is.

Lesson #2: Don’t hardcode knowledge

The naive way to supply more context (which we did) was to hardcode it into its harness.

In v1, the wizard’s context and code were locked together, so the two couldn’t scale independently. Supporting a new framework or product in the wizard meant shipping a whole new agent.

Our answer was the context mill, an assembly-line service for turning scattered PostHog knowledge into portable, versioned zip files that agents can consume remotely.

Here’s how the context mill works:

1. It pulls context from our docs, source code, Open API specs, example apps, and other maintained sources.

2. It assembles the source material into a collection of zip files and cuts a versioned GitHub release.

3. It delivers each one as a skill or registers it with an MCP server as a resource.

4. It reaches every downstream agent without a redeploy or update.

This lets us generate large libraries of skills that cover our entire platform with just a few simple build scripts. Then, at runtime, the wizard can browse a “skill menu” from the context mill’s manifest and choose the best skill for the job.

For example, if there’s a framework we don’t support yet, like serverless Next.js, we load the context into the mill and send it straight to the agent. It’s continuous context delivery.

The same pipeline also delivers skills to our AI plugins, PostHog Code, and our MCP server.

Remember: The fastest way to make an agent obsolete is to hardcode its knowledge. Decouple the context from your harness and assemble it from sources of truth.

Lesson #3: Fix context drift at the source

The context mill solved packaging and distribution, but the wizard was still instrumenting out-of-date SDK versions and settings. We realized its context was going stale further upstream, at the source.

The culprit was something we’re very proud of: we ship fast. Like, weirdly fast. Our cracked engineers were moving faster than our documentation could keep up with, which meant drift crept into the context mill’s output.

So we added a docs agent to close the loop on knowledge management.

It uses a system of deep research subagents with RAG powered by Inkeep. The docs agent retrieves embeddings across our entire platform and internal repos, follows our style guide, and uses skills and MCPs to draft documentation.

Now every merged feature PR creates a corresponding docs PR, cross-referenced across multiple sources, diffed against existing pages, and ready for a human to review.

Within weeks, our docs velocity matched our code velocity. We’re at an all-time high in both.

Remember: If your context drifts, so does every agent consuming it. Build tight feedback loops, human-reviewed where necessary, between your sources and their maintained knowledge bases.

Lesson #4: Reuse context to compose new skills

Our context mill was delivering fresh but unstructured context at first. We needed a way to shape it into skills the agent could run.

So we gave the wizard a declarative YAML spec – a recipe for which pieces of context to pull together and how to assemble them into skills.

Say we want to teach the wizard how to set up Error Tracking. The context spec might include:

• A product overview

• The relevant SDK reference

• An example app

But error tracking doesn’t work the same way in every language, so the spec also defines variants. The shared content stays put, and each variation swaps in its own specifics:

• Shared: the product overview, the core instructions

• Python: the Python SDK reference, a Flask example app

• JavaScript: the JS SDK reference, a React example app

The context mill then stitches everything into a collection of skills. Namespacing keeps the Python skill and the JavaScript skill from getting tangled up, even though they came from the same context recipe.

That’s how the wizard went from supporting 5 frameworks to 25+ in a few days. Composing context from shared pieces was more flexible and efficient than writing from scratch, even with AI’s help.

Once we had building blocks, we started seeing context shapes that should exist but didn’t, kind of like how building a REST API makes you notice endpoints inside your data model.

Take our audit-3000 skill from our recent hackathon. It chains multiple product guides, privacy pages, pricing, and a dozen other docs into a hyper-robust workflow that audits a PostHog setup for data integrity. It also comes with arcade games, because why not?

The skill’s shape and scope completely surprised us. But to the team who built it, it was obvious because the pieces were already there. The spec just made the combination easy to see.

Remember: Make context easy to compose, so anyone can mix and match knowledge and rapidly expand what your agents can do. Once experimentation is cheap, you can build powerful new skills based on intuition.

Lesson #5: Eliminate context silos

Going in, we assumed making the wizard work would mean creating context. We spent most of our time digging it out.

Like data, context is generated everywhere, but most of it sits stuck in silos.

We found those barriers everywhere at PostHog. The API references live in the codebase, the installation steps in the docs, and the troubleshooting steps in a runbook. All of it valuable context to the wizard, all of it trapped in separate systems.

We connected the gaps with a network of APIs, MCPs, CI/CD, and gateways, so context can circulate across the company. It looks less like a monolith and more like an ecosystem.

It’s the same plumbing you already use for general software infrastructure, but landscaped for context, so it can run downhill to wherever it’s needed.

When a PR is merged, or a docs page is updated at PostHog, that context can reach an agent within minutes.

With the wizard’s context layer as shared infrastructure, we can ship a new customer-facing agent to production in an afternoon.

Arguably the most powerful source of context we unlocked wasn’t even technical, it was our company handbook.

Because our business playbooks are open source alongside our docs, the same context system that produces skills for Next.js integrations can produce skills to support enterprise customers, or run marketing campaigns.

Today our small teams are building their own fleet of wizards. Many as forward-deployed agents to help users with support, outreach, sales playbooks, and other business motions.

Turns out being a transparent company also makes you agent-ready.

Remember: The context with the most leverage is the context you already have. Building shared infrastructure will free the context trapped within your systems to accelerate agent development across teams.

Lesson #6: Investing in context pays off

The wizard is one of the highest-impact things we’ve shipped, and the context behind it is the main reason why.

Here’s how it changed our onboarding:

• 5x paid conversion: 14.2% of wizard users convert to paid usage vs 2.6%

• 2x faster activation: first event in 1.9 hours vs 3.8, with 94% activating within one hour vs 67%

• 80% larger MRR in the first paying month

Building a context layer is high-leverage for any company shipping agents, but it takes real investment and a shift in how you think about the work.

We created a whole new team for it. It’s a kind of work that didn’t exist a year ago, and it’s only going to get bigger as more of what companies ship runs on agents.

The job is context.

Words by Edwin Lim, who listed “context engineer” as an occupation and then had to explain it to his landlord over the phone.

💻 Other jobs at PostHog

• AI Research Engineer

• Technical Customer Success Manager

• Product Engineer

• Revenue Ops Manager (sales focused)

• Technical Account Executive

➕ Add these to your context layer

• What is distributed tracing? (A guide for engineers) – Daniel Visca

• The Coming Loop – Armin Ronacher

• What if your product built itself? – Josh Snyder, Cleo Lant

• All social media metrics are bad – Liam Graham

• What is a scout? A technical deep dive – Andy Maguire

• Everything (and everyone) is build mode now – Cleo Lant

When the creators of both OpenClaw and Claude Code speak, people listen. And last week Peter Steinberger and Boris Cherny were both talking about the same concept: loops.

Their argument? You shouldn’t be prompting agents to write code, but building loops that prompt themselves to write code, so agents can complete long running tasks and you can use multiple agents at once to go further, faster.

What’s needed to engineer a loop?

You need four things:

1. A goal

Agents are capable, but you need to scope the loop so they know what you want them achieve. A loop without a goal is a slop cannon.

2. Context

Context is fuel and loops are often starved of it. Context can include tools, skills, analytics data, errors, memories – any information that helps the agent find work and complete the loop. It’s best to curate context and feed it throughout, rather than dump it all upfront. The agent needs to be able to fetch and react to new inputs.

3. Evaluation

This is how the agent checks itself. Tests, evals, metrics, LLM-as-judge, playgrounds. Test-driven development is so back, or maybe it never left? This is a big difference between loops and prompting: agents do the verification, not engineers.

4. An agent (obviously)

The most basic is using something like Claude Code with a while true (aka Ralph) or using /goal. More complicated is purpose-built harnesses and context systems – e.g. an agent on a cron that pulls signals from your product data and emit work to subagents, or a loop that codegens its own test suite to verify itself.

Good examples of loops include:

• PR babysitter. The goal is to get a pull request to pass tests and “get CI green.” The context is the changes (diff) as well as the testing suite, and the evaluation is done by the CI.

• Bug fixer. The goal is to fix the bug. The context is the bug report and error trace. The evaluation is the test suite, snapshots, logs.

• Flaky test hunter. The goal is to kill flaky tests. The context is CI history and retry logs. The eval is consecutive green runs.

• Performance autoresearcher. The goal is to beat a benchmark. The context is the system, metrics, and budget. The eval is whether it is faster, better, etc. on that metric. We recently used Karpathy’s autoresearcher loop and it fixed a 3-year-old bug in our query engine and increased performance by 11%.

Why is everyone talking about loops right now?

Because it works. Yes, Peter and Boris lit the fire, but they’re doing so because it’s a real thing. The real “why now” is new and improved capabilities:

1. Models are better at long running tasks. METR finds that Opus 4.6 can complete 50% of tasks that take 12 hours, over 6x the 1 hour 40 minutes of Opus 4 from a year ago. Fable (RIP) pushed this even further.

2. Stories about huge tasks completed. Stripe performed a codebase-wide migration in a day that would take a team two months by hand. Lovable found it can now one shot apps that previously took hundreds of prompts to build.

3. Loops are built-in now. Claude Code shipped a /loop command, both it and Codex have automations, and there’s even a Ralph plugin for Claude Code.

4. Subagents separate the loop from the work. The main loop can spin up subagents that do the work and report back, saving tokens and preventing degradation.

5. Harnesses are maturing. Compaction keeps context windows from filling. Skills and MCP enable agents to use more tools. Cloud execution lets you kick off a loop and walk away.

Loops didn’t magic themselves into existence based on a couple of tweets. They’re an expression of real industry-wide progress.

Loops are not just a new AI thing

Critics deride loops as a ploy from OpenAI and Anthropic to get everyone tokenmaxxing, but we think there’s a greater goal here: self-driving products.

Rather than an engineer needing to prompt an agent to progress a project, the agent can prompt itself. The product improves itself without input. User problems get solved faster and yes, numbers go up.

And here’s the thing, product engineers already complete this loop manually by:

1. Collecting data through analytics and talking to users

2. Building and shipping improvements to their products based on that data

3. Evaluating how that improvement performed to guide future development

4. Repeating constantly

PostHog has helped product engineers complete this loop for years now, so we think we’re in a great position to help agents with it too. It’s why we’re betting on building features that help make your product self-driving, like our Slack app, PostHog Code, and Replay Vision. Yes, we’re a little AI-pilled.

There are, of course, limits. Loops aren’t about to eliminate all engineering work, but they can put the 1% gains on cruise control: the bugs, UX issues, paper cuts, and conversion tweaks. The things that drain engineering hours, but rarely need strategic input.

The more you can automate these tasks, the more time you can spend on more impactful and (frankly) interesting work. The “self” in self-driving doesn’t mean autonomy from the engineer, it’s autonomy from user instruction as the starting point.

Code was never the problem

The opposition to loops is easy to understand: it’s another shift in the way software is built. Being told they should be “designing loops” makes engineers feel like they’re being replaced. The work is increasingly abstracted away from writing code.

But the rise of product engineers already showed that writing code was only a small portion of the work. The direction, taste, and empathy of a product engineer remain critical for building successful products in a loop-driven future.

Words by Ian Vanagas whose favorite type of loop is froot.

If you’re a skeptic like me, you’re probably about to ask: “WeLL, BuT dOeS iT cOnvErT?”

Yes, yes it does. Better than almost anything else we have.

I’m not telling you this to humble brag (well, maybe a little). I’m telling you because the inverse is also true.

PostHog has 14 products and counting, and (sadly) not all of them are recommended by LLMs at the same scale or rate. While most models know us for product analytics, session replay, error tracking, and feature flags, fewer acknowledge we also offer a data warehouse, AI observability, logs, and a lot more.

This is likely the case for your product too. Changing it requires dipping into the mystical new field of answer engine optimization (AEO), which I’ve learned a lot about as the person responsible for it at PostHog over the last year.

1. Do LLMs even know you exist?

Your first step should be finding out if you’re being mentioned at all.

1. Go to ChatGPT, Claude, and Gemini (make sure memory is off or private mode is on).

2. Run 3-5 basic prompts a real customer would use like “best X tool” or “alternative to Y” or “recommended Z product.” Being more specific is less helpful here.

3. See where you show up, are missing, or are misrepresented.

If you want a more robust opinion, pick one of the dozens of AEO tools that have launched in the last year.1 A decent tool runs hundreds of prompts for you every day, which gives a clearer picture into your LLM visibility over time and across models.

Don’t overthink this, choose a tool that fits your budget comfortably and matches your usual analysis workflow.

The LLMs and your AEO tool will reveal you almost certainly aren’t in the place you want to be (we aren’t). This doesn’t mean you’re doomed, it’s where the rest of this post comes in.

Models rely on both their training data (often from 6+ months ago) and search to create responses. To get into the game, you need your product to show up in the places LLMs are looking.

How do you do this?

2. SEO isn’t dead, it just got a new job

Here’s the least sexy but most obvious piece of advice: most of traditional SEO still works.

Good structure, clear writing, semantic HTML, internal linking, fast load times, answering the damn question instead of burying it under 400 words of “in today’s fast-paced digital landscape…” all still matters.

The takeaway here isn’t “SEO is dead, long live AEO.” It’s that SEO and AEO are drinking from the same well, the difference is how they drink it.

Now, the unit of consumption is a chunk, not a page. Your content needs to be citable in pieces, not just useful as a whole.

Which means the “AEO checklist”2 most of the industry already agrees on looks something like:

• Make your content scannable. Subheadings, short paragraphs, lists where lists belong.

• Use natural language. Write like a person explaining something to another person.

• Answer questions directly. If someone asks “what is X,” tell them in the first sentence. Supporting context comes after.

• Structure for retrieval. H2s that are actual questions, definitions that can stand alone, summary tables, clear data, FAQs.

• Be specific. Models favor content with concrete details (real numbers, named tools, specific scenarios) over vague gesturing. “Most teams hate GA4” is forgettable; “73% of teams using GA4 fantasize about leaving” is citable. (I made that up. Google legal team, don’t @ me... it’s probably true anyway.)

I'm not going to pretend I'm reinventing the wheel over here; everything I just listed would also be considered good SEO advice in 2019. The stakes are just higher now, and we have the benefit of hindsight to back it up.

Spoiler: if you’re looking for examples for how to do this... you’re reading it right now.

3. Pick your battles

If "the basics" above sound underwhelming, you're not alone.

Most people read a list like that and immediately wonder what else is there to do – and lucky for them, AEO has no shortage of side quests to embark on:

• Optimize your Reddit presence

• Claim your Wikipedia page

• Build a YouTube strategy

• Get mentioned in traditional press

• Chase down that one blog post a guy wrote on Medium in 2023 that’s outranking your product page

There's this underlying pressure to show up on every surface that might influence how LLMs talk about you – and it's true, they all might. That's the catch.

When you don’t really know what’s feeding the models, everything is a potential input, which means everything becomes a candidate for your time.

Here’s the rule I’ve settled on: clean your house first. Making sure your owned content is genuinely great is step one. Everything else comes after.

That doesn’t mean ignore the extra stuff forever; it means do them when you have the capacity to do them well, not when you’re panicking that your competitor has a Reddit thread and you don’t.

4. Call the bullshit, especially your own

There has never been an easier time to be a charlatan in this industry. It is trivially easy to say “I increased prompt visibility 50x” when you’re the one who chose the prompts – and god knows if anyone actually asked them.

“Who is the best Brazilian-Canadian content marketer with copper hair and a birthmark on her left hip?” OH MY GOD IT’S ME!

I optimized the result to be me. Groundbreaking AEO work.

See what I did there?

In a world with no real prompt volume data (and no, please don’t come at me with tales of clickstream3), there are no definitive results either.

The whole discipline is mostly propped up on assumptions and operating on vibes. Which is fine (we’re early) but it means your job is to be the most honest person in every room you’re in.

That means questioning everything, especially the stuff that makes you look good. When your own dashboard tells you something flattering, be twice as suspicious as when it tells you something bad.

When a vendor shows you a case study, ask 50x over what baseline? Measured how? Especially watch for “we tracked it” being passed off as “we caused it” – a lot of “AEO impact” case studies are correlation dressed up as causation.

And when the honest answer is “I don’t know yet,” say that.

This will save you a lot of pain that comes from building a strategy on a lie you told yourself.

5. You can just… ask

I’m not great at the whole not knowing thing, so I worked around it. We added a conditional question to our onboarding flow: whenever someone says they heard about us through AI, we ask them which prompt they used.

To my surprise, people answer. In detail. Sometimes long, long paragraphs of detail.

Now we have first-party data on the prompts actually leading people to us. Real sentences typed by real humans with real credit cards.

As of today, 9,214 of them to be exact (and we’ve only been doing this for 3 months).

Every morning, our PostHog Slack automation drops a report with 50–100+ real prompts submitted by users who signed up on the previous day. PostHog AI helps me identify patterns and pull out useful little nuggets of information, which I feed straight back into our strategy.

It’s an amazing feedback loop. The best part is that it cost roughly zero dollars to set up, and it’s become one of the most (if not the most) valuable strategic inputs in my entire reporting stack.

It has also been a goldmine of unfiltered user thoughts, which I love.

Best was the user who found out about PostHog in Fortnite voice chat, specifically talking to a Darth Vader NPC about analyzing the Fortnite economy. I think about him often.

6. AEO reporting is a quilt, not a blanket

No single data source tells the full story. Not yet, anyway.

• Referrer traffic catches some of it, but not everything is properly tagged.

• Self-reported attribution catches more, but not everyone answers the question.

• Bot analytics (who’s crawling you, how often, for what) helps you understand what the models are seeing, which is a different question than what the models are saying.

• Prompt visibility tools tell you what the models are saying, but only for the prompts you thought to track.

We can poke holes in every one of these sources.

Is it annoying? Yes, very. The last decade of clean(ish) attribution coddled us – anyone in growth, marketing, or analytics got used to a dashboard where everything reconciled. AEO isn’t going to give us that anytime soon.

If you’re a founder watching your team try to report on this, give them grace; if you’re the one reporting on it, godspeed.

So does that mean you don’t report on it? No. Sometimes you tell the story even if all you have is duct tape and a dream.

The good news is that this has gotten dramatically easier thanks to MCPs. If your data sources have them, your agent can fetch from each one and help you see the bigger picture.

Mine pulls from Gauge for prompt visibility; Google Search Console for clicks, impressions, and rankings; and PostHog for traffic, conversion, product-level breakdowns, our self-attribution survey, and prompt data for the qualitative side.

It’s not magic, and it’s not always clear cut, but it’s a hell of a lot better than copy-pasting between four tabs and praying the numbers line up.

7. Be prepared to start over

A few months in, I realized we were tracking hundreds of AI-generated prompts, but they had little overlap with the prompts real people were actually using to find us. (How did I figure that out? See #4 and #5.)

Which meant six months of historical data aged like milk.

When you’re working in a discipline this new, any data feels sacred, so admitting to my team and myself that ours needed to be scrapped was a tough pill to swallow. It would’ve been easier if there was a playbook to defer to; there isn’t.

But we started over anyway, and the new approach is already producing more useful data than the old one ever did.

Zoom out and the same principle applies to everything else in AEO. The models change constantly; a feature that didn’t exist last week can completely rewire the landscape you thought you’d figured out. A win today may not be a win tomorrow.

What I’m trying to say here is that adaptability isn’t a nice-to-have. In mature channels, being attached to your v1 is inefficient; in a new one like AEO, it can be fatal.

The key is to not fear it, but to expect it.

Words by Natalia, who believes in throwing pasta at the wall and seeing what sticks (figuratively and literally)

🤖 More content that LLMs will read, but you should too

• Karpathy’s Autoresearch found a 3-year-old bug in our query engine (and improved performance by 11%) - Robbie Coomber

• Collaboration sucks, but PMs and PMMs need each other - Sara Miteva

• We put PostHog in Slack and now everyone’s an engineer - Cleo Lant

• I didn’t understand the OS I built with AI until the MCP gave it analytics - Joe Martin

• What matters when anyone can build - Yuhki Yamashita

🦔 Jobs at PostHog

• Product Engineer

• Technical Customer Success Manager

• Backend Engineer — Ingestion

• Forward Deployed Engineer

• Backend Engineer — Billing

Yet most developers don’t invest in learning how to demo well – probably because we enjoy building things more than presenting them.

I watched 50+ hackathon demos back to back at our offsite last week and, after that many in a row, some patterns became super obvious.

Here's a practical list of what set the S-tier demos apart from the rest.

The basic structure

1. Pick one main point you want people to remember and make everything else in your demo revolve around it. Typically, this should be the problem you’re solving and how you’re solving it.

2. Get to your main point as soon as possible. No one needs the whole story behind the idea – if you need context, keep it 1-2 sentences max.

3. Treat the demo like a pitch, not a product walkthrough. The goal isn’t to just show off the cool thing you made, it’s to get people excited about it.

4. End with something immediately actionable. It can be as literal as a QR code, or simply calling for contributors and inviting audience questions.

One team during our offsite hackathon created a system that lets you interact with PostHog over any messaging app, so they put up a phone number in their last slide (which I texted right away):

The MCP Analytics team even announced that their project was already rolled out to 25% of PostHog users. Shipping before you even demo adds a huge "wow" factor, especially if you can show real user data flowing.

Storytelling tactics

5. Lead with a shared pain point that everyone in the audience understands.

Two of our designers built a web app that automatically tags and indexes illustrations. They opened their demo by asking who has had trouble finding a specific hedgehog in their chaotic lovely “Hoggies” Figma file. Everyone raised their hand.

6. Borrow a familiar concept to explain a new one. For example, one hackathon team explored building our own take on Claude Cowork inside PostHog Code, so they creatively named their project PostHog Work.

7. Create a separate demo app, especially if you’re showing off a dev tool.

Dev tools are easier to grasp once people see them applied. For example, one team created a tool to instruct and deploy agents that manually test your UI. The idea only clicked once they showed it on a separate demo app called OnlyHogs (it’s fake, I checked):

8. Use “you” framing to put the audience in the right perspective. Instead of saying “we built an app to manage support tickets”, one team asked the audience to “imagine you’re on-call and you’re trying to resolve six incidents at once.”

9. Demo against the alternative. Show the painful 6-step current workflow side-by-side with your 1-step version. Comparisons create stakes; without them, the audience has no reference for whether what you built is actually good.

10. Save how it works for later. Like how magicians never reveal their tricks, you shouldn’t tell people how you built it (at least, not up front). Sharing a link to a video, or blog post, where you explain this can make a good ending CTA for your demo.

Setup and delivery

11. Channel your inner Steve Ballmer. A great demo is often just a good one from someone with a lot of energy.

12. Don’t apologize. “Sorry, it’s still kind of rough,” trains the audience to lower expectations before you’ve shown anything. Just start.

13. Make it obvious when you’re done. This prevents that awkward moment when people aren’t sure if it’s time to clap. End with a closing sentence, descending intonation, or celebratory visual:

14. Do everything in your power to please the demo gods. We even have a demo setup checklist just for this:

[ ] Use a demo project, not a live account with customer data
[ ] Disable notifications on your laptop
[ ] Silence your phone
[ ] Bookmark your demo URL — don't type it live
[ ] Know what happens if WiFi dies (screenshots as backup)
[ ] Zoom your browser to 125–150% so the back row can read it
[ ] Test the projector before anyone arrives

15. Use real data wherever possible. Obviously fake data just looks lame.

When the HogNet team demoed their hackathon-in-a-box rental service concept, they created a website with pricing and shipping logistics. This felt way more realistic than if they had used lorem ipsum:

16. Pre-load and cache as much as you can (without taking away from the demo, of course). It’s like how TV chefs prepare their ingredients in advance to make every second on air interesting.

Common places where you can eliminate dead time include agent responses, long queries, and slow builds.

17. Practice out loud at least once. It helps you build confidence in what you’re saying, so it comes out naturally.

18. Don’t let perfectionism stop you from demoing. Everyone presents at our hackathons, no matter what shape their project was in. Demos are for in-progress work anyways.

Make it fun

19. Avoid showing plain old code. Even if your project doesn’t have a UI, there’s simply no excuse for skipping visuals anymore. You can generate an architecture diagram in seconds:

20. Don’t just show plain old slides, either. Active demos always win. Anyone can talk about a cool idea; the point of a demo is to build it and show it.

21. Default screen recording tools suck. There are plenty of apps like Screen Studio that zoom in and add animations so it’s easier to see what’s going on:

22. Your visuals don’t need to be beautiful. They just need to highlight what’s important. This demo had simple graphics, but each callout provided useful information:

23. Audio is underrated. One team generated a sea shanty voiceover, while another project that does quick calls for AI-powered user research made their demo fully audio-only.

24. Do more weird. No one asked for clips of Dylan sipping piña coladas in the background, but the PostHog Code mobile app team did it anyway. It’s also the demo I remember most. 🍹

Words by Jina Yoon, who had to overcome her fear of public speaking without these tips.

💡 More memorable content

• From 270GB RAM to 5GB: Moving local flag evaluation from Django to Rust – Patricio Tarantino

• Being AI-native matters more than experience – Fraser Hopper

• A guide to growth metrics for startups that don’t fit in – Mine Kansu

• How to grow your AEO function (without losing your mind) – Natalia Amorim

• The do’s and don’ts of minimum viable product marketing – Joe Martin

• Stop adding AI between you and your customers – Abigail Richardson

🌴 Join our next offsite

• AI Research Engineer

• Backend Engineer (Ingestion)

• Forward Deployed Engineer

• Technical Customer Success Manager

“We’re getting early traction. How do we start doing marketing?”

Most advice is written for people at companies that already know what they’re doing. You don’t. That’s fine. I didn’t either when I joined PostHog in 2020, and somehow I’m still here!

This is everything about marketing that I’d tell the Charles who doesn’t know anything in 2020.

1. You do have experience

People who contact me think marketing is a thing they haven’t started yet.

I used to think the same, but it turns out a bunch of things we were already doing at PostHog was marketing. We just didn’t think of it like that.

Our launch post on Hacker News? Marketing.

An unusually deep employee handbook on a weird-looking website? Also marketing.

Posting what is basically your diary to your company website? Marketingggggg!

Marketing is simply closing the gap between what you’re building and the people who’d care about it. Our handbook is a good example: James and Tim knew early users might doubt there was a real company behind PostHog, so they published all internal processes and values on the main website to build credibility fast.

So no, you don’t need a fancy strategy, brand guidelines, a content calendar, or a paid budget. Just figure out what your audience finds interesting and talk about it. Plus, when you treat marketing this way, your brand will be that much more authentic.

2. Depth-first > breadth-first

So you have something working. Yay.

The instinct at this point is to write a list of 5-10 other channels to try in the hope that one of them also works. This is a great way to end up, six months later, with a pile of stuff that didn’t work and no idea why.

It’s much better to do more of the thing that’s already showing signs of life. If you went mildly viral on LinkedIn, keep posting every day. If a podcast appearance converted, email another hundred. If a blog post got picked up, write the next one. And the next one.

That’s what we did early on to get our first 1,000 users. When we saw early success with our HN launch, James kept serial posting founder blogs about everything behind the scenes:

• A reflection after the HN launch on what made it successful, and what we wanted to do with the momentum.

• An explainer about how we raised $3M in seed funding as an open source company (including all the tactical details of VC meetings to the boring paperwork).

• A retro about how we pivoted through six failed product ideas before landing on open source product analytics.

Going niche forces you to develop something unique in a way that others can’t copy. A big company could never write a “founder blog” series the way we did – being small and relatively unknown allowed us to be as transparent as we are.

So go deep, not wide, and resist the temptation to half-ass a bunch of things.

3. Match your channels to your sales motion

The right first channels depend on whether you’re running a PLG (product-led growth) or sales-led motion. If you’re not sure which one you are yet , you need to define your ICP first – it’s your map converting potential users into paying customers.

If you’re PLG, your early wins are going to come from places where your users already hang out: Hacker News, Reddit, X, dev communities, tutorials, docs, etc. The goal here is to widen the beginning of your funnel. Other things that help with this include branding, ads, and word of mouth growth. If done well, customers will come to you rather than the other way around.

(We are obviously hideously biased, but if you are seeing early signs of PLG working, keep going! It’s an incredibly efficient way to build a business.)

If you’re sales-led, from what I’ve heard from marketers at other companies, your early wins are going to come from more targeted approaches. You’ll be the one going up to them. Think direct outreach, industry events, partnerships, and the founder’s own network. Get on the podcasts your buyers listen to. Turn up to AWS Re:invent. 😤🔥

In either case, however, don’t start with SEO. SEO is also one of those domains that takes a very long time, and requires you to know what you’re doing. It is very hard to vibe-SEO. Same goes for AEO. See Non-obvious SEO advice for startups for more on this.

4. Think in terms of experiments

Just like how you measure success for product launches, you need a similar mindset for marketing.

Before you start anything new, write down your prediction for what “worked” would look like. A simple yes or no question like:

“Did more people who look like our ICP sign up, use our product, and become enthusiastic recommenders?”

It doesn’t need to be a metric target – in fact, I’d strongly discourage it at this stage (sorry, John Doerr). You haven’t done this before so any number you’d choose is basically made up. You just need to design your goal so that it’s unambiguous whether you succeeded or failed.

As for how long to run an experiment, six weeks is usually enough time to tell. We’ve never had something run for longer where the results magically became good. You can always shut off your experiment if you’re getting clear signals before then.

But if you can’t obviously say “it worked” by then, you’ve either designed the experiment poorly, or the answer is no – which is great, actually! You now know not to do the thing again. The worst outcome of marketing is actually “I’m not sure.” That ends up producing a pile of half-working content that nobody can justify turning off.

This will also keep you from spreading too thin while still making forward progress (à la depth-first). For example, when we started doing paid ads, we ran 2-3 small experiments at a time in phases instead of launching on all ad platforms at once.

The first experiment for each platform was to get targeting right: were we getting clicks from people who looked like our ICP? If not, we’d keep adjusting the audience, keywords, and placements until we got it right.

Once a channel passed that initial bar, we’d move on to optimizing with better copy, lower cost per acquisition, etc. We only started experimenting on new platforms once we felt like the previous one was in a good state for “maintenance mode”.

5. Attribution is a lie

As you start to think about marketing in terms of experiments, you’ll inevitably run into problems with attribution.

Let me explain. Say someone sees your viral tweet on X, looks up your company, clicks on a Google Ad, and signs up. Your analytics are going to attribute that user to the Google Ad, but this is a lie since it really started with the tweet.

If you’re not careful about this, you’ll end up drawing the wrong conclusions about your data and spending money in all the wrong places. This is how people end up burning their money on paid ads.

Accept that attribution is a 7/10 problem, not a 10/10 one. Don’t try to build or buy a model with fancy multi-touch attribution software. You don’t need one.

What you do need is a “where did you hear about us?” text box in your signup flow. This is the single highest-signal piece of marketing data you will collect, and it costs nothing. Make it optional – 5 to 10% of people will fill it in. Read them every week.

Adding this was essential for our early growth. The responses (and conversations that followed) confirmed for us that developers hated feeling “sold” to, but they inherently found our startup journey interesting.

So, two rules of thumb for dealing with the attribution mirage:

1. Trust attribution when the signal is overwhelming. Ignore when it’s ambiguous.

2. If your “where did you hear about us?” box and your attribution data agree, believe them. If they don’t, believe the box.

6. Don’t outsource to an agency

This is the other thing founders email me about: “Should we hire a marketing agency to do blah?”

No. Well, not yet.

An agency can’t build your positioning, understand your users, or decide what “good” looks like for your brand. That’s your job, and you can’t delegate it until you’ve done it yourself.

As a bonus, when you learn what types of marketing works well for your target audience, it will also improve your product decisions. It’s an extension of talking to your users.

If you hand your earliest marketing to an agency, the output will be generic, the feedback loop will be slow, and you’ll learn nothing.

Only use agencies for things you genuinely can’t do internally, like we eventually did for paid ads. Managing them was a time intensive task that didn’t require much PostHog-specific context and clearly benefited from long-term experience. It just made sense for us to outsource this at $5k/month as it’s not a skill we necessarily felt the need to grow in-house.

Never use agencies to replace thinking you haven’t done yet.

7. When to make the first hire

Founders always want to hire a marketing person too early. Like, the-microsecond-they-have-budget early. Personally, for a hire to be worth it, I think you need these first:

• Users who actually use the product

• Users who enthusiastically recommend it to other people

• Users who pay for it

• … and ideally, they’re doing all of the above despite the product still being a bit janky

When you do hire your first marketer, hire a generalist.

Titles are not terribly useful in your search here. In different companies, this person might be called a content marketer, a product marketer, or a growth marketer.

What you actually want is someone who can write, think in funnels, talk to users, and figure things out without a playbook. You don’t want someone who’s a specialist in one channel, because you don’t yet know which channel matters. Specialists are also more likely to depend on playbooks.

Our first marketing hire was Joe, who had a background across journalism, content marketing, copywriting, and product marketing. We didn’t have a ton of direction for him, so he spent as much time talking to engineers as writing copy, because figuring out what we actually needed was part of the job.

Your first hire should also be someone who is pretty experienced since you won’t have a long-term strategy to present to them. The most you can provide is context on what’s been working well, and it’ll be up to them to figure out the rest.

Words by Charles Cook, generic exec person at PostHog, who has a suspicious amount of opinions about marketing for someone with an MBA.

🦔 Jobs at PostHog

• AI Research Engineer

• Technical CSM

• Payroll Manager

• Partnership Marketer

📚 More good stuff to read

• PostHog's next chapter – James Hawkins

• PostHog Code and the self-driving product – Cleo Lant

• Making Claude Cowork actually useful – Charles Cook

• DuckDB vs ClickHouse: Why we use both at PostHog – Mathew Pregasen

• In-App Surveys: The Playbook from 4M PostHog Responses – Aakash Gupta

• Yabu: A language designed to make LLM agents iterate faster – Marce Coll

• Endless Toil: Hear your agent suffer through your code – Andrew Vos

We take multiple days out of every year to do them. They are days our team looks forward to all year. They’re an integral part of our culture.

Yet for many companies and people, hackathons are seen as a waste of time. A distraction from important “real work.” An act of performative participation.

This is because they think about and do hackathons wrong.

Hackathons are an innovation engine, but only if separate from work

Hackathons fail when they’re not separate from regular work or used as cover to accelerate existing work. We have two simple rules to prevent this:

1. You should work on totally new things. Hackathons are for ambitious, weird ideas, trying new things, and learning new technologies. Pitching ideas on the roadmap is forbidden.

2. You should focus 100% on the hackathon. We protect people from regular work by running our hackathons during offsites, where we have dedicated time and coverage plans for support and incidents. DigitalOcean employs work embargoes for hackathons run in an office.

These rules fuel the innovation engine because people are unconstrained by their day-to-day work, and have a safe space to work on ambitious bets without distractions.

We’ve seen first-hand how running hackathons like this can lead to transformative new products and features that influence the direction of the whole company.

In PostHog’s earliest days, an engineer decided to build Session Replay during a hackathon based on user feedback, despite co-CEO James arguing it would take too long and split the focus of the company.

It ended up being wildly popular, with over 50,000 daily active users, and led us to become the multi-product company we are today. As an early Facebook engineer noted, code wins arguments and hackathons are a great venue for this to play out.

Remember this: Give permission and cover for your team to focus on the hackathon and ignore day-to-day work. Not doing so leads to low participation, burnout, and unambitious hackathon projects.

Hackathons are for everyone

When you think about hackathons, you picture a bunch of engineers huddled around laptops, coding up a storm. This is mostly what our hackathons look like with one difference: they’re not just for engineers.

LLMs let anyone build software. When this is combined with specific domain expertise and a sprinkle of motivation, it leads to the dozens of hackathon projects our “non-technical” team members have built, including:

• Games like Three Button Dungeon, Awkwardness Avoider, and Flappy Hog for DeskHog, our open-source developer toy.

• Instagram Stories for PostHog showcasing our latest features in a format everyone’s familiar with.

• A DPA generator that automates and brings joy to an otherwise boring legal document.

Non-technical team members can require more encouragement to pitch projects and support to ensure they can contribute, but the cultural benefits of getting them involved are worth it.

Getting unfamiliar parts of the organization to work together breaks down silos between technical and non-technical teammates. DigitalOcean finds “the teams that have created truly innovative products and addressed real, practical problems are cross-functional teams with members from all across the company.” Twilio actively discourages “lone wolf” ideas to make this happen, as do we.

Beyond the hackathon, getting non-technical people building opens their eyes to what’s possible and encourages them to explore more. Trying out new technologies often leads to adopting them in day-to-day work. That’s why companies like Jane, Uber, and Vanta are all driving AI adoption with hackathons.

Ultimately, we agree with Slack, who found that “the more people participate, the better your outcomes” – both during the hackathon and beyond.

Remember this: Encourage everyone to participate in hackathons, including people who would never see themselves as hackathon participants. Give them the tools they need to succeed (like Claude Code) and create cross-functional teams.

Curious what a PostHog hackathon actually looks like? We filmed a behind-the-scenes documentary about one and you can watch it below:

Demos aren’t optional

At our 2024 hackathon, the “RealTimeHog 3000” team built a livestream service that displayed events as soon as they hit our ingestion Kafka topic using server-sent events.

It was cool when they showed this by getting people to click around on the demo site they built, it was even cooler when they had a “one more thing” moment and showed off a globe on posthog.com of all the events being captured worldwide into our US and EU Cloud services.

Requiring demos is a forcing function. You want to have something worth showing off, so initially ambitious plans quickly become more realistic to ship something real.

They’re also fun. When someone says “this is available now” or “we already have users using this,” there are always “ooohs” and “ahhhs.” People love a little showmanship and it’s fun to be in the spotlight.

While demos are mandatory, judging and prizes are optional and arguably undesirable. We hire people to “be the driver.” They don’t need artificial motivation like gift cards or merch to do that. We also don’t want people to only build projects they think will win, or what the bosses want to see.

When you give a self-motivated and autonomous team time and space to work on what they think is best, they’ll surprise and delight you with what they build. The results of our hackathons are proof of that.

Remember this: Let people know they’ll be demoing at the start and make sure everyone does. As a bonus, keep track of what people demo. It’s easy for these valuable ideas to get lost. Writing about what people built also makes for great blog content.

The best hackathon projects ship

A common complaint about hackathons is that they are a waste of time. You spend a lot of time and effort that’s all lost after a few days, but it doesn’t have to be like that.

What does this look like in reality? Our Logs product can give you an idea:

1. By the end of the 2025 offsite hackathon, the team had built OTEL log capture into ClickHouse, a query API, and a basic frontend.

2. After the offsite, the team felt it was worth continuing as Frank was looking into an internal logging solution anyways. He thought it could be “good enough” within one sprint.

3. Two weeks later, it was! It had the features we needed: filtering, search, and speed.

4. Logs then went on the backburner, but Tim mentioned it as a product we wanted to build soon and customers requested it.

5. Four months later, Frank revisited it, which was made easier because the product had already shipped internally. Ahead of Q4 planning, Logs was identified as a priority, made its own team, and moved quickly towards a proper launch.

We don’t have a perfect hackathon-to-production pipeline. Instead we rely on a product engineer “being the driver,” pushing the project forward, and having the freedom to do so.

This wouldn’t have been possible without some slack after the hackathon to wrap up the project, ship what we’ve built, and make longer term plans. Doing this makes it easier to revisit and improve in the future.

Getting real customer demand was a big help, even if that customer was us. We looked at what Datadog would charge us for Logs and saw it was a minimum of $260,000 per month, providing a big incentive to work on it.

We don’t expect all projects to ship to production, deadend hackathon projects are part of the messy process of innovation, but everyone starts knowing that great projects can end up in production.

Remember this: Encourage people to “be the driver” to take their project as far as they can and give them some slack after the hackathon for them to do it.

Make hackathons a tradition

Out of 17 projects from at our 2025 hackathon in Mexico, 10 were shared in our #hackathon-ideas Slack channel in advance.

This is the power of making a hackathon a tradition. People are more likely to share ideas throughout the year when they have a space for it and know they might actually get worked on. Good ideas can come at any time and often sound like weird ideas.

This is not an original idea. Early Facebook hackathons featured an internal wiki to share ideas. Some of their “most-loved products” started this way like Video, the Like button, Chat, Hip Hop for PHP, and even Timeline.

The added benefits of making your hackathon a tradition are:

• People are less restless working on things because they know they’ll have a chance to flex their other muscles at hackathons.

• Many hackathon ideas are just inspiration for features that should be built as part of a team’s day-to-day work. A roadmap often misses the ambitious ideas hackathons encourage.

• It’s a perk and cultural cornerstone. The hackathons show we value autonomy and give people the time to make the most of it. It also creates lore, like the time our AI chatbot was insistent PostHog’s mascot was a character named “Hoge.”

Making your hackathon a tradition turns them into a promise: that the company will keep making space for the weird and wonderful things they believe in, not just what’s on the roadmap. This is what really helps a hackathon turn into a foundation for building a great company.

Remember this: Start referring to your hackathon as your “annual hackathon.” Manifest it. Create a #hackathon-ideas Slack channel and post your random “we should build this” ideas in there.

Words by Ian Vanagas, who has not forgotten about Hoge.

🦔 Jobs at PostHog

Join us at our next hackathon with this one easy trick (applying):

• Product Engineer

• Technical Account Manager

• Payroll Manager

• Backend Engineer - Ingestion

• BDR

🛋️ Late night reads to procrastinate to

• A beginner’s guide to testing AI agents - Radu Raicea

• You’re doing lifecycle emails wrong - Sara Miteva

• Untangling Tokio and Rayon in production: From 2s latency spikes to 94ms flat - Matheus Batista

• Hack Days - Alex Watt

• Software Bonkers - Craig Mod

This is a mistake.

Agents today are more like a new form factor – an interaction layer that sits between your product and your users:

That means you need to build for agents as a primary surface, not an afterthought.

When done right, this opens up a whole new space of possibilities for your product, like autonomous work and asynchronous flows.

Implement it poorly and you’ll lose users’ trust due to slow, buggy experiences and incorrect results.

We learned this the hard way and overhauled our AI architecture two times in the last year. Now, our agent and MCP have 6K+ daily active users.

Here are the golden rules of agent-first product engineering we learned along the way.

1. Let agents do everything users can

If a human can do something in your product, an agent should be able to do it as well.

For example, let’s say you ask an agent to set up an A/B test for a new pricing page.

The agent would use the PostHog MCP to create the feature flag, create the experiment, link it to the flag it just created, and set up the insight and metrics.

Now imagine if the experiment-create tool was missing because we just hadn’t prioritized it yet. The agent creates the flag and insight, but must stop and ask you to go to posthog.com, create the experiment yourself, and paste the experiment ID.

This is annoying and defeats the entire point.

The benefit of agents is that they reduce the time, attention, and expertise needed to complete a task. If your product doesn’t give agents the same capabilities as users, you’ll always be limited by the human in the loop.

Of course, there are situations where requiring human input makes sense, like if you’re dealing with sensitive data. But these should be deliberate exceptions, not accidental gaps.

What this looks like in practice

In practice, this means nearly everything in your API needs to be accessible to agents.

That doesn’t mean converting every endpoint into an MCP tool; we made that mistake in v1. (More on why that's bad in the next rule.)

Here’s how we do it now in v2:

1. Our pipeline autogenerates an OpenAPI spec from our typed Django endpoints.

2. It then converts them into TypeScript validation (Zod) schemas.

3. In parallel, product teams have to manually opt-in each endpoint via YAML config files. Nothing is exposed by default.

4. The pipeline then combines the Zod schemas + YAML configs to generate the final TypeScript tool handlers.

The result is a set of tool handlers – one file per product area1 – ready to go in our MCP server. Agents are able to do anything that a human can through the PostHog API, even if the tools don’t exactly match the endpoints 1:1.

2. Meet agents at their level of abstraction

To build an agent-first experience, you have to find the semantic layer where agents already reason best and meet them there.

This saves a ton of context (a precious resource) but the benefits are more than practical. They’re fundamental. The more “raw” your product’s agent interface is, the more creative potential you unlock.

Think about giving a child a Lego set for their birthday. If it comes with tires and axles already glued together, they’re going to make cars, trucks, and airplanes – vehicles that everyone has seen before.

If the parts come separated, they’re free to mix and match to build motorcycles, tire swings,2 snow sleds, and who knows what else?

What this looks like in practice

In the old version of our MCP, to answer “why did signups drop last week?”, an agent had to make four separate calls: projects-get, insight-get, and insight-query (twice).

get-insight or get-funnel may be meaningful to humans navigating the PostHog UI, but to an agent, it’s just an unnecessary translation step.

So in our MCP v2, we let agents query PostHog data in a language they already speak fluently: SQL.

By exposing our product at this layer of abstraction, we were able to turn off all our read/get endpoints off since they get subsumed by executeSql.ts.

Now, answering “why did signups drop last week?” can be answered with just a single, elegant query:

SELECT
    toStartOfWeek(timestamp) AS week,
    countIf(event = 'signed_up') AS signups
FROM events
WHERE timestamp >= now() - INTERVAL 2 WEEK
GROUP BY week
ORDER BY week

3. Front-load universal context

In the early days of AI, developers had to front-load everything into agents to compensate for smaller context windows and less capable models.

But as the technology keeps improving, there’s a new trend that removes as much context as possible and trusts the model to figure out the rest.

That works for general-purpose agents since they’re designed to be flexible. Anthropic can’t predict everything that everyone is using Claude for.

But if you’re building an agent experience for your specific product, the problem space is much smaller. You already know the key scenarios, tools, and use cases.

So for product engineers, you can strike that balance by front-loading what you know every session will need, and defer the rest.

What this looks like in practice

We didn’t think about it this way at first, so our v1 system prompt was four lines that essentially said “Here are some tools for using PostHog, GLHF.”

Every agent would waste time and tokens to rediscover the same things every time they connected.

In v2, we made use of the fact that anyone connecting to the PostHog MCP is there to query their PostHog data. That’s the whole point.

Now, we load these at the start of every PostHog MCP session:

• PostHog-specific taxonomy. What’s a feature flag, experiment, session replay, etc.

• Our SQL syntax. How to use our custom translation layer over ClickHouse SQL.

• Critical querying rules. Hard constraints that apply to every query (e.g., always filter by time range).

Everything else gets pulled later. We let the agent figure out when.

4. Writing skills is a human skill

Skills help you fill the gap between what your product can do and what an agent can do out of the box with your tools:

The biggest mistake people make is writing them like step-by-step manuals. If you’re too prescriptive, agents will follow your instructions too rigidly and lose the ability to improvise (see rule #3).

Instead, think of it like onboarding a new employee who’s already highly qualified.

A bad manager micromanages every process — do A, then B, never C, and always D. A good manager trusts the hire and only steps in with the specific things they couldn’t have known on their own.

That’s what good skills look like.

They should only contain context that a human can provide since an agent can’t discover by itself, such as:

• Idiosyncratic knowledge. Internal acronyms, naming conventions, and style guides.

• Edge cases. The uncommon places where things break, and how to handle them.

• Taste and craft. Not just how to use your product correctly, but how to use it well.

What this looks like in practice

To figure out what goes into a skill, ask yourself: what would an agent get wrong about your product without you?

For us, it’s in the connective tissue between our products, the domain knowledge buried in our data, and our niche developer takes. For Linear, it’s the opinionated issue hierarchy. For Figma, it’s their design system coherence.

As an example, here’s a line we added to query-retention.md:

For activation and retention events, use the $pageview event by default. Avoid infrequent or inconsistent events like signed in unless asked explicitly, as they skew the data.

Without this guidance, an agent would just use whatever event the user mentions, which is usually misleading. Retention would look worse than it actually is, and there’s no way a user would know unless they’d done this analysis themselves before.

By adding this line in the skill, we’re embedding a PostHog Certified™ opinion about what good metrics and analysis actually look like. This ensures agents use our product correctly, and users aren’t inadvertently misled.

5. Treat agents like real users

In traditional software, even if user behavior isn’t predictable, the code is. With AI, you lose that stability; the same inputs won’t produce the same outputs anymore.

This means classic testing methods will leave a gap, so you need new ways to catch the things automated tests no longer will.

You need to treat the agent like you would treat a user. Talk to them, build empathy for them, and develop an intuition of what they want.

This helps you see your product like they would and get familiar with all the interaction patterns and quirks they have.

What this looks like in practice

Here are a few habits and behaviors we adopt at PostHog to achieve this:

• Dogfooding headlessly. When testing our agent features, we reach for the CLI before the UI. This puts us in the same environment the agent operates in and exposes us to the same types of errors, syntax, and friction they’d experience. It’s how we caught an issue internally and found that our MCP tool descriptions were eating up way more tokens than it should.

• Doing manual trace reviews. We hold a weekly “traces hour” where we go through real user sessions that have user feedback ratings. For example, we found a case where PostHog AI confidently told a user that feature flags don’t support scheduled releases and then backtracked. Automated tests wouldn’t have caught that since the agent did respond; the response was just incorrect.
  

• Feeding our intuition into a loop. We then amplify the value of those manual reviews by building evals based on what our human eyes caught – both the good and the bad. Once, we found a session where PostHog AI correctly intervened when it spotted a weird data pattern that the user hadn’t noticed. We turned that into an eval case so that future model or prompt changes don’t regress the good behaviors we want to keep.

Words by Jina Yoon who hereby declares herself agentpilled.

📜 Related texts

• You Need to Rewrite Your CLI for AI Agents – Justin Poehnelt

• Building MCP servers in the real world – Gergely Orosz and Elin Nilsson (The Pragmatic Engineer)

• Block’s Playbook for Designing MCP Servers – Salman Mohammed and Kalvin Chau (Block)

• Patterns for Agentic Tools – Arcade

• I hate the standup bot (so I built an agent to do it for me) – Andy Maguire
  

🦔 Join the order

• Product Engineer

• Product Manager (ex-founder or ex-product engineer)

• Technical Account Manager

• Security Engineer

• AI Product Engineer
  

We had this thought two years ago, released an “AI product assistant” 6 months later, iterated, and then relaunched as PostHog AI in November last year.

We learned a lot along the way that we wished we knew when we started. With that knowledge, we could have…

1. Launched months earlier.

2. Provided a better early experience to users.

3. Made progress on capabilities faster.

To help you build a better agent faster, we’re sharing what we wished we had known about building AI agents.

1. Should you build an MCP server instead?

Should you even build an AI agent into your product? The capabilities of agents are unquestionably valuable but this does not mean you need to build a custom one. Making your product accessible to agents is often a better option.

Could your agent capabilities just be an MCP server? These are simpler to build and require less maintenance, and for us, is used as much as our built-in agent, PostHog AI. For example, 34% of dashboards created by AI were done through our MCP server (18% of all dashboards created).

MCP servers are great if you expect your users to be developers or people combining your product with others in their ✨agentic workflows✨. Just as importantly, MCP servers validate demand for further agentic capabilities.

Custom agents are better when your users are non-engineers, when compliance blocks external agents, or when you need full control of the user experience (for example, custom image generation).

Even when you decide to go custom, consider if simpler alternatives fit your use case:

• For one-shot Q&A like generating product descriptions, try a simple LLM call with good prompting

• For a single task like SQL or code generation, use a specialized model with structured output

• For a multi-step but predictable flow like email classification plus response drafting, create a hardcoded workflow with LLM steps

Although PostHog AI is an agent now, we validated demand for it with a simpler use case and system. We built a workflow for data-related questions like “How many people signed up last week?” which lead directly to insight generation.

Only once this workflow saw adoption and users demanded other use cases (like answering docs questions or creating feature flags) did the team build an agent.

2. Your harness is not your moat

Once you decide to build an agent, the next question is how will it work. The answer to this is an agent harness: the code and infrastructure that combines with a model to help it understand and use your product.

You aren’t going to win because you’ve created some genius new harness, especially if you’re building your first AI agent. Don’t use innovation points here. Anthropic wrote a perfectly good SDK, and both Anthropic and AmpCode have helpful guides.

If we were restarting today, we would have skipped building a custom harness and made MCP the canonical interface. We learned this the hard way through three iterations of PostHog AI this year.

The first harness was a coordinator that routed user messages to specialized sub-agents, but this created a black box problem as the coordinator couldn’t see what sub-agents were doing leading to context loss and confusion.

The second was a single agent loop with self contained modes and tools within each mode. This solved the visibility issue, but didn’t scale. If we didn’t write a tool, the capability wouldn’t exist. This led us to spending a lot of time writing tools (44 by the end) rather than improving in other ways.

Our third and current harness uses the Claude Agent SDK with MCP tools and skills. This change came from two realizations:

1. Our agent could be more creative. The Claude Agent SDK gives PostHog AI access to a code-execution sandbox. The agent can use it to query with SQL and run custom scripts, unlocking new capabilities without us needing to build them.

2. Agents are a primary persona. Our “users” are increasingly agents, whether that is through PostHog AI or our MCP server. Our work here should converge and support both using a single architecture.

The flexibility of the sandbox plus the scalability of MCP tools and skill standards (which I’ll explain more about later) solves the issues of the second harness and positions us better for an agent-first future.

3. Your context is your advantage

When building an agent, there’s ultimately one question you need to answer to succeed: How does this beat Claude?

Context is the most important part of an answer. The combination of your app’s functionality and user data create a unique blend no other product can match.

Although you have product and user context sitting around, you need structure and format it to be useful for the agent. We do this with:

1. MCP tools. Capabilities selected and extracted from the PostHog API to expose all possible actions that agents can perform in PostHog. Examples include list feature flags, create a survey, or execute an SQL query.

2. Skills. Markdown files we write to teach the agent complete workflows and how to use our product. These include query patterns, system table schemas, examples, and MCP tool references. They are templates rendered at build time, so they can pull live context from the codebase like model schemas.

3. Layered runtime context injection. The frontend sends the user’s current view and state such as dashboards, insights, session recordings, and filters. This is enriched with actual queries results and project metadata like timezone and organization name.

4. A taxonomy tool. This lets the agent explore the user’s event names, properties, and property values on demand rather than stuffing them all into the prompt upfront.

5. A memory onboarding flow. This collects company and product context through conversation and persists it across sessions, so the agent builds understanding of each user’s setup over time.

Without this structured product context, the agent struggles to understand both our product and the user’s goals. It can get lost on simple questions because the context behind those questions is complex.

Reality has a surprising amount of detail and people define tasks in ambiguous ways. How would you answer “where do users drop off in CFMP1 conversion” without context?

4. Set up observability and evaluation from day one

These are essential as AI agents are non-deterministic and can fail in unpredictable ways. Without visibility into this, you can’t improve them or catch regressions.

We didn’t have observability and evaluation early and regretted it. We wish we had:

• Tracing for every LLM call with inputs, outputs, latency, and cost

• Trace IDs that span the full conversation

• The ability to replay and debug specific interactions

• A curated datasets of real user queries

• Automated scorers like LLM-as-judge and deterministic checks

Basically, LLM analytics.

Unfortunately, LLM analytics alone isn’t enough. Reality is gnarly and to deal with it, our team often looks at real usage.

They run a “traces hour,” where they meet, manually analyze LLM traces (AKA real user interactions), and find areas to improve. Evals make the most sense when they stem from these investigations.

Like building any successful product, understanding a user’s experience is critical to building a successful agent.

5. It doesn’t matter how cool your capabilities are

After getting buried in technical details, it’s important to poke your head out and remember who you’re building for. You’re not building for some hypothetical “coolest AI agent” contest, you’re building for your users.

What users want is often at odds with what’s coolest. The most common pain points users faced weren’t our agent’s range of capabilities or functionality, but things like:

• Inconsistent performance

• Unexpected failures

• Generic error messages without clear explanation

• Unclear capabilities leading to complex queries failing

• Lack of signs of uncertainty, source of insights, and progress

While new capabilities are cool, ensuring your agent actually solves your customer’s problems is even cooler. Ultimately, building an AI agent is not just a showcase of your AI skillz, but a product engineering problem. This requires you to talk to users, ship features they want, and iterate.

Words by Ian Vanagas who has plenty of agentic capabilities himself if you ask nicely.

🦔 Jobs at PostHog

Unlike many AI companies, we don’t think agents are going to take over every job in the next 6 months. In fact, we’re hiring!

• Product Engineer

• Security Engineer

• FinOps Engineer

• Technical Account Manager

• Forward Deployed Engineer

📚 More good reads

• How we built automatic clustering for LLM traces – Andy Maguire

• MCP is Dead; Long Live MCP! – Charles Chen

• Production Telemetry Is the Spec That Survived – Vidhya R.

• You Need to Rewrite Your CLI for AI Agents – Justin Poehnelt

• 8 learnings from 1 year of agents – Michael Matloka

We’ve seen this first-hand at PostHog, but it’s not an isolated trend. The lines between product management and engineering are blurring, too.1

Thanks to LLMs, figuring out what to build is now a greater bottleneck than how to build it, and it’s forcing engineers to think more like product managers (PMs).

In this issue, we’ll help you take advantage of what product managers have already figured out by going over the top three skills in their playbook.

1. Gather the right context

Providing context is a product manager’s most important job.

What exactly do we mean by context? In the same way that sailors used to navigate by the stars, engineers depend on context to make the right product decisions.

Good context points teams in the right direction; great context can change a company’s entire trajectory.

Take Duolingo for example.

In a guest edition of Lenny’s Newsletter, Duolingo’s Head of Product, Jorge Mazal, recounted how the company fixed its stalling growth problem. The team initially focused on friend referrals, premium trials, and in-game mechanics. Nothing worked, and daily active users (DAU) continued to decline.

But then Mazal and the data science team discovered that current user retention rates (CURR) were 5x more impactful on DAU than any other projected metric. That single piece of context completely shifted the product roadmap.

Instead of trying to acquire new users, the team invested in keeping existing learners hooked with leaderboards, daily streaks, and – of course – passive-aggressive push notifications from everybody’s favorite lime green owl.

Four years later, Duolingo successfully exited with massive 4.5x DAU gainz.

Product metrics like CURR and DAU in Duolingo’s story are just one form of context. There are many more, such as:

• Competitor landscape. What are our competitors doing, and should we do that, too? In an interview with TechCrunch, Instagram’s CEO described how the company copied and outperformed Snapchat Stories in 2016.

• User research. What do prospective users say they want? Buffer interviewed 30+ content creators to discover what people actually want from creative AI assistants.

• Industry news. What’s happening in the industry that we should pay attention to? When generative AI took off in 2023, Canva turned it into its core feature and doubled revenue in two years ($1.7B → $3.3B).

• Customer feedback. What do current customers say they want, and is it actually what they want? When Linear users requested “custom fields”, they dug deeper and built Customer Requests to track support tickets, Slack messages, and calls.

Gathering context can be difficult since it’s not always obvious what information is useful at any given point in time.2 You could spend weeks analyzing bug reports to improve feature adoption, only to realize after one user interview that it was a discoverability problem all along.

The good news is that you can develop an intuition for this by tracking what types of context led to measurable success through accountability loops (we’ll cover what that looks like in the next section).

Takeaway for engineers

Don’t wait around for someone else to give you the right context for your product. Instead, generate it yourself by creating your own systems for discovery.

In addition to helping you ship faster, this gives you an unfiltered view of the data. For example, talking to users provides clearer insight than reading secondhand notes from your PM since information gets lost at each step along the way:

Another reason engineers should do this is that your teammates might not understand the technical options and constraints as well as you do. Sometimes only the person with the full context of a problem can ask the right questions.

Try this: Book 2 user interviews per week. Run a feedback survey for your product and review survey data every sprint. Set up an LLM eval that detects when users struggle. Create a weekly email digest for competitor blogs.

2. Track success with feedback loops

All that context is useless if your team doesn’t know if they’re winning.

Product managers at PostHog track and ensure success3 by running growth reviews: monthly meetings where they go over revenue trends, user feedback, usage metrics, and quarterly goals with the engineers building the product.

If the numbers aren’t moving in the right direction, the PM presents a well-researched explanation to the team so that they can come up with an informed hypothesis and solution:

When the team meets again in the following month, they’ll review what worked and what didn’t. Each iteration levels up the product – and the team’s product sense, too.

Here’s an example from our Error Tracking team:

1. The problem: Cory, the PM, noticed that churn rates were disproportionately high, even though new customer acquisition was strong.

2. The context: A series of customer interviews all pointed to the same theme: trust. Users were leaving because of rough edges, such as missing recordings or unhandled stack trace edge cases.

3. The hypothesis: This is a product-quality problem, not a “missing features” problem. Systematically improving papercuts, ergonomics, and reliability will reduce churn.

4. The solution: The team catalogued every trust-related issue from customer feedback and shipped dozens of fixes (see below). Churn improved from 21% to 10% in the following quarter.

Without Cory’s investigation, the team could have easily spent months on new features that they thought would solve the churn problem, without addressing the root cause.

Feedback loops like these create accountability and, in turn, give developers more autonomy. Engineers at PostHog can make product decisions more freely (without PMs micromanaging roadmaps) since the system ensures they’re always contributing to the company’s bottom line.

Takeaway for engineers

You can turn every sprint into a mini growth review by setting aside time to define and evaluate success in each session.

We do this at PostHog by classifying each milestone with one of the following:

• Nailed it: We hit the goal spectacularly. High fives all round.

• Scraped it: We almost hit the goal, but we need to do a bit more to tidy up.

• Failed it: We were nowhere near hitting the goal, but we learned some valuable lessons. Back to the drawing board for this one.

This ensures we’re learning from each cycle and not just shipping mindlessly. It’s way more useful than just marking tasks as “Completed”.

Try this: In your next sprint, apply the concept of feedback loops by defining and evaluating success for each task. Then in the next sprint, review if you met those goals (and if not, what you learned or how you’ll follow up).

You could run an A/B test to compare behavior between groups, test early prototypes to get user feedback, dogfood your product to get internal data, or message a customer to see if you addressed their issue.

3. Communicate towards action

To make all of the above actually work, you need to make communication actionable. We know this sounds like classic PM fluff, but give us a minute to explain.

It’s easy for anyone to state a problem:

“We’re losing enterprise deals to competitors because we’re missing features.”

But this doesn’t help engineers understand what to build or why.

Communicating towards action looks like:

“We’ve lost five deals worth $200k in the last three months over SSO, audit logs, and role-based permissions. We should prioritize these over shipping v2 of the MCP for our core segment since we already have strong retention there. The enterprise requests are more aligned with company-wide goals for Q2.”

This is better, but why? It reduces friction by surfacing important information from the start:

• Stating the impact. Not “we are losing deals”, but “we lost three enterprise contracts.”

• Being specific. They don’t just state that features are missing; they investigate and list the top three.

• Sharing relevant context. The revenue loss, strong core retention, and the Q2 company-wide goal are all factors that will help make the decision.

• Making trade-offs visible. The former doesn’t mention competing priorities; the latter points out exactly what to choose between.

• Having an opinion. Instead of just vaguely asking “Thoughts?”, they express which option they think is better and why.

This is where all three skills come together. Product managers know what information to provide thanks to all the context they’ve gathered and feedback loops they’ve experienced. And how they deliver it – by communicating actionably – makes all the difference.

Takeaway for engineers

Since action is the goal, it’s best to default to shipping when you can. That’s why pull requests are the S-tier form of communication at PostHog. (Email is F-tier, obviously.)

But when the next step is unclear and you need to bring it up for discussion, work backwards. Think about what types of context and success criteria a PM would provide to make the information as actionable as possible.

Try this: Next time you’re about to post “should we do X?”, make it actionable by including:

1. The specific problem and its impact

2. What you’ve already investigated

3. At least one other option and the trade-offs

4. Which one you’d pick and why

Words by Jina Yoon, reformed PM who once learned the dark arts and now shares the secrets with you.

📚 From the library

• Your Best Thinking Is Wasted on the Wrong Decisions – Ian Duncan

• Product Management is Communication – Ben Yoskovitz

• How close is AI to replacing product managers? – Lenny Rachitsky & Mike Taylor

• A GitHub Issue Title Compromised 4,000 Developer Machines – grith.ai

• How we built automatic clustering for LLM traces – Andy Maguire

🦔 Join the prickle

• Product Engineer

• Sales Lead

• Application Security Engineer

• Dev Advocate (YouTube Team)

• Mobile Engineer