A release can leave behind a chart that shows a metric moved and none of the context needed for the next decision. The team shipped several changes around the same time. Nobody can tell which users received the feature, what happened inside the system, or whether those users completed the task they came to do.
I began using PostHog to preserve that context. I wanted each release to carry enough evidence for a later engineer to recover the decision, exposure, system behavior, and user outcome.
Analytics then begins with the decision the team expects to make. That decision defines the evidence. The evidence defines the events worth keeping.
Suppose a team changes onboarding. The next decision may be to expand the release, revise the flow, or remove the change. Define the success measure and guardrails before naming events. Success may mean that more users reach their first useful result. Guardrails may cover errors, support requests, latency, and abandonment later in the flow.
The event model can stay small:
- exposure to the new flow
- completion of each meaningful step
- validation or system failure
- the first useful product action
- a return within the relevant period
Each event is a contract between code and a question. It needs a stable name, clear property definitions, and an owner. Treat the schema with the same care as an API because future decisions will depend on it.
Ambiguous events create precise charts on weak foundations. Two engineers may read the same event name as different user actions. A property may change meaning during a refactor while old and new values continue to appear in one series.
Autocapture helps during exploration by revealing unexpected paths. Turn durable questions into explicit events tied to user intent and outcome.
PostHog keeps events, funnels, session replay, errors, and feature flag exposure in one product context. A funnel locates where behavior changed. Error data shows which failures occurred on that path. Session replay provides examples to inspect. Together, those signals shorten the investigation from a metric to a plausible cause.
I start with aggregate data to find a pattern, then use replay to inspect examples inside that pattern. A memorable session can dominate a discussion. Sampling and a clear question keep the investigation tied to the wider distribution.
Replay requires privacy rules from the start. Mask sensitive inputs, restrict access, set retention limits, and record which teams have a legitimate reason to view sessions. The capture policy should account for the most sensitive data the product may expose.
Deployment starts the release. Feature flags control exposure while the system produces evidence. Code can reach production behind a disabled flag. The team verifies the production path, opens the feature to a small cohort, and watches the success measure and guardrails. Each expansion has a named owner.
The release plan should define the first cohort, expected user outcome, conditions that pause expansion, rollback procedure, and owner of each decision. PostHog can connect flag exposure to the events used to judge the result, which helps separate the effect of concurrent changes.
Every release flag also needs an owner, review date, and cleanup issue. A temporary branch adds another product state to test, observe, and support. Remove the flag and obsolete path after the decision settles.
A fluent LLM response and a successful status code reveal little about quality. The model may choose the wrong tool, miss relevant context, loop through an expensive path, or produce an answer the user immediately corrects. Application monitoring captures the request, latency, and status. Product telemetry should also capture the model’s path and the user’s result.
PostHog’s AI observability records traces, generations, tool calls, latency, token use, and cost. Those signals explain how the system produced an answer.
Product events describe what followed. A user may accept the result, retry with a changed prompt, correct the output, complete the surrounding task, abandon the flow, or return later. Evaluate both levels together. A trace explains model behavior. Product events place that behavior inside a user outcome. Cost, speed, and model quality become useful when attached to task completion and retention.
Offline evaluations provide stable cases during development. Production traffic adds unclear requests, missing context, tool failures, and unexpected behavior. Move important production failures back into the evaluation set.
LLM telemetry can contain prompts, retrieved documents, tool results, and internal context. Use redaction, access rules, retention limits, and sampling from the beginning. Selective capture preserves useful evidence while limiting sensitive material.
PostHog’s open source core preserves a deployment choice. Its self-hosted deployment can help a team meet firm requirements for data location, access, or infrastructure ownership. Those requirements matter when product events and LLM traces contain sensitive context.
Self-hosting makes the team responsible for upgrades, security, backups, capacity, monitoring, and recovery. PostHog documents self-hosting as unsupported, and paid features remain in Cloud. For a small team, I would begin with Cloud and spend engineering time on the product. Firm data residency or infrastructure requirements may justify the operating cost later.
Event names become part of the product’s history.
I use lowercase snake case with a subject and completed action:
signup_completed, search_submitted, and report_exported.
The name records a fact.
Properties carry context such as entry point, experiment variant,
model, result count, and schema version.
Product language makes events understandable across engineering, product, and data. Interface details such as button color, component name, or screen position belong in properties when they matter to the question.
Keep a short contract for every durable event. Record its definition, firing point, owner, required properties, allowed values, and schema version. When the meaning changes, create a new event or add an explicit version so historical data keeps its original meaning.
PostHog is one implementation of the release evidence loop. The practice begins with a decision, defines stable evidence, controls exposure, connects system behavior to user outcomes, and limits the data captured. Each release then leaves a useful record of what happened and which decision should follow.