This guide teaches you how to build a complete, automated A/B testing workflow using PostHog — covering feature flag setup, experiment targeting, data collection, and automated statistical reporting. You can follow it end to end in approximately 90 minutes. It is written for developers and product teams at SMBs in Australia, Singapore, Canada, and the United States who want to run rigorous experiments without a dedicated data science team.

What You'll Build

  • A working PostHog experiment with feature flags wired into a Next.js 15 app
  • An automated event capture pipeline that feeds experiment data in real time
  • A statistical significance check that runs on a nightly schedule via GitHub Actions
  • A Slack notification that fires automatically when a winner is detected
  • A reusable workflow you can clone for every future A/B test

What You'll Need

  • Node.js 20+ and pnpm 9
  • A PostHog Cloud account (free tier covers up to 1 million events/month as of 2026)
  • A Next.js 15 project (App Router)
  • A GitHub repository with Actions enabled
  • A Slack workspace with an incoming webhook URL
  • Basic familiarity with React and environment variables

Step 1: Install and Configure PostHog in Your Next.js App

Install the PostHog JavaScript SDK. This single package handles feature flags, event capture, and session recording.

pnpm add posthog-js posthog-node

Create a PostHog provider so every page has access to the client. Place this file at app/providers/PostHogProvider.tsx.

'use client';
import posthog from 'posthog-js';
import { PostHogProvider as PHProvider } from 'posthog-js/react';
import { useEffect } from 'react';

export function PostHogProvider({ children }: { children: React.ReactNode }) {
  useEffect(() => {
    posthog.init(process.env.NEXT_PUBLIC_POSTHOG_KEY!, {
      api_host: process.env.NEXT_PUBLIC_POSTHOG_HOST ?? 'https://app.posthog.com',
      capture_pageview: false, // We control this manually
      persistence: 'localStorage',
    });
  }, []);

  return {children};
}

Wrap your root layout with the provider inside app/layout.tsx.

Why disable automatic pageview capture?

Next.js App Router handles navigation differently from traditional SPAs. Letting PostHog fire pageviews automatically causes duplicate events on soft navigations. You will fire posthog.capture('$pageview') manually inside a usePathname effect instead — this gives you accurate session data without double-counting.

Step 2: Create the Feature Flag in PostHog

Log in to your PostHog dashboard. Navigate to Feature Flags → New Feature Flag.

  1. Set the key to checkout-cta-copy.
  2. Choose Multiple variants.
  3. Add two variants: control (weight 50%) and treatment (weight 50%).
  4. Under Release conditions, target all users initially. You can narrow by cohort or property later.
  5. Save the flag but leave it disabled until Step 4.

What if you want to target a subset of users?

PostHog supports property-based rollout. Set a condition like country = AU or plan = pro to limit exposure. This is useful when you want to test a change on paying users before rolling it out broadly — a common pattern for SaaS teams in Singapore and Canada who have smaller active user bases.

Step 3: Implement the Experiment in Your Component

Use the useFeatureFlagVariantKey hook to branch your UI. This example tests two versions of a checkout button label.

'use client';
import { useFeatureFlagVariantKey } from 'posthog-js/react';
import { useEffect } from 'react';
import posthog from 'posthog-js';

export function CheckoutButton() {
  const variant = useFeatureFlagVariantKey('checkout-cta-copy');

  useEffect(() => {
    if (variant) {
      posthog.capture('experiment_viewed', {
        experiment: 'checkout-cta-copy',
        variant,
      });
    }
  }, [variant]);

  const label =
    variant === 'treatment' ? 'Start Your Free Trial' : 'Get Started';

  function handleClick() {
    posthog.capture('checkout_clicked', {
      experiment: 'checkout-cta-copy',
      variant,
    });
  }

  return (
    
  );
}

Why capture experiment_viewed separately?

Tracking impression and conversion as separate events gives you an exposure rate. Without it, you cannot compute the true conversion rate — only the raw click count. PostHog experiments expect both events to calculate statistical significance correctly.

Common pitfall: Do not render the button conditionally on variant !== undefined. This causes layout shift on first paint and inflates your Core Web Vitals CLS score. Render a default label immediately and let it update when the flag resolves.

Step 4: Define the Experiment in PostHog

With events flowing, go to Experiments → New Experiment in PostHog.

  1. Link it to the checkout-cta-copy feature flag.
  2. Set the exposure event to experiment_viewed.
  3. Set the goal metric to checkout_clicked (unique users).
  4. Set the minimum detectable effect to 10% relative lift (a realistic baseline for CTA copy tests).
  5. PostHog will calculate the required sample size automatically — typically 2,000–5,000 exposures per variant for a 10% lift at 80% power.
  6. Enable the experiment. Then go back and enable the feature flag.

Step 5: Automate Statistical Significance Checks with GitHub Actions

PostHog exposes experiment results via its REST API. You will write a small Node.js script that fetches results and posts to Slack when significance is reached.

Create scripts/check-experiment.mjs:

import fetch from 'node-fetch';

const POSTHOG_API_KEY = process.env.POSTHOG_PERSONAL_API_KEY;
const PROJECT_ID = process.env.POSTHOG_PROJECT_ID;
const EXPERIMENT_ID = process.env.EXPERIMENT_ID;
const SLACK_WEBHOOK = process.env.SLACK_WEBHOOK_URL;
const SIGNIFICANCE_THRESHOLD = 0.95;

async function checkExperiment() {
  const res = await fetch(
    `https://app.posthog.com/api/projects/${PROJECT_ID}/experiments/${EXPERIMENT_ID}/results`,
    { headers: { Authorization: `Bearer ${POSTHOG_API_KEY}` } }
  );

  const data = await res.json();
  const significance = data?.result?.significance ?? 0;
  const winningVariant = data?.result?.winning_variant ?? 'unknown';

  console.log(`Significance: ${significance}, Winner: ${winningVariant}`);

  if (significance >= SIGNIFICANCE_THRESHOLD) {
    await fetch(SLACK_WEBHOOK, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        text: `✅ Experiment *checkout-cta-copy* reached significance (${(significance * 100).toFixed(1)}%). Winning variant: *${winningVariant}*. Review and ship it.`,
      }),
    });
    console.log('Slack notification sent.');
  } else {
    console.log('Not yet significant. Skipping notification.');
  }
}

checkExperiment();

Now create the GitHub Actions workflow at .github/workflows/experiment-check.yml:

name: Check Experiment Significance

on:
  schedule:
    - cron: '0 8 * * *'   # Runs daily at 08:00 UTC
  workflow_dispatch:       # Allows manual trigger

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: node scripts/check-experiment.mjs
        env:
          POSTHOG_PERSONAL_API_KEY: ${{ secrets.POSTHOG_PERSONAL_API_KEY }}
          POSTHOG_PROJECT_ID: ${{ secrets.POSTHOG_PROJECT_ID }}
          EXPERIMENT_ID: ${{ secrets.EXPERIMENT_ID }}
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Add all four secrets to your GitHub repository under Settings → Secrets and variables → Actions.

When should you skip the nightly check?

Skip it in the first seven days of any experiment. Checking for significance too early causes peeking bias — a statistical error where you stop a test prematurely because of random early fluctuations. Most e-commerce teams in Australia and the US run experiments for a minimum of two full business cycles (usually 14 days) before acting on results.

Step 6: Add a Server-Side Variant Evaluation for SEO Pages

Client-side flag resolution causes a flash of the default content. For SEO-critical pages, evaluate the flag server-side using the posthog-node SDK.