Device & Network Emulation Weighting #

Emulation weighting transforms synthetic performance testing from isolated lab runs into deterministic CI gates. By mapping synthetic device and network profiles to real-user traffic distributions, teams enforce performance budgets that reflect actual user impact. This methodology directly feeds into the broader Threshold Calibration & Baseline Management strategy for deterministic CI gating, ensuring that pull requests only merge when weighted metric aggregates stay within approved limits.

Architecture of the Weighting Matrix #

Construct a deterministic profile matrix that maps Lighthouse/WebPageTest presets to observed RUM traffic. The matrix must normalize to 1.0 and align with CI runner concurrency limits.

Implementation steps:

• Extract RUM traffic distribution by device class (mobile, tablet, desktop) and connection type (4G, 3G, slow-4G).
• Map distributions to standard emulation presets (moto-g4, desktop-chrome, 3G-fast, 4G-good).
• Normalize all weights to sum exactly to 1.0.
• Validate total concurrent runs against CI runner capacity constraints.

weighting-matrix.json

{
  "profiles": [
    {
      "id": "mobile-3g",
      "device": "moto-g4",
      "network": "3G-fast",
      "weight": 0.45
    },
    {
      "id": "mobile-4g",
      "device": "moto-g4",
      "network": "4G-good",
      "weight": 0.25
    },
    {
      "id": "desktop-4g",
      "device": "desktop-chrome",
      "network": "4G-good",
      "weight": 0.2
    },
    {
      "id": "desktop-eth",
      "device": "desktop-chrome",
      "network": "desktop",
      "weight": 0.1
    }
  ],
  "metadata": {
    "version": "1.2.0",
    "normalized_sum": 1.0,
    "source": "rum_traffic_q3_2024"
  }
}

lighthouse-emulation-config.js

const matrix = require("./weighting-matrix.json");

module.exports = matrix.profiles.map((p) => ({
  id: p.id,
  lighthouseFlags: {
    chromeFlags: ["--headless", "--no-sandbox"],
    formFactor: p.device.includes("desktop") ? "desktop" : "mobile",
    throttlingMethod: "simulate",
    throttling: {
      rttMs: p.network === "3G-fast" ? 150 : p.network === "4G-good" ? 40 : 0,
      throughputKbps:
        p.network === "3G-fast"
          ? 1600
          : p.network === "4G-good"
            ? 9000
            : 10000000,
      cpuSlowdownMultiplier: p.device.includes("desktop") ? 1 : 4,
    },
  },
}));

Step-by-Step CI Execution & Aggregation Pipeline #

Execute parallel emulation runs using a matrix strategy. Aggregate results deterministically before applying CI gates.

ci-pipeline.yml (GitHub Actions)

name: Weighted Performance Gate
on: [pull_request]
jobs:
  emulate:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        profile: [mobile-3g, mobile-4g, desktop-4g, desktop-eth]
      fail-fast: false
    steps:
      - uses: actions/checkout@v4
      - name: Run Lighthouse
        run: npx lighthouse https://staging.example.com --config-path=./lighthouse-emulation-config.js --output=json --output-path=./results/${{ matrix.profile }}.json
      - uses: actions/upload-artifact@v4
        with:
          name: lh-${{ matrix.profile }}
          path: ./results/${{ matrix.profile }}.json

  aggregate:
    needs: emulate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          path: ./results
      - name: Weighted Aggregation
        run: node ./scripts/aggregate-weights.js

aggregate-weights.js

const fs = require("fs");
const path = require("path");
const matrix = require("../weighting-matrix.json");

const resultsDir = "./results";
let weightedLCP = 0,
  weightedINP = 0,
  weightedCLS = 0;

matrix.profiles.forEach((p) => {
  const filePath = path.join(resultsDir, `lh-${p.id}.json`);
  const report = JSON.parse(fs.readFileSync(filePath, "utf8"));
  const { lcp, inp, cls } = report.audits;

  weightedLCP += lcp.numericValue * p.weight;
  weightedINP += inp.numericValue * p.weight;
  weightedCLS += cls.numericValue * p.weight;
});

console.log(`Weighted LCP: ${weightedLCP.toFixed(2)}ms`);
console.log(`Weighted INP: ${weightedINP.toFixed(2)}ms`);
console.log(`Weighted CLS: ${weightedCLS.toFixed(4)}`);

if (weightedLCP > 2500 || weightedINP > 200 || weightedCLS > 0.1) {
  process.exit(1);
}

Handling Synthetic Variance & Flakiness #

Synthetic environments introduce metric variance that skews weighted aggregates. Outlier filtering must precede weight application per established Statistical Noise & Flakiness Reduction protocols.

Implementation steps:

• Run three baseline iterations per profile to establish variance bounds.
• Apply a median filter to LCP, INP, and CLS values before multiplying by profile weights.
• Configure CI to retry any single profile run exceeding 15% standard deviation from the median.
• Log variance deltas to a structured output file for QA review and pipeline auditing.

Calculating Weighted Thresholds for CI Gating #

Convert raw synthetic metrics into a single pass/fail gate using weighted aggregation. The formula ensures high-traffic profiles dominate the CI decision.

Formula: Weighted_Metric = Σ(Profile_Weight * Metric_Value)

Implementation steps:

• Define base metric targets per profile (e.g., Mobile-3G LCP ≤ 2.8s, Desktop-4G LCP ≤ 1.5s).
• Apply weight multipliers to each profile’s median metric value.
• Compute the aggregate weighted score across all active profiles.
• Set CI exit codes based on weighted thresholds, aligning with Percentile-Based Threshold Tuning for accurate user-impact modeling.

Example threshold logic:

const THRESHOLDS = { LCP: 2500, INP: 200, CLS: 0.1 };
const weightedScores = { LCP: 2340, INP: 185, CLS: 0.085 };

Object.keys(THRESHOLDS).forEach((metric) => {
  if (weightedScores[metric] > THRESHOLDS[metric]) {
    console.error(`CI Gate Failed: ${metric} exceeded threshold`);
    process.exit(1);
  }
});

Advanced Configuration: Dynamic Weight Adjustment #

Static matrices degrade as traffic patterns shift. Implement environment-driven overrides to adjust weights dynamically without pipeline redeployment.

Implementation steps:

• Inject traffic-shape JSON via CI environment variables (TRAFFIC_MATRIX_OVERRIDE).
• Configure fallback to the static weighting-matrix.json on fetch failure or parse error.
• Validate weight normalization in a pre-flight script before spawning parallel jobs.
• Enable conditional gating per deployment environment (e.g., stricter weights for production, relaxed for staging).
• Transition to regional traffic shaping, demonstrating how geo-specific routing overrides integrate with Weighting Budgets by User Geography for localized budget enforcement.

dynamic-weight-loader.js

const fs = require("fs");
const defaultMatrix = require("./weighting-matrix.json");

function loadMatrix() {
  try {
    const override = process.env.TRAFFIC_MATRIX_OVERRIDE;
    if (!override) return defaultMatrix;
    const parsed = JSON.parse(override);
    const sum = parsed.profiles.reduce((acc, p) => acc + p.weight, 0);
    if (Math.abs(sum - 1.0) > 0.001)
      throw new Error("Weights do not normalize to 1.0");
    return parsed;
  } catch (err) {
    console.warn(
      "Dynamic matrix load failed, falling back to static:",
      err.message,
    );
    return defaultMatrix;
  }
}

module.exports = loadMatrix();

QA Validation & Rollout Checklist #

Enforce strict validation before merging weighted gating into production pipelines.

Implementation steps:

• Execute a dry-run against the main branch with CI_DRY_RUN=true to verify artifact collection and aggregation logic.
• Compare weighted scores against historical baselines to detect threshold drift.
• Validate CI gating exit codes (0 for pass, 1 for fail) under simulated budget breaches.
• Document weight matrix versioning and attach RUM data snapshots to the PR.
• Enable production enforcement only after QA sign-off and a 7-day shadow mode period.