Generating a Third-Party Inventory from HAR Files

A HAR capture is a complete record of one page load. Turn a set of them into a first inventory in an afternoon, and know exactly what that inventory cannot see.

You need a third-party inventory, you have no field collection in place, and you would like something usable today rather than after a telemetry project.

Triage: Decide Which Loads to Capture

A HAR is a faithful record of one page load in one browser under one set of conditions, so the value of the exercise is decided entirely by which loads you capture.

  1. List your distinct page templates — home, article, product, checkout, account — rather than individual URLs.
  2. For each, decide whether an authenticated capture is needed. Logged-in pages routinely load vendors anonymous ones do not.
  3. Decide the consent states to capture: untouched, accepted, declined. These are three different pages.
  4. Note the browser and region you are capturing from, because both change what loads.
What a capture set has to cover Three dimensions multiply into the set of captures needed. Page templates such as home, article and checkout each load different vendors. Consent state — untouched, accepted and declined — changes which vendors run at all. And session state, anonymous or authenticated, changes which templates are reachable and which vendors they include. Capturing one URL in one state produces an inventory that is confidently incomplete. templates home, article, product checkout, account different vendors each consent state untouched accepted declined — three pages session state anonymous authenticated different templates One capture of one URL is a sample of one. The interesting vendors live in the combinations.

Root Cause: You Cannot Enumerate From Source

The reason to capture traffic rather than read code is that a third-party surface is not fully described anywhere in your repository. Tags arrive through a tag manager without a commit; vendors load their own dependencies at runtime; and a script added for a campaign three years ago is still in a template nobody reads.

A HAR records what actually happened: every request, its initiator, its size, its timing and its headers. That makes it a complete record of one load — which is both its strength and its boundary. It sees everything that occurred and nothing that did not, so a vendor that only loads for visitors in one region, or after a particular interaction, is simply absent rather than reported as absent.

The practical consequence is that a HAR-derived inventory is a floor rather than a total. It is an excellent floor — it will contain vendors nobody remembered and dependencies nobody knew about — and stating it as a floor is what keeps it honest when someone later asks whether the list is complete.

What the initiator chain reveals A HAR entry records how each request was initiated. A vendor script requested by the document is one you declared. A vendor script whose initiator is another vendor script is a dependency you did not declare and probably do not know about. A request with a script initiator inside a tag container is a tag someone added without a commit. The initiator field is what turns a list of hosts into a map of responsibility. initiator: the document you declared this one — it is in your markup initiator: a tag container added without a commit; check who has console access initiator: another vendor a dependency you did not choose and cannot see in source The host list is the obvious output. The initiator chain is the useful one.

Resolution: Capture, Parse, Group

Capture from DevTools → Network → Export HAR, with the cache disabled and a fresh profile for each consent state. Then parse:

// har-to-inventory.mjs — one HAR in, grouped third-party rows out.
import { readFileSync } from 'node:fs';

const FIRST_PARTY = 'www.example.com';

export function inventory(harPath) {
  const har = JSON.parse(readFileSync(harPath, 'utf8'));
  const rows = new Map();

  for (const e of har.log.entries) {
    const url = new URL(e.request.url);
    if (url.host === FIRST_PARTY) continue;

    // Registered domain, so CDN shards collapse into one vendor row.
    const host = url.host.split('.').slice(-2).join('.');
    const row = rows.get(host) ?? { host, requests: 0, bytes: 0, initiators: new Set() };
    row.requests += 1;
    // _transferSize is a Chromium extension to the HAR format; bodySize is the fallback.
    row.bytes += e.response._transferSize ?? Math.max(e.response.bodySize, 0);
    row.initiators.add(e._initiator?.url ? new URL(e._initiator.url).host : 'document');
    rows.set(host, row);
  }

  return [...rows.values()]
    .map((r) => ({ ...r, initiators: [...r.initiators] }))
    .sort((a, b) => b.bytes - a.bytes);
}

Grouping by registered domain rather than by full host is what makes the output readable: a vendor serving from four CDN shards should be one row, not four, and the raw host list is dominated by shards on most sites.

Merge the per-capture results by taking the union of hosts and the maximum observed size per host, and record which captures each host appeared in. That last field is the most interesting column in the finished table — a vendor that appears only in the accepted-consent capture is correctly gated, and one that appears in the declined capture is a finding.

Verification: The Declined Capture Is the Test

Compare the three consent captures. The declined capture should contain only vendors you have classified as strictly necessary; anything else present in it is running without permission. The accepted capture minus the declined capture is your gated surface, and it should match your consent configuration exactly.

Three captures, two comparisons The three consent captures produce two useful comparisons. The declined capture should contain only strictly necessary vendors, so anything else in it is a gating failure. The accepted capture minus the declined one is the set of gated vendors, and it should match the consent configuration exactly — a vendor in one and not the other is a configuration drift. declined capture necessary vendors only anything else is a leak accepted minus declined the gated surface should match the config in config, in neither a tag that never fires dead configuration The third column is worth cleaning up: dead tags accumulate and confuse every later audit.

Turning a One-Off Into a Baseline

The afternoon’s work produces a snapshot, and a snapshot decays. Committing the generated inventory to the repository turns it into a baseline that later captures can be diffed against, which is what converts the exercise from an audit into a check.

The diff is where the ongoing value is. A new host appearing between one month’s capture and the next is a change to your third-party surface that nothing else would report, and because the baseline is a committed file, the change arrives as a pull request with an author and a reviewer rather than as a line in a report. That is the same mechanism the drift detection relies on, reached from the opposite direction — starting with captures rather than with field collection.

Automating the capture is a smaller step than it looks. A headless browser can load the template list, exercise the consent states, and export a HAR for each in a scheduled job, which removes the manual work while keeping the output identical. That is worth doing once the manual version has demonstrated its value, and rarely worth doing before — an automated pipeline producing a report nobody has yet found useful is a common way to spend a fortnight.

Common Pitfalls

  • Capturing with the cache warm. Cached resources report zero size, so a warm capture understates every vendor and omits some entirely.

  • Capturing one URL. A homepage is not representative of a checkout, and the vendors that matter most for compliance often live on the pages that handle data.

  • Ignoring the initiator field. Without it the output is a list of hosts; with it, the output distinguishes what you chose from what was chosen for you.

  • Treating the result as complete. It is a floor derived from the loads you captured, and saying so protects the credibility of everything else in the table.

  • Comparing captures taken on different networks. Transfer sizes are comparable across networks but timings are not, and a capture from a fast office connection will look unlike one from a home connection for reasons that have nothing to do with the vendors.

Frequently Asked Questions

Does a HAR contain anything sensitive?

Frequently, yes — request and response bodies, cookies, authorisation headers and query strings are all included by default, and an authenticated capture can contain session tokens and personal data. Treat a HAR from a real session as sensitive material rather than as a log file.

DevTools offers a sanitised export that omits sensitive content, and it is the right default for anything that will be shared or committed. Where you need the full capture for analysis, keep it out of the repository and delete it when the analysis is done; the inventory it produces is the artefact worth keeping.

How does this compare with field collection?

They answer different questions and the inventory wants both. A HAR set is exhaustive for the loads it covers and blind to everything else; field collection is partial per session but covers every cohort, region and experiment your real traffic contains. The union is closer to complete than either alone.

Start with HAR because it needs no deployment and produces something usable the same day. Add field collection once the inventory has proved its worth, and reconcile the two — the vendors that appear in field data and not in your captures are the ones your capture set is missing, which tells you which dimension to add.

Can we get the same data from Lighthouse instead?

Partly. The third-party summary audit gives you a grouped list with transfer sizes and blocking time, which is a faster route to a first ranking and requires no parsing. What it does not give you is the initiator chain, the per-consent-state comparison, or control over the capture conditions — and those are the three things that make the exercise more than a list.

Use it as a sanity check on your HAR output rather than as a replacement. The two agreeing is reassuring; a vendor in one and not the other usually means the two runs differed in a way worth understanding.

How often should the captures be repeated?

Quarterly by hand, or monthly once automated. The rate of change in a third-party surface is driven by how often anyone adds a tag, which for most organisations is a handful of times a year with clusters around campaigns and product launches — so a quarterly cadence catches most changes within a reasonable window.

Time the captures deliberately rather than at a fixed calendar point. Running one shortly after a major campaign launch or a marketing tooling change surfaces exactly the additions that were made without a commit, which is the population this technique exists to find.

Up: Auditing and Inventorying Third-Party Scripts

More in Performance Monitoring