Detecting Silent Vendor Script Changes in CI

A vendor can change your production JavaScript without a commit. Run a scheduled digest comparison so the change arrives as a pull request rather than as an incident.

Your repository is unchanged, your deploy log is empty, and the JavaScript executing on your production pages is different from what it was yesterday — because a vendor published a new build to a URL you reference.

Triage: Establish What You Would Currently Notice

Before building anything, establish the honest answer to “how would we find out”.

  1. Pick a vendor script your site loads and record its current digest.
  2. Ask what monitoring would fire if that file doubled in size, added a dependency, or began contacting a new origin.
  3. Check whether any existing alert is keyed on third-party bytes, on a new origin appearing, or on a digest.
  4. Check whether the vendor publishes a changelog, and whether anyone is subscribed to it.
What normally notices a vendor change Four candidate signals and whether each would detect a silent vendor change. A commit hook sees nothing, because there is no commit. A pull-request performance gate sees nothing, because no pull request exists. A production performance monitor may notice if the change is large enough to move a metric. Only a scheduled digest comparison detects the change itself, regardless of size. commit hooks nothing — there is no commit pull-request gates nothing — there is no pull request production perf monitoring sometimes, if the change is large enough to move a metric scheduled digest comparison always, and regardless of size Three of the four are keyed to your changes. Only the fourth is keyed to theirs.

Root Cause: Your Pipeline Watches Your Repository

Continuous integration is built around the assumption that changes arrive as commits. Every gate — tests, budgets, review — is triggered by a change to code you control, which is exactly right for the code you write and structurally blind to the code you merely reference.

A third-party script is a runtime dependency resolved on every page load. The vendor may change it at any time, for any reason, with no notification you are required to receive, and the change takes effect immediately for every visitor. There is no artefact in your repository that differs before and after, so nothing in a repository-triggered pipeline can fire.

The result is that the largest source of change to your production JavaScript is the one your engineering process does not model. A scheduled comparison closes that gap directly: it is the only mechanism whose trigger is time rather than your own activity, and time is the only thing correlated with a vendor’s release schedule.

Two change streams, one pipeline Two streams of change reach production. Your own commits pass through review, tests and gates before deploying. Vendor changes reach production directly, with no review, no test and no gate, because they arrive as a new response at an unchanged URL. A scheduled job is the only mechanism that can bring the second stream into the same process as the first. your commits review → tests → gates → deploy every step is triggered well covered vendor changes published → live no review, no test, no gate the larger stream, uncovered A scheduled digest job is the bridge: it converts a vendor change into a pull request.

Resolution: A Scheduled Job That Opens a Pull Request

// ci/check-vendor-digests.mjs — run on a schedule, not on push.
import { VENDORS } from '../src/vendors.js';
import { createHash } from 'node:crypto';
import { writeFileSync } from 'node:fs';

const drift = [];

for (const [name, v] of Object.entries(VENDORS)) {
  const res = await fetch(v.src, { cache: 'no-store' });
  if (!res.ok) { drift.push({ name, kind: 'unreachable', status: res.status }); continue; }

  const bytes = Buffer.from(await res.arrayBuffer());
  const actual = 'sha384-' + createHash('sha384').update(bytes).digest('base64');
  const expected = v.integrity[0];

  if (actual !== expected) {
    drift.push({ name, kind: 'changed', expected, actual, size: bytes.length });
  }
}

if (drift.length) {
  // Write a machine-readable report the workflow turns into a pull request.
  writeFileSync('vendor-drift.json', JSON.stringify(drift, null, 2));
  console.error(`${drift.length} vendor script(s) changed`);
  process.exit(1);
}
console.log('all vendor digests match');

The job should open a pull request rather than only alert. A report that says “the digest changed” leaves someone to find the new value, edit the registry and open the change; a job that does all three leaves a reviewer with a diff, which is a two-minute task rather than a context switch. That difference decides whether the check survives its third month.

Two refinements are worth adding once the basic job runs. Record the response size alongside the digest, so the pull request says whether the vendor’s bundle grew by 4 KB or 400 KB — a number that frames the review immediately. And treat an unreachable URL as a distinct outcome from a changed one, because a vendor outage and a vendor release want different responses.

Verification: Prove It Can Fail

Point the job at a deliberately rolling URL once and confirm it reports drift. A check that has never failed is a check nobody has verified, and a scheduled job that silently stopped running is indistinguishable from one that is passing.

Two properties to verify about the job itself Two checks on the monitoring rather than on the vendors. Pointing the job at a known-rolling URL must produce a drift report, proving it can detect a change. And a heartbeat — a recorded successful run — must exist, because a scheduled job that stops running produces exactly the same silence as one that is passing. can it fail? point it at a rolling URL drift reported proves detection is it running? record each successful run alert on absence silence is ambiguous neither verified a job that stopped looks exactly like passing The second is the one that catches a job disabled by an unrelated CI change six months ago.

What to Look at When It Fires

A drift report is not automatically a problem, and treating every one as an incident is how the check becomes noise. Most will be ordinary vendor releases, and the review is short: read the changelog if one exists, check the size delta, and accept.

Three signals justify more attention. A large size increase with no corresponding announcement is worth asking the vendor about, because it frequently means a new dependency was bundled. A new origin appearing in the diff — visible if you also record the origins the script references — is a change to your third-party surface that your inventory and CSP allowlist have not accounted for. And a change to a script that has been stable for a year is worth a closer read than one to a script that changes weekly.

Where the vendor publishes no changelog and the diff is minified beyond reading, that is itself a finding to record. It means you are accepting changes you cannot review, which is a legitimate thing to do deliberately and an uncomfortable thing to do by default — and it is a much easier conversation to have with a procurement team when you can show how often it happens.

Recording What Changed, Not Only That It Changed

A digest tells you a file is different; it does not tell you how. For a minified bundle the raw diff is unreadable, which is why most teams accept the change without looking and why the check gradually becomes a rubber stamp.

Two cheap additions restore some meaning. Recording the response size makes the magnitude visible, and a bundle that grew forty per cent overnight is worth a question even when the diff is unreadable. Running the script in a headless browser and recording the set of origins it contacts and the storage keys it writes turns the report into a behavioural comparison — “the file changed and it now contacts two new origins” is a sentence a reviewer can act on.

Neither requires understanding the code. They are observations about what the script does rather than what it says, and they are precisely the observations that matter for a third party: a new origin is a change to your data-sharing surface, and a new storage key is a change to what is kept on the visitor’s device. Both belong in the pull request the job opens, alongside the digest.

Common Pitfalls

  • Running on push instead of on a schedule. A push-triggered job checks the vendors whenever you change, which is uncorrelated with when they change.
  • Alerting without automating the fix. A notification that requires manual work to resolve will be muted within a quarter.
  • Fetching through a cache. A cached response reports yesterday’s bytes. Use no-store and, where possible, a region matching most of your traffic.
  • No heartbeat. A job that stops running is silent in exactly the same way as a job that is passing.

Frequently Asked Questions

How often should the job run?

Hourly is comfortable for most sites and daily is defensible. The cost is one request per vendor per run, which is negligible, so the constraint is how long you are willing to be unaware of a change rather than anything technical.

Consider running it more often for the vendors that matter most and less often for the rest. A payment or authentication SDK justifies a tight interval; a font service does not, and splitting the schedule keeps the noise proportionate to the risk.

What if the vendor genuinely changes every day?

Then the digest check is telling you something important rather than being noisy: you are executing code that changes daily with no review, on every page load. That may be an acceptable trade for the vendor in question, but it should be a recorded decision rather than an unexamined default.

The practical response is to ask for a versioned URL, which most vendors have and few document prominently. Failing that, record the vendor as unpinnable in the inventory and exclude it from the job explicitly — an exclusion with a reason is much better than a check that is muted because it fires too often.

Can the job detect behavioural changes rather than byte changes?

Not directly, and it does not need to. A behavioural change requires a byte change, so the digest catches every one of them — what it cannot do is tell you which byte changes matter, which is the reviewer’s job rather than the job’s.

Where you want more signal, the useful addition is recording what the script does rather than what it contains: load it in a headless browser and record the origins it contacts and the storage it writes. Diffing that alongside the digest turns “the file changed” into “the file changed and it now contacts a new origin”, which is a considerably more actionable sentence.

Up: Subresource Integrity and the Vendor Supply Chain

More in Third-Party Isolation