Harden before Lockdown

[kriskowal]

Refs: #2983, #1686

Reviewer instructions

This change introduces two packages, with implementation factored out of ses and lightly modified. To highlight those modifications, the commits are individually reviewable. In separate commits, I have copied certain modules to new locations verbatim, then applied changes in place. For example, the ses/commons.js effectively moves out to another package, but parts stay in ses and other parts move to harden/commons.js since they are not otherwise shared. The harden tests are duplicated and lightly edited, since the intended behavior is similar before and after lockdown.

Description

This change introduces an @endo/harden package that provides the implementation of harden for ses but also a relaxed implementation of harden suitable before lockdown, for modules that should interoperate between HardenedJS and non-HardenedJS. The expectation is that we will use @endo/harden in most of Endo, like marshal and pass-style, to make CapTP usable with or without Lockdown.

Security Considerations

The harden provided pre-lockdown differs only in that it does not visit prototypes. The harden and intrinsics packages remain inside the SES TCB and are linted according to the same stricter restrictions for access to free variables and use of polymorphic calls.
Hardening before and after lockdown should converge on the same state.

Scaling Considerations

Hardening before and after lockdown should converge on the same state, but redundantly.
Hardening before lockdown and after lockdown use distinct tables for tracking prior work.

Documentation Considerations

The new modules have minimal README documentation. Further documentation for the harden package in particular should follow.

Testing Considerations

Tests preserved and copied with the appropriate variation for pre-lockdown.

Compatibility Considerations

Compatibility is preserved.

The harden and intrinsics packages may be versioned on an independent cadence of ses since they are not directly exposed. That is, intrinsics may make breaking changes to its API without breaking ses.

The intrinsics package may or may not be a suitable foundation for a future get-intrinsics shim which we expect to be entangled with the internals of the SES permits. Future designs will determine how permits and the get intrinsics shims get packaged and it is our hope that this structure does not impede progress in that direction.

Upgrade Considerations

None.

[kriskowal]

I should note that I am not hoping for timely review of these changes.

[erights]

What is the scope of "project"? Do you mean "package"?

[kriskowal]

Yes. I’ll fix uniformly in follow-up. This boilerplate is in every package.

[erights]

Suggested change

	If used in composition with Lockdown in [HardenedJS](https://hardenedjs.org),
	If used in composition with `lockdown` in [HardenedJS](https://hardenedjs.org),

[erights]

Suggested change

	behaviors transitively produces values that sare safe to share with guest code.
	behaviors transitively produces values that are safe to share with guest code.

[erights]

Suggested change

	freeze an object and its transitive properties, making them shallowly
	freeze an object and its transitive own properties, making them shallowly

Additional pedantic detail that can be omitted in a high level summary, but needs to be explained somewhere. It does a "transitive reflective" walk of properties. The difference is that avoids invoking getters. Rather it proceeds to walk both getters and setters as objects.

[erights]

We should cite Jordan's recent Error stack proposal.

[erights]

Nice that you pulled these out here. However, should the ses error handling also be pulled out into a separate low level module, they may need to coordinate here. Not sure.

[erights]

Separately, I have been thinking that once Immutable ArrayBuffers are in the language and adequately deployed, we should retire harden's special case for TypedArrays. If someone wants to harden a TypedArray, they should start with a frozen TypedArray.

[erights]

I should note that I am not hoping for timely review of these changes.

Well, you got one anyway. LGTM!

[kriskowal]

Agreement out-of-band from @mhofman: There is a hazard that an object hardened before lockdown might not be fully hardened under a lockdown regime. There are sophisticated and subtle ways to maintain this invariant in an application, but one simple way: if lockdown senses that a harden was used before lockdown, throw an error. There should only be two kinds of applications: those that use harden and never call lockdown and other applications that call lockdown before ever calling harden.

So, I can introduce a coarse mechanism to maintain the partition of lockdown and no-lockdown hardening (that can be frustrated by eval twins, but eval twins are simply frustrating). It involves a single bit of module scope state for which I hope to be forgiven.

[mhofman]

I also want to change the harden semantics to only verify that the prototype has been previously (or concurrently) hardened under lockdown. In non lockdown it would not check the prototype. This is a breaking change and I would prefer if it happened at the same time or before to avoid having to incur a future breaking change of the standalone harden (unless we reify this as a 3rd configuration state for the prototype traversal option)

[kriskowal]

I think that change can be made additive, even with the current framing. I don’t expose the makeHarden function directly to any consumer. If you import @endo/harden, you get a singleton with the pre-lockdown configuration, but which falls thru to globalThis.harden if present. I exposed a makeHarden variant for ses to import at @endo/harden/make-harden.js. This could be parameterized further, or a new method name could be added, for the new behavior. I take it that we will also want a lockdown option to migrate to the new behavior gracefully.

[gibson042]

The harden provided pre-lockdown differs only in that it does not visit prototypes.

🥳

The harden and intrinsics packages remain inside the SES TCB and are linted according to the same stricter restrictions for access to free variables and use of polymorphic calls.

Where do we document the scope of the SES TCB? Ideally, it should be trivial to look up and verify constraints on all transitive dependencies from the SES package itself, and also to identify any specific package as belonging to that collection and verify the presence of expected constraints. This is relevant for e.g. #2822, because the proposed cache-map is also in the trusted compute base.

Comments that must be resolved before merging

• suggestion: rename @endo/intrinsics to @endo/primordials

[gibson042]

https://bugs.webkit.org/show_bug.cgi?id=222538 was marked resolved in 2021; it's probably time to either drop the workaround or update its explanation.

[erights]

On this one, I'm suspicious. So we should test before we assume that they "resolved" it because they actually fixed it. A one-time manual test would be fine.

[gibson042]

Suggested change

	* See those reference for a complete explanation:
	* See this reference for a complete explanation:

[gibson042]

How much typing information do we want here? I ask because #2730 includes the Object.entries/Object.fromEntries analog to this, which could just be brought all the way down.

[gibson042]

Is this intentionally excluding strings from the domain of searchValue? If so, we should comment to that effect (e.g., "This typing intentionally constrains searchValue to the regular expression interface, because at call sites it is too easy to overlook that use of a string is limited to one replacement."). Or if not, we should fix the typing:

Suggested change

	/**
	* @type { &
	* ((that: string, searchValue: { [Symbol.replace](string: string, replaceValue: string): string; }, replaceValue: string) => string) &
	* ((that: string, searchValue: { [Symbol.replace](string: string, replacer: (substring: string, ...args: any[]) => string): string; }, replacer: (substring: string, ...args: any[]) => string) => string)
	* }
	*/
	export const stringReplace = /** @type {any} */ (
	uncurryThis(stringPrototype.replace)
	);
	/**
	* @type { &
	* ((that: string, searchValue: ({ [Symbol.replace](string: string, replaceValue: string): string; } | string), replaceValue: string) => string) &
	* ((that: string, searchValue: ({ [Symbol.replace](string: string, replacer: (substring: string, ...args: any[]) => string): string; } | string), replacer: (substring: string, ...args: any[]) => string) => string)
	* }
	*/
	export const stringReplace = /** @type {any} */ (
	uncurryThis(stringPrototype.replace)
	);

[kriskowal]

I’m choosing to walk past this since this is a refactor, and I don’t think we need a more expressive type since this is currently internal to ses and can be relaxed without breaking later.

[gibson042]

Minor suggestion:

Suggested change

	/**
	* @param {any} guard
	* @returns {asserts guard}
	*/
	const assert = guard => {
	if (!guard) {
	throw new TypeError('assertion failed');
	}
	};
	/** @type {(condition: any) => asserts condition} */
	const assert = condition => {
	if (!condition) {
	throw new TypeError('assertion failed');
	}
	};

[gibson042]

Interesting; I've always thought of prototype chains in the other direction (such that following them would be descent rather than ascent).

Suggested change

	* @param {boolean} ascendPrototypeChains
	* @param {boolean} traversePrototypeChains

[gibson042]

Suggested change

	// The harden implementation exported by @endo/harden does not ascend
	// prototype chains but may be imported into programs after lockdown.
	// In this case, the weak hardener must give way to the strong hardener
	// on the global object, lazily.
	// The harden implementation exported by @endo/harden does not traverse
	// prototype chains but may be imported into programs after lockdown, or
	// before lockdown for later use.
	// In these cases, the weak hardener must give way to the strong hardener
	// on the global object, and to support the latter
	// (`import { harden } from "@endo/harden"; import "@endo/init";`), it must do
	// so lazily.

[gibson042]

Suggestion: simplify.

Suggested change

	let { harden } = {
	/**
	* @template T
	* @param {T} root
	* @returns {T}
	*/
	harden(root) {
	/** @type {<T,>(root: T) => T} */
	let harden = root => {

[gibson042]

Suggestion: simplify.

Suggested change

	({ harden } = (innerHarden => ({
	/**
	* @template T
	* @param {T} root
	* @returns {T}
	*/
	harden(root) {
	// Use a native hardener if possible.
	if (typeof globalThis.harden === 'function') {
	const globalHarden = globalThis.harden;
	return globalHarden(root);
	} else {
	return innerHarden(root);
	}
	},
	}))(harden));
	const innerHarden = harden;
	harden = root => {
	// Use a global hardener if possible.
	const globalHarden = globalThis.harden;
	if (typeof globalHarden === 'function' && globalHarden !== harden) {
	return globalHarden(root);
	}
	return innerHarden(root);
	};

[gibson042]

But I must admit, this harden being vulnerable to future globalThis manipulation seems to fly in the face of our goals. What if it were instead a one-time sensitivity?

Suggested change

	({ harden } = (innerHarden => ({
	/**
	* @template T
	* @param {T} root
	* @returns {T}
	*/
	harden(root) {
	// Use a native hardener if possible.
	if (typeof globalThis.harden === 'function') {
	const globalHarden = globalThis.harden;
	return globalHarden(root);
	} else {
	return innerHarden(root);
	}
	},
	}))(harden));
	let innerHarden = harden;
	harden = root => {
	// Capture the global hardener (or lack thereof) on first invocation, and
	// use it if present.
	if (mode === undefined) {
	mode = ascendPrototypeChains;
	const globalHarden = globalThis.harden;
	if (typeof globalHarden === 'function' && globalHarden !== harden) {
	innerHarden = globalHarden;
	}
	}
	return innerHarden(root);
	};

or

Suggested change

	({ harden } = (innerHarden => ({
	/**
	* @template T
	* @param {T} root
	* @returns {T}
	*/
	harden(root) {
	// Use a native hardener if possible.
	if (typeof globalThis.harden === 'function') {
	const globalHarden = globalThis.harden;
	return globalHarden(root);
	} else {
	return innerHarden(root);
	}
	},
	}))(harden));
	let innerHarden = harden;
	let globalHarden;
	harden = root => {
	// Capture the global hardener (or lack thereof) on first invocation,
	// require it to remain constant, and use it if present.
	if (mode === undefined) {
	mode = ascendPrototypeChains;
	globalHarden = globalThis.harden;
	if (typeof globalHarden === 'function' && globalHarden !== harden) {
	innerHarden = globalHarden;
	}
	}
	if (globalThis.harden !== globalHarden) {
	throw new TypeError('a global hardener must not be replaced');
	}
	return innerHarden(root);
	};

[naugtur]

please release a version of SES before merging this

[kriskowal]

Toward reviving this stale branch, I had a conversation out-of-band with @mhofman that uncovered a hazard we want to address before merging this change as designed. That is, if a hardened module initializes before lockdown, it may be in a state where its prototypes are unfrozen until the first instance is hardened.

This is related to an orthogonal hazard of having modules that explicitly support both shallow hardening before lockdown and deep hardening after lockdown. We do not believe we can in good conscience relax post-lockdown hardening so that it throws an exception if it encounters an unhardened prototype instead of implicitly capturing it in the closure of to-be-hardened objects. This would create a hazard where a module uses shallow pre-lockdown harden for its own partial defense but is incompatible with a post-lockdown mode because it neglected to pre-harden its prototypes. As a condition of proceeding with this design @mhofman proposes that we add a lockdown mode that issues warnings when harden encounters an unhardened prototype, in order to softly encourage code to pre-harden prototypes and eliminate a window between creation and instantiation where an attacker can manipulate an exported prototype.

We currently have a mechanism in place that causes a pre-lockdown harden to throw an exception if it is called again post-lockdown, but should add a mechanism that causes repairIntrinsics to throw an exception if it is called after a pre-lockdown harden was called. (It occurs to me as I summarize this, that the latter mechanism may obviate the former.)

This new mechanism has some requirements:

• Older and newer versions of both @endo/harden and ses need to tolerate meeting in the wild.
• Multiple instances of @endo/harden must be able to coëxist in an application that never locks down. So, we can’t simply have every implementation call through to a global implementation, as that would invite cyclic recursion. We might be able to instead invite the winner of the race to detect that they won the race and use their implementation instead of the global.

We also wish to encourage code going forward to adopt @endo/harden as a veneer over the post-lockdown harden. To that end, in a distant future, the point of coördination between @endo/harden and lockdown may not be globalThis.harden but something else, tentatively the existence of a Object.@@harden property.

It occurs to me as I summarize that the simplest formulation of this problem is that, between eval twins of the pre-lockdown harden and the installation of a post-lockdown harden, we can and should only ever use one implementation of these. For pre-lockdown hardens, this is not a race to initialization, but a race to first-use, with the additional requirement that lockdown should fail if it loses the race.

@mhofman proposes a mechanism that will universally cause lockdown to fail if repairIntrinsics occurs after any pre-lockdown harden gets used: the installation of a non-writable non-configurable property on an intrinsic, e.g., Object.@@harden.

Proposed change

So, the proposed design is to have pre-lockdown hardens and hardenIntrinsics itself race to install a Object.@@harden and then defer to Object.@@harden going forward. They will need to fall back to globalThis.harden if it is present in order to function with versions of ses that predate the decision to install harden at Object.@@harden.

As a courtesy to developers, we add explicit code to repairIntrinsics that detects Object.@@harden and throws a more informative error if they are using a more recent version of ses than this change.

I believe we have the option of removing the trap that detects a change to globalThis.harden since we’ll throw earlier with repairIntrinsics.

[erights]

Just requesting changes in order to turn off my approval in light of upcoming big changes. I want to be sure to review again after these big changes but before merge. I do not currently have any changes in mind to request.

[gibson042]

For the record, I am of the opinion that the user-facing harden installed on the global should behave identically before and after lockdown, traversing prototypes but stopping when such traversal reaches a host-realm primordial—i.e., that it be independent of lockdown and therefore equally usable with or without it, including before it.

[kriskowal]

For the record, I am of the opinion that the user-facing harden installed on the global should behave identically before and after lockdown, traversing prototypes but stopping when such traversal reaches a host-realm primordial—i.e., that it be independent of lockdown and therefore equally usable with or without it, including before it.

This was actually how harden worked before my time at Agoric, and I became uneasy with the arbitrary boundary between the “frontier” that must be kept permanently in sync between “harden” and “lockdown”. It is not actually possible for us, at least at this time, to commit to an extent in “lockdown” that will not grow, such that an old “harden” and a new “lockdown” would fail to coördinate at the point of repairs.

[mhofman]

For the record, I am of the opinion that the user-facing harden installed on the global should behave identically before and after lockdown, traversing prototypes but stopping when such traversal reaches a host-realm primordial—i.e., that it be independent of lockdown and therefore equally usable with or without it, including before it.

I wanted to make them behave the same before and after lockdown, but in the opposite direction: that neither would check the prototypes. It would then become a matter for other layers like pass-style to enforce that prototype chains are hardened, which means that layer would now need to be lockdown aware. It also meant that it was easier for code wanting to harden to do an insufficient thing if another layer like pass-style was never involved.

I think that having different behaviors pre and post lockdown is the only way to reconcile this. I would like to offer better diagnostics to pre-lockdown harden users (not just post-lockdown), and this can potentially be done by harden pre-lockdown building a list of primordials to not warn about, but since it's for diagnostics only, that would be fine (it could also warn if it is hardening some primordial object, even pointing users to lockdown itself)

[gibson042]

This was actually how harden worked before my time at Agoric, and I became uneasy with the arbitrary boundary between the “frontier” that must be kept permanently in sync between “harden” and “lockdown”. It is not actually possible for us, at least at this time, to commit to an extent in “lockdown” that will not grow, such that an old “harden” and a new “lockdown” would fail to coördinate at the point of repairs.

@kriskowal I would like to hear more about this «old “harden” and a new “lockdown”» scenario. Right now harden is installed by lockdown, and what I'm proposing is that it instead by installed by the file that defines lockdown (i.e., packages/lockdown/pre.js). What would drive them out of sync?

I wanted to make them behave the same before and after lockdown, but in the opposite direction: that neither would check the prototypes. It would then become a matter for other layers like pass-style to enforce that prototype chains are hardened, which means that layer would now need to be lockdown aware. It also meant that it was easier for code wanting to harden to do an insufficient thing if another layer like pass-style was never involved.

@mhofman What would be the benefit of totally ignoring prototypes like that?

[kriskowal]

As much for my own edification, with propositions labeled for reference:

1. I start from the (urgent) premise that it be possible to join an ocap network without lockdown or oblige the application author to coordinate the order of installation of shims. To that end,
2. Suppose that we partition all valid programs into those that use a pre-lockdown harden and those that use a post-lockdown harden, and any application that tries to do both fails when they try to lockdown.
3. To ensure A, suppose that co-tenant @endo/harden implementations race to install non-writable, non-configurable Object[Symbol.for('harden')] upon the first invocation of harden. Any instance of @endo/harden is eligible to win the race and determines the semantics (and holder of the memo), and the losers silently tolerate losing the race. Whereas, if lockdown loses the race, it throws an exception.
4. We accept this is not the usual strategy for shims, which lately tend to encourage overwriting what was found, because there is a possibility applications will rely on the behavior of the shim and break if a future version of the language behaves differently than the shim predicted. We also accept that this kind of shimming is weirdly compatible with HardenedJS because it tolerates a previously locked-down environment.
5. The @endo/harden module would fail upon first attempt to harden inside a Compartment in composition with a version of SES that does not install Object[Symbol.for('harden')] and where globalThis.harden was not endowed. No such versions of ses currently exist, as compartments receive harden by default. No future version of ses would break because it will introduce Object[Symbol.for('harden')] which cannot be denied by the creator of a Compartment. It is a shared intrinsic that cannot be omitted from a Compartment except under extreme duress.
6. To ensure that @endo/harden works with versions of SES that predate the decision to install Object[Symbol.for('harden')], it will also sense (upon first invocation) that it lost the race if it instead finds globalThis.harden is installed, and will fall-through to that instead.
7. There would be no versions of @endo/harden that predate the choice to install Object[Symbol.for('harden')], so no mutual compatibility concerns among versions of @endo/harden.
8. Using the "exports" directive in the @endo/harden package.json, we can let the user of an application or the creator of a bundle elect a mode of harden that it will use if it wins the race. So, we can choose a default mode and elective alternate modes. Electing an alternative mode would look like bundle-source -C shallow-harden
9. The mode most useful for bundling applications that can presume that they are executed in a locked-down environment would simply omit its own implementation of harden and use the one installed at globalThis.harden or Object[Symbol.for('harden')]. That would look like bundle-source -C lockded-down. So, bundles that migrate from assuming the existence of a post-lockdown globalThis.harden can migrate to use @endo/harden, become more portable, without incurring a significant bundle size increase.
10. For every mode of @endo/harden, the mode variant only applies to the pre-lockdown behavior.
11. Ideally, a library that is built to use @endo/harden does not need to separately test its behavior pre- and post-lockdown. If it works with pre-lockdown, it should not break post-lockdown.
12. All post-lockdown harden options must guarantee that the transitive closure of both prototypes and properties are frozen, because calling harden should relieve the reviewer of any concern of reachable transitive mutability, even for code that might be run pre-lockdown.
13. Regardless of whether harden closes over prototypes when hardening instances, a library that does not immediately and explicitly harden a prototype will be vulnerable to mutation in coordination with other linked modules until the first instance is hardened, which may be considerably later, so we ideally provide some signal or encouragement to opportunistically harden.

The dimensions to the design of a harden implementation are:

• pre vs post
• harden extent the extent that harden freezes
• throwing extent the extent in which harden will throw if it finds an unfrozen object
• warning extent the extent in which harden will warn if it finds an unfrozen object
• for each extent, whether it close over:
  • surface transitive closure over properties of an object
  • volume transitive closure over both properties and prototypes of an object
  • global globalThis
  • undeniable intrinsics intrinsics that can’t be denied because they’re reachable with syntax, like AsyncFunction.prototype.
  • shared intrinsics the shared intrinsics, a set that may grow version over version
  • special intrinsics intrinsics that are reachable by calling methods of shared intrinsics, a set that may grow version over version

harden variants of interest.

• fake harden which returns the subject unaltered
• assuring harden for which the harden extent is the volume of the subject (only current behavior)
• throwing harden for which the harden extent is the surface of the subject and otherwise the throwing extent is the volume of the subject.
• warning harden for which the harden extent is the surface and otherwise the warning extent is the volume
• harden until intrinsics for which the harden extent is the volume of the subject except for the volume of shared intrinsics
• harden until global for which the harden extent is the volume of the subject except for the volume of global and special intrinsics

Discussion of variants:

• assuring harden would not be eligible before lockdown because it hardens shared intrinsics in a way that would interfere with lockdown, and also can interfere with shims of anything captured by a hardened instance. It’s technically at peace with lockdown because (2) ensures that using harden before lockdown foreswears ever calling lockdown. But, aggressively hardening prototypes down to the intrinsics is inconsistent with the values of a pre-lockdown application.
• warning harden pre-lockdown entrains complications about reporting warnings, because we will want knobs to mute or redirect those warnings, which will have to be on process.env. This isn’t a deal-breaker.
• warning harden post-lockdown runs afoul of (12) because it may proceed after harden for a subject with unfrozen objects in its volume.
• A library designed to work pre-lockdown with any eligible flavor of harden except throwing harden may fail to work post-lockdown with a throwing harden post-lockdown if the author failed to harden a prototype of a hardened instance.
• Moving to use throwing harden would be a breaking change for some existing libraries and would need to be opted-in with a lockdown option.
• Neither harden until intrinsics or harden until global are eligible post-lockdown because (12) they leave shared mutable objects in their wake. These are strictly pre-lockdown variants.
• Both harden until intrinsics and harden until global close over an extent that can get captured in a bundled version of @endo/harden. This is fine for post-lockdown usage because the bundled version will be ignored. This may be fine pre-lockdown because hardening an instance won’t accidentally freeze something that might be shimmed, particularly not objects that hide behind method invocation, unless you consider the case of shimming an object exported by another module by pre-emptively importing it.

Tangentially, the regarding whether @endo/harden should prefer Object[Symbol.for('harden')] over globalThis.harden:

• Favor Object because Object will always be frozen post-lockdown and globalThis is not necessarily. This is a weakly preferable since Compartments are not generally co-tenant and co-tenant Compartments must freeze globalThis.
• Favor globalThis because this gives individual compartments the option of receiving from without or installing from within a specific alternative globalThis.harden.

My tentative preference is:

• ses post-lockdown provides assuring harden by default
• ses post-lockdown provides throwing harden if configured so folks can opt into early discovery of unhardened prototypes. (Recall that we can’t do warning harden post-lockdown.)
• @endo/harden races to provide a fake harden implementation
• @endo/harden with -C hardened presumes lockdown and throws upon initialization if it doesn’t find harden in the environment
• @endo/harden with other -C conditions to race to install stronger pre-lockdown harden like throwing harden or warning harden, maybe suppressing warnings for objects reachable from the volume of global or known shared intrinsics.

[kriskowal]

Closing with intention to instead pursue #2976