Skip to main content

Add a reference data field or section

New constants and tables the model reads belong in ReferenceData, not in code, so administrators can see, change and version them. This walk-through adds a hypothetical constant, costs.onCostCap, and a new view.

1. The type​

packages/core/src/reference/types.ts: add the field with a doc comment saying what it is and its units.

costs: {
  /** Default salary → employer cost multiplier. */
  employerCostMultiplier: number;
  /** Upper bound on employer cost per person (£); null for none. */
  onCostCap?: number | null;
};

For a new section, add it to ReferenceData and to REFERENCE_SECTIONS (the satisfies clause keeps the list honest).

2. Validation​

packages/core/src/reference/schema.ts:

  • add the field to referenceDataSchema with its type and range (z.number().positive().max(10_000_000).nullable().optional());
  • add cross-checks in crossCheck for relationships with other tables, as an error (blocks publishing) or a warning (shown, doesn't block), with a dot path pointing at the field.

3. The built-in data​

packages/core/src/reference/builtin/index.ts (and the topic file it imports from): set the built-in value, which seeds version 1 on new databases.

4. Compatibility with published versions​

This is the step that's easy to miss. Published versions are immutable JSON that don't have the new field, and datasets are pinned to them:

  • the server validates a version when it loads it (loadReference → validateReferenceData), so a required new field makes every older version fail with Reference data version N is invalid and breaks every dataset pinned to it;
  • the browser compiles the published JSON as stored (useReferenceVersion → compileReference(r.data)), without zod parsing, so a zod .default() would be applied on the server but not in the browser, and the two would score differently.

So make the field optional in the type and schema, and apply the fallback in core code that both runtimes run, typically compileReference or the model function that reads it:

const cap = ref.data.costs.onCostCap ?? null; // absent in versions published before the field existed

The fallback must reproduce the old behaviour exactly, so existing datasets keep their figures. When a new version is published through the editor, the field is filled in.

5. Compilation (if needed)​

If the field is looked up often or needs preparing (regexes, maps, sets), add it to Reference in reference/compile.ts and build it in compileReference. Keep lookups pure and memoised per data object.

6. Use it in the model​

Read it from ref.data (or the compiled Reference) in the relevant core module. Never copy the value into a code constant.

If it changes a figure, quote it in the metric's provenance in metric-provenance.ts (build(ref) reads ref.data), so the explanation always matches the number. See Provenance.

7. The admin editor​

apps/web/src/pages/admin/reference/sections.tsx:

  • add a control to the view that owns the section (here, a NumberField in ModelConstants), using withSection(p, "costs") to write it back; or
  • for a new section, write an Editor and add a ReferenceView to REFERENCE_VIEWS with its id, group, label, description and the dot paths it edits. The paths drive the issue links, the "changed" markers and the error counts in the navigation.

Use DataGrid for tables (with ordered for first-match lists and keyPath for keyed records), NumberField and LinesField for constants and lists, and JsonEditor for deeply nested structures. CSV import and export come with DataGrid for free.

8. Tests​

  • packages/core/test/reference.test.ts: the built-in data still validates; the new field's range and cross-checks; an older document without the field still validates and scores identically.
  • Model tests for the behaviour the field controls.
  • apps/web/test/referenceEdit.test.ts if you touched the editor helpers.
  • apps/api/test/reference.test.ts if publishing or pinning behaviour changes.

9. Roll it out​

New databases get the field from the built-in data. Existing databases keep their published versions; an administrator publishes a new version with the value (the editor shows the field with its fallback until then). Mention it in CHANGELOG.md and the user guide's Reference data page.