Preload Configuration

Ontoserver includes a mechanism to preload content (including ValueSet, CodeSystem, ConceptMap and StructureDefinition resources) on startup.

There are different ways to use this feature. Some of these are demonstrated below.

1. Preload a bundle of resources

The Ontoserver docker image includes a preload feed that points to a FHIR Bundle at file:///data/preload.json.

This can be overwritten with a new Bundle to make Ontoserver load other resources.

For example, the following Bundle includes a very simple CodeSystem resource:

  {
    "resourceType": "Bundle",
    "type": "batch",
    "entry": [
      {
        "resource": {
          "resourceType": "CodeSystem",
          "url": "http://sample.com/CodeSystem/1",
          "status": "draft",
          "content": "complete",
          "concept": [
            {
              "code": "A",
              "display": "First concept"
            },
            {
              "code": "B",
              "display": "Second concept"
            }
          ]
        }
      }
    ]
  }

Having created this in a file called preload.json, we can make a new docker image using a Dockerfile like:

  FROM quay.io/aehrc/ontoserver:ctsa-6.1
  COPY preload.json data/preload.json

We can then put the Dockerfile and our preload.json file in a directory (e.g. myOntoserver), and tweak our docker-compose.yml file to use our new image instead of the base Ontoserver image, by replacing the line

    image: quay.io/aehrc/ontoserver:ctsa-6.1

with

    build: myOntoserver

and running it with a build step, such as

docker-compose -f docker-compose.yml build
docker-compose -f docker-compose.yml up -d

2. Preload with a feed of resources

If you prefer to load resources separately rather than bundling them up, another option is to create a syndication feed and preload content from that.

For example, the following feed includes a CodeSystem and a ValueSet.

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:ncts="http://ns.electronichealth.net.au/ncts/syndication/asf/extensions/1.0.0">
    <title type="text">My preload feed</title>
    <link rel="alternate" type="application/atom+xml" href="file:///data/preload.xml"/>
    <updated>2019-06-30</updated>
    <id>urn:uuid:b617906a-e0f8-45f7-a0f3-0742a2bb3135</id>
    <ncts:atomSyndicationFormatProfile version="http://ns.electronichealth.net.au/ncts/syndication/asf/profile/1.0.0"/>

  <entry>
      <title>My CodeSystem</title>
      <category term="FHIR_CodeSystem" label="FHIR Code System" scheme="http://ns.electronichealth.net.au/ncts/syndication/asf/scheme/1.0.0" />
      <id>7f6af166-cf09-4741-83d3-ac2be4880e00</id>
      <published>2019-06-30</published>
      <summary type="text">This is my CodeSystem</summary>
      <link rel="alternate" href="file:///data/myCodeSystem.json" type="application/fhir+json" />
      <ncts:contentItemIdentifier>http://example.com/CodeSystem/1</ncts:contentItemIdentifier>
      <ncts:contentItemVersion>1.0.0</ncts:contentItemVersion>
      <ncts:fhirVersion>4.0</ncts:fhirVersion>
  </entry>

  <entry>
      <title>My ValueSet</title>
      <category term="FHIR_ValueSet" label="FHIR Value Set" scheme="http://ns.electronichealth.net.au/ncts/syndication/asf/scheme/1.0.0" />
      <id>539a689f-cd46-4756-94be-29aed8544cb2</id>
      <published>2019-06-30</published>
      <summary type="text">This is my ValueSet</summary>
      <link rel="alternate" href="file:///data/myValueSet.json" type="application/fhir+json" />
      <ncts:contentItemIdentifier>http://example.com/ValueSet/1</ncts:contentItemIdentifier>
      <ncts:contentItemVersion>1.0.0</ncts:contentItemVersion>
      <ncts:fhirVersion>4.0</ncts:fhirVersion>
  </entry>
</feed>

We can then put our files in a directory structure such as

+ myOntoserver/
  - Dockerfile
  + data/
    - preload.xml
    - myCodeSystem.json
    - myValueSet.json

with a Dockerfile such as 

FROM quay.io/aehrc/ontoserver:ctsa-6.1
COPY data/* data/

and, like option 1, change our docker-compose.yml file to use our new Docker image, by replacing 

    image: quay.io/aehrc/ontoserver:ctsa-6.1

with

    build: myOntoserver

and running it with a build step, such as

docker-compose -f docker-compose.yml build
docker-compose -f docker-compose.yml up -d

3. Preload using a remote feed or multiple feeds

If it is necessary to use either remote syndication feeds, or multiple feeds, this can be done using the atom.preload.feedLocation configuration parameter.

This setting can be included in the docker-compose.yml file.

For example, we could add the following line to the environment section of our ontoserver container

    - atom.preload.feedLocation=https://acme.com/acme-terminology.xml,https://some.department.gov.au/fhir-resources.xml

4. Keeping a read-only mirror in sync with its feeds

By default, preload only ever adds content: if a resource is removed from the preload feeds, the copy already loaded into Ontoserver remains indefinitely. For a read-only server that mirrors one or more preload feeds, you may instead want Ontoserver to track those feeds — removing content it previously loaded once the feed no longer offers it.

Setting atom.preload.sync=true enables this, but only when the server is also in read-only mode (ontoserver.deployment.readOnly=true). The two must be set together; atom.preload.sync has no effect on a writable server, because the sweep's safety relies on the preload feeds being the sole source of content.

When enabled:

  • Ontoserver records which preload feed entry delivered each resource. A resource delivered by more than one feed entry is tracked against each of them. Only content that preload itself installs (or that you explicitly adopt, below) is tracked; the sweep can only ever delete tracked content.
  • At the end of a fully successful preload (no entry failed to load and no upstream feed was unreachable), any tracked resource all of whose delivering feed entries have disappeared from the feeds is deleted. This includes superseded versions and members dropped from a package that is otherwise still listed.
  • Content loaded by any other means — for example, manually via the Syndication API or a direct FHIR write — carries no such record and is never deleted by this mechanism, unless you adopt it.

    If a preload does not complete successfully, no deletions occur, so a transient feed outage cannot cause content to be removed. As a further safeguard, the sweep will refuse to run if it would delete more than atom.preload.sync.deleteLimit resources (default 10) in a single preload — protecting against a truncated or partially-retrieved feed being mistaken for a mass withdrawal. Set the limit below 0 to remove the cap entirely. A value of 0 is rejected. When a legitimate large withdrawal trips the cap, raise it for a single run without changing the configured default by passing POST /synd/redoPreload?deleteLimit=N (a value below 0 makes that one run unbounded); this too requires sync to be enabled.

    Before enabling deletions on an existing deployment, set atom.preload.sync.dryRun=true. In this mode Ontoserver logs the resources the sweep would delete without removing anything, letting you confirm the effect first. Both dryRun and deleteLimit scope to the sweep only: explicit feed retractions (below) are authoritative publisher withdrawals and always apply.

Adopting pre-existing content

When you first point an already-populated read-only server at its feeds, the content loaded earlier (by hand, or by preload before atom.preload.sync was enabled) is untracked: the feed still ships it, but it carries no provenance record, so the sweep neither maintains nor deletes it. A normal preload will not silently take ownership of that content — adoption is a deliberate, one-shot operator action.

To enrol the pre-existing content that matches the current feeds into the sweep's management, run a preload once with the adopt parameter:

POST /synd/redoPreload?adopt=true

For that run only, any untracked resource the feeds still deliver is stamped with the delivering entry, after which ordinary preloads keep it up to date (and the sweep removes it once the feeds drop it). Adoption is not a standing setting: the scheduled and startup preloads never adopt, and there is nothing to turn off afterwards. The request is rejected (HTTP 400) unless the server is in sync mode (read-only with atom.preload.sync=true), because adoption only makes sense where the sweep runs. atom.preload.sync.dryRun does not suppress adoption — it gates the sweep only — so an adopt run enrols content and previews (rather than performs) any resulting deletions.

5. Withdrawing individual items

Regardless of atom.preload.sync, a preload feed can explicitly withdraw a specific item by including a retraction entry — an entry whose category is one of the ..._RETRACT categories (for example FHIR_CodeSystem_RETRACT) and that carries no content link. During preload, Ontoserver will not install content matching a retraction, and will remove any matching content already present. A retraction only affects a resource whose date is present and not later than the retraction entry's published date, so a publisher can re-issue newer content under the same identity without it being removed. A resource with no date is never retracted (it cannot be shown to predate the retraction) and is logged.