Wednesday, May 18, 2022 · 14 min read

Astonishing Serializations & Schemas of Hyperborea¶

Yanick Champoux

on this page

Preparing the ground JSON is the worst Setting up the validation train Starting on the schema Adding stats and definitions One sample doesn't serious testing make More schemas!Referencing other parts of the schema Custom keywords More of the same

Originally published at techblog.babyl.ca.

For the last two years I am part of a band of intrepid adventurers joining forces every Thursday night via the teleporting magic of Discords, semi-arguably doing our best not to die horrible deaths in the raucously unforgiving world of [Astonishing Swordsmen and Sorcerers of Hyperborea][assh], a pulpy cousin of Dungeons & Dragons. The game is orchestrated by evil dungeon mastermind Gizmo Mathboy, and it's a massive amount of fun.

But the world of Hyperborea is not only besieged by monsters. Oh no. It is also a realm filled with rules, and statistics, and all manners of fate-defining dice rolls. And of the nexus capturing a lot of those arcane laws is -- unsurprising to all savvy to the genre -- the character sheet.

Being good little nerds, we usually do a good job of keeping the character sheets up-to-date. But we're all fallible creatures; mistakes creep in. Which made me think... surely there are ways to automate some validations on those character sheets. In fact, we already keep our sheets as YAML documents. JSON Schemas can totally be used to define document schemas... surely it could twisted a little bit more to accommodate the exotic logic of a game?

The answer is that, of course, everything can be twisted provided the spell is dark enough. This blog entry and its associated project repository, while not an exhaustive solution (yet), is intended to the goodies that JSON Schema could bring to the table, as well as the tools of the ecosystem.

So... Interested, fellow adventurers? Then gird those loins, sheath those blades, and follow me: into the JSON Schema jungle we go!

Preparing the ground¶

First, let's introduce the core tools I'll be using for this project.

For all things JSON Schema, we'll be using ajv (and ajv-cli for cli interactions). It's a fast, robust implementation of the JSON Schema specs with a lot of bonus features, and to ice the cake it provides an easy mechanism to add custom validation keywords, something we'll abuse before long.

And since we'll do a lot of command-line stuff, I'll bring in Task, a YAML-based task runner -- basically Makefile with the insane whitespace-based syntax replaced by, uh, a different insane whitespace-based syntax I'm comfortable with.

Incidentally, the final form of all the code I'm going to discuss in this article is in this repo.

JSON is the worst¶

Okay, that's overly mean. JSON is a great serialization format, but it's a soulless drag to edit manually. But that's not much of a problem, as JSON Schema is kind of a misnomer: both the target documents and the schemas themselves are ultimately just plain old data structure -- JSON just happens to be the typical serialization for it. Well, typical be damned, we'll go with YAML as our source. And for convenience for the other pieces to come, we'll convert those YAML documents to JSON via [transerialize][].

1# in Taskfile.yml
2tasks:
3    schemas: fd -e yml -p ./schemas-yaml -x task schema SCHEMA='{}'
4
5    schema:
6        vars:
7            DEST:
8                sh: echo {{.SCHEMA}} | perl -pe's/ya?ml/json/g'
9        sources: ["{{.SCHEMA}}"]
10        generates: ["{{.DEST}}"]
11        cmds: transerialize {{.SCHEMA}} {{.DEST}}

Oh yeah, task is unfortunately janky where loops are concerned, so I'm using fd and re-entries to deal with all the individual schema conversions.

Setting up the validation train¶

Before we go hog-wild on the schema itself, we need to figure out how we'll invoke things. And to do that, let's seed our schema and sample document in the most boring, minimalistic manner possible.

1# file: schemas-yaml/character.yml
2$id: https://hyperboria.babyl.ca/character.json
3title: Hyperboria character sheet
4type: object

1# file: samples/verg.yml
2
3# Verg is my character, and always ready to face danger,
4# so it makes sense that he'd be volunteering there
5name: Verg-La

We have a schema, we have a document, and the straightforward way to validate it is to do the following.

1⥼ ajv validate -s schemas-yaml/character.yml -d samples/verg.yml
2samples/verg.yml valid

Sweet. Now we just need to formalize it a little bit in Taskfile and we're ready to roll.

1# file: Taskfile.yml
2# in the tasks
3validate:
4    silent: true
5    cmds:
6        - |
7            ajv validate  \\
8                --all-errors \\
9                --errors=json \\
10                --verbose \\
11                -s schemas-yaml/character.yml \\
12                -d {{.CLI_ARGS}}

Starting on the schema¶

To warm ourselves up, let's begin with some easy fields. A character has obviously a name and a player.

1# file: schemas-yaml/character.json
2$id: https://hyperboria.babyl.ca/character.json
3title: Hyperboria character sheet
4type: object
5additionalProperties: false
6required:
7    - name
8    - player
9properties:
10    name: &string
11        type: string
12    player: *string

Nothing special there, except for the YAML anchor and alias, because I'm a lazy bugger.

1⥼ task validate -- samples/verg.yml
2samples/verg.yml invalid
3[
4    ...
5    "message": "must have required property 'player'",
6    ...
7]

Woo! Validation is screaming at us! The output is abbreged here because I configured it to be extra-verbose in the taskfile. But the gist is clear: we're supposed to have a player name and we don't. So let's add it.

1# file: samples/verg.yml
2name: Verg-La
3player: Yanick

And with the player name added, all is well again.

1⥼ task validate -- samples/verg.yml
2samples/verg.yml valid

Adding stats and definitions¶

Next thing, core statistics! All statistics are following the same rules (numbers between 1 and 20). Copying and pasting the schema for all stats would be uncouth. Using anchors as in the previous section is an option, but in this case it's better to use the a schema definition, to make things a little more formal.

1# file: schemas-yaml/character.yml
2# only showing deltas
3required:
4    # ...
5    - statistics
6properties:
7    # ...
8    statistics:
9        type: object
10        allRequired: true
11        properties:
12            strength: &stat
13                $ref: "#/$defs/statistic"
14            dexterity: *stat
15            constitution: *stat
16            intelligence: *stat
17            wisdom: *stat
18            charisma: *stat
19$defs:
20    statistic:
21        type: number
22        minimum: 1
23        maximum: 20

Note that the allRequired is a custom keyword made available by ajv-keywords, and to use it we have to amend our call to ajv validate in the the taskfile:

1# file: Taskfile.yml
2validate:
3    silent: true
4    cmds:
5        - |
6            ajv validate \\
7                --all-errors \\
8                --errors=json \\
9                --verbose \\
10                -c ajv-keywords \\
11                -s schemas-yaml/character.yml \\
12                -d {{.CLI_ARGS}}

To conform to the schema, we add the stats to our sample character too:

1# file: samples/verg.yml
2statistics:
3    strength: 11
4    dexterity: 13
5    constitution: 10
6    intelligence: 18
7    wisdom: 15
8    charisma: 11

And we check and, yup, our sheet is still valid.

1⥼ task validate -- samples/verg.yml
2samples/verg.yml valid

One sample doesn't serious testing make¶

So far, we've used Verg as our test subject. We tweak the schema, run it against the sheet, tweak the sheet, rince, lather, repeat. But as the schema is getting more complex, we probably want to add a real test suite to our little project.

One way would be to use ajv test, which has the appeal that no additional code is required.

1⥼ ajv test -c ajv-keywords \\
2    -s schemas-yaml/character.yml \\
3    -d samples/verg.yml \\
4    --valid
5samples/verg.yml passed test
6# bad-verg.yml is like verg.yml, but missing the player name
7⥼ ajv test -c ajv-keywords \\
8    -s schemas-yaml/character.yml \\
9    -d samples/bad-verg.yml \\
10    --invalid
11samples/bad-verg.yml passed test

But what it has in simplicity, it lacks in modularity. Those schemas are going to get a little more involved, and targeting pieces of them would be good. So instead we'll go with good old unit test, via [vitest][].

For example, let's test statistics.

1// file: src/statistics.test.js
2import { test, expect } from "vitest";
3
4import Ajv from "ajv";
5
6import characterSchema from "../schemas-json/character.json";
7
8const ajv = new Ajv();
9// we just care about the statistic schema here, so that's what
10// we take
11const validate = ajv.compile(characterSchema.$defs.statistic);
12
13test("good statistic", () => {
14    expect(validate(12)).toBeTruthy();
15    expect(validate.errors).toBeNull();
16});
17
18test("bad statistic", () => {
19    expect(validate(21)).toBeFalsy();
20    expect(validate.errors[0]).toMatchObject({
21        message: "must be <= 20",
22    });
23});

We add a test task to our taskfile:

1# file: Taskfile.yml
2test:
3    deps: [schemas]
4    cmds:
5        - vitest run

And just like that, we have tests.

1⥼ task test
2task: [schemas] fd -e yml -p ./schemas-yaml -x task schema SCHEMA='{}'
3task: [schema] transerialize schemas-yaml/test.yml schemas-json/test.json
4task: [schema] transerialize schemas-yaml/character.yml schemas-json/character.json
5task: [test] vitest run
6
7 RUN  v0.10.0 /home/yanick/work/javascript/hyperboria-character-sheet
8
9 √ src/statistics.test.js (2)
10
11Test Files  1 passed (1)
12     Tests  2 passed (2)
13      Time  1.41s (in thread 5ms, 28114.49%)

More schemas!¶

Next step: the character class. While we could just slam an enum in the main schema and call it done, it's a list that might be re-used somewhere else, so it might pay off to define it in its own schema, and refer to it in the character sheet schema.

Addititional challenge! In Hyperborea you can have a generic class, or a class and sub-class. Which can be schematized explicitly, like this:

1oneOf:
2    - enum: [ magician, figher ]
3    - type: object
4      properties:
5        generic: { const: fighter }
6        subclass: { enum: [ barbarian, warlock, ... ] }
7    ...

But that's a lot of repetitive typing. Instead, it'd be nice to have the source be more compact, if a little less JSON Schemy. Say, something like this:

1$id: https://hyperboria.babyl.ca/classes.json
2title: Classes of characters for Hyperborea
3$defs:
4    fighter:
5        - barbarian
6        - berserker
7        - cataphract
8        - hunstman
9        - paladin
10        - ranger
11        - warlock
12    magician: [cryomancer, illusionist, necromancer, pyromancer, witch]

And then have a little script massage the data as we turn the YAML into JSON. Fortunately (what a lucky break!), transerialize does allow for a transformation script to be wedged in the process. So we can change our taskfile schema task to be:

1schema:
2    vars:
3        TRANSFORM:
4            sh: |
5                echo {{.SCHEMA}} | \\
6                    perl -lnE's/yml$/pl/; s/^/.\//; say if -f $_'
7        DEST:
8            sh: echo {{.SCHEMA}} | perl -pe's/ya?ml/json/g'
9    cmds:
10        - transerialize {{.SCHEMA}} {{.TRANSFORM}} {{.DEST}}

And then we slip in a transform script that looks like this:

1# file: schemas-yaml/classes.pl
2sub {
3    my $schema = $_->{oneOf} = [];
4
5    push @$schema, { enum => [ keys $_->{'$defs'}->%* ] };
6
7    for my $generic ( keys $_->{'$defs'}->%* ) {
8        push @$schema, {
9            type => 'object',
10            properties => {
11                generic => { const => $generic },
12                subclass => { enum => $_->{'$defs'}{$generic} }
13            }
14        }
15    }
16
17    return $_;
18}

With that, the output schema is inflated to what we want. We're having our concise cake eating the big fluffy one too. Nice!

So what is left is to link the schemas together. We refer to the classes schema from the character schema:

1# file: schemas-yaml/character.yml
2required:
3    # ...
4    - class
5properties:
6    # ...
7    class: { $ref: "/classes.json" }

We also need to tell ajv of the existence of that new schema:

1validate:
2    silent: true
3    cmds:
4        - |
5            ajv validate \\
6                --all-errors \\
7                --errors=json \\
8                --verbose \\
9                -c ajv-keywords \\
10                -r schemas-json/classes.json \\
11                -s schemas-json/character.json \\
12                -d {{.CLI_ARGS}}

Finally, we add Verg's class to his sheet:

1# file: samples/verg.yml
2class:
3  generic: magician
4  subclass: cryomancer

And just like that, Verg (and our character schema) is all classy and stuff.

Referencing other parts of the schema¶

So far we can set up our character sheet schema to ensure that we have the fields that we want, with the types and values that we want. But something else we want to do is to validate the relations between properties.

For example, characters have a health statistic. Each time the character levels up, the player rolls a dice and increases the health accordingly. As you image, forgetting to get that bonus can prove to be a lethal mistake, so it'd be nice to ensure that never happens.

We'll do it through the magic of JSON Pointers and avj's $data, like so:

1# file: schemas-yaml/character.yml
2level: { type: number, minimum: 1 }
3health:
4    type: object
5    required: [ max ]
6    properties:
7        max: { type: number }
8        current: { type: number }
9        log:
10            type: array
11            description: history of health rolls
12            items: { type: number }
13            minItems: { $data: /level }
14            maxItems: { $data: /level }

Basically (and once we add a --data flag to ajv to tell it to enable that feature), any mention of { $data: '/path/to/another/value/in/the/schema' } will be replaced by the value for which that JSON pointer resolves to in the document being validated. That's something that is not part of JSON Schema proper, but it's a mightily useful way to interconnect the schema and the document being validated.

Word of caution, though: I say 'any mention of $data', but that's overselling it. There are a few cases where $data fields won't be resolved. If you are to use that feature, make sure to reserve a few minutes to read the AJV docs about it. Trust me, it'll save you a few "what the everlasting heck?" moments.

Custom keywords¶

In the previous section, we checked that the number of rolls for health is equal to the level of the character. That's already something. But the logical next step is to ensure that the sum of those rolls are equal to the max health points we have. We'd need something like:

1# file: schemas-yaml/character.yml
2health:
3    type: object
4    properties:
5        max:
6            type: number
7            sumOf: { list: { $data: 1/log } }
8        log:
9            type: array
10            items: { type: number }

That's where custom keywords enter the picture. AJV allows us to augment the JSON Schema vocabulary with new keywords.

There is a few ways to define that custom keyword. The one I opted for is defining it as a JavaScript function (here made a little more complex because we're dealing internally with JSON pointers):

1// file: src/sumOf.cjs
2
3const _ = require("lodash");
4const ptr = require("json-pointer");
5
6function resolvePointer(data, rootPath, relativePath) {
7    if (relativePath[0] === "/") return ptr.get(data, relativePath);
8
9    const m = relativePath.match(/^(\d+)(.*)/);
10    relativePath = m[2];
11    for (let i = 0; i < parseInt(m[1]); i++) {
12        rootPath = rootPath.replace(/\/[^\/]+$/, "");
13    }
14
15    return ptr.get(data, rootPath + relativePath);
16}
17
18module.exports = (ajv) =>
19    ajv.addKeyword({
20        keyword: "sumOf",
21        $data: true,
22        errors: true,
23        validate: function validate(
24            { list, map },
25            total,
26            _parent,
27            { rootData, instancePath }
28        ) {
29            if (list.$data)
30                list = resolvePointer(rootData, instancePath, list.$data);
31
32            if (map) data = _.map(data, map);
33
34            if (_.sum(list) === total) return true;
35
36            validate.errors = [
37                {
38                    keyword: "sumOf",
39                    message: "should add up to sum total",
40                    params: {
41                        list,
42                    },
43                },
44            ];
45
46            return false;
47        },
48    });

As usual we have to tell ajv to include that new bit of code via -c ./src/sumOf.cjs. But beside that, congrats, we have a new keyword!

More of the same¶

By now we have most of the tools we want, all that is left to do is to turn the crank.

Experience points? Much of the same logic as for the health points:

1# file: schemas-yaml/character.yml
2experience:
3    type: object
4    properties:
5        total:
6            type: number
7            sumOf:
8                list: { $data: '1/log' }
9                map: amount
10        log:
11            type: array
12        items:
13            type: object
14            properties:
15                date: *string
16                amount: *number
17                notes: *string

The other basic attributes are trivial:

1# file: schemas-yaml/character.yml
2gender: *string
3age: *number
4height: *string
5appearance: *string
6alignment: *string

Fields based on lists? Been there, done that:

1# file: schemas-yaml/character.yml
2  race: { $ref: /races.json }
3  languages:
4    type: array
5    minItems: 1
6    items:
7      $ref: /languages.json

Spells are only for magicians? Not a problem.

1# file: schemas-yaml/character.yml
2type: object
3properties:
4    # ...
5    spells:
6      type: array
7      items: { $ref: /spells.json }
8      maxSpells:
9        class: { $data: /class }
10        level: { $data: /level }

With the new keyword maxSpells:

1// file: src/maxSpells.cjs
2
3const _ = require("lodash");
4const resolvePointer = require('./resolvePointer.cjs');
5
6module.exports = (ajv) =>
7    ajv.addKeyword({
8        keyword: "maxSpells",
9        validate: function validate(
10            schema,
11            data,
12            _parent,
13            { rootData, instancePath }
14        ) {
15            if (schema.class.$data) {
16                schema.class = resolvePointer(
17                    rootData, instancePath, schema.class.$data
18                );
19            }
20
21            if( schema.class !== 'magician'
22                && schema.class?.generic !== 'magician'
23                && data.length ) {
24                validate.errors = [
25                    {
26                        message: "non-magician can't have spells",
27                    },
28                ];
29                return false;
30            }
31
32            return true;
33        },
34        $data: true,
35        errors: true,
36    });

Gears? Pfah! Sure.

1# file: schemas-yaml/character.yml
2properties:
3    # ...
4    gear: { $ref: '#/$defs/gear' }
5$defs:
6  gear:
7    type: array
8    items:
9      oneOf:
10        - *string
11        - type: object
12          properties:
13            desc:
14              type: string
15              description: description of the equipment
16            qty:
17              type: number
18              description: |
19                quantity of the item in the
20                character's possession
21          required: [ desc ]
22          additionalProperties: false
23          examples:
24            - { desc: 'lamp oil', qty: 2 }

By now you get the point. A lot of constraints can be expressed via vanilla JSON Schema keywords. For the weirder things, new keywords can be added. And for anything that is onerous to type of, we have to remember that underneath it's all JSON, and we know darn well how to munge that.