Checks#

This module offers get_* functions to check for empty files, misindented JSON files, and invalid JSON files. See “pytest example” in each function’s documentation for usage examples.

This module also offers validate_* methods to test JSON Schema. Each method’s behavior is customizable, and not all methods are relevant to all schema.

The typical usage is to first define a test method like so:

from jscc.testing.filesystem import walk_json_data
from jscc.schema import is_json_schema
from jscc.testing.util import http_get

schemas = [(path, name, data) for path, name, _, data in walk_json_data() if is_json_schema(data)]
metaschema = http_get('http://json-schema.org/draft-04/schema').json()

@pytest.mark.parametrize('path,name,data', schemas)
def test_schema_valid(path, name, data):
    validate_json_schema(path, name, data, metaschema)

You can edit metaschema to be more strict and/or to add new properties. Then, define the validate_json_schema method that uses the validate_* methods. For example:

import jsonref
from jscc.testing.checks import (
    validate_array_items,
    validate_codelist_enum,
    validate_deep_properties,
    validate_items_type,
    validate_letter_case,
    validate_merge_properties,
    validate_metadata_presence,
    validate_null_type,
    validate_object_id,
    validate_ref,
    validate_schema,
)
from jsonschema import FormatChecker
from jsonschema.validators import Draft4Validator

validator = Draft4Validator(Draft4Validator.META_SCHEMA, format_checker=FormatChecker())


def validate_json_schema(path, name, data, schema):
    errors = 0

    errors += validate_schema(path, data, validator)
    errors += validate_array_items(path, data)
    errors += validate_items_type(path, data)
    errors += validate_codelist_enum(path, data)
    errors += validate_letter_case(path, data)
    errors += validate_merge_properties(path, data)
    errors += validate_ref(path, data)
    errors += validate_metadata_presence(path, data)
    errors += validate_object_id(path, jsonref.replace_refs(data))
    errors += validate_null_type(path, data)
    # Here, we don't add to `errors`, in order to not count these warnings as errors.
    validate_deep_properties(path, data)

    assert not errors, "One or more JSON Schema files are invalid. See warnings below."

You can monkeypatch warnings.formatwarning to customize and abbreviate the warning messages:

import os
import warnings
from jscc.exceptions import DeepPropertiesWarning

cwd = os.getcwd()

def formatwarning(message, category, filename, lineno, line=None):
    # Prefix warnings that count as errors with "ERROR: ".
    if category != DeepPropertiesWarning:
        message = 'ERROR: ' + message
    # Remove the path to the current working directory.
    return str(message).replace(cwd + os.sep, '')

warnings.formatwarning = formatwarning

jscc.testing.checks.get_empty_files(include=<function _true>, **kwargs)[source]#

Yields the path (as a tuple) of any file that is empty.

JSON files are empty if their parsed contents are empty (empty array, empty object, empty string or null). Other files are empty if they contain whitespace only.

Parameters:: include (function) – a method that accepts a file path and file name, and returns whether to test the file (default true)

pytest example:

from jscc.testing.checks import get_empty_files
from jscc.testing.util import warn_and_assert

def test_empty():
    warn_and_assert(get_empty_files(), '{0} is empty, run: rm {0}',
                    'Files are empty. See warnings below.')

jscc.testing.checks.get_misindented_files(include=<function _true>, **kwargs)[source]#

Yields the path (as a tuple) of any JSON file that isn’t formatted for humans.

JSON files must be indented with two spaces, mustn’t escape non-ASCII characters (no \uXXXX sequences), and must have a newline at end of file.

Parameters:: include (function) – a method that accepts a file path and file name, and returns whether to test the file (default true)

pytest example:

from jscc.testing.checks import get_misindented_files
from jscc.testing.util import warn_and_assert

def test_indent():
    warn_and_assert(get_misindented_files(), '{0} is not indented as expected, run: ocdskit indent {0}',
                    'Files are not indented as expected. See warnings below, or run: ocdskit indent -r .')

jscc.testing.checks.get_invalid_json_files(**kwargs)[source]#

Yields the path and exception (as a tuple) of any JSON file that isn’t valid.

JSON files must be parsed without error by the json module, and JSON objects mustn’t have duplicate keys.

See https://tools.ietf.org/html/rfc7493#section-2.3

pytest example:

from jscc.testing.checks import get_invalid_json_files
from jscc.testing.util import warn_and_assert

def test_invalid_json():
    warn_and_assert(get_invalid_json_files(), '{0} is not valid JSON: {1}',
                    'JSON files are invalid. See warnings below.')

jscc.testing.checks.validate_schema(path, data, validator)[source]#

Warns and returns the number of errors relating to JSON Schema validation.

Uses the jsonschema module.

Parameters:: validator – The validator to use
Returns:: the number of errors
Return type:: int

jscc.testing.checks.validate_letter_case(*args, property_exceptions=(), definition_exceptions=())[source]#

Warns and returns the number of errors relating to the letter case of properties and definitions.

Property names must be lowerCamelCase. Definition names must be UpperCamelCase. All must be ASCII letters only.

Parameters:

property_exceptions (list, tuple or set) – property names to ignore
definition_exceptions (list, tuple or set) – definition names to ignore

Returns:

the number of errors

Return type:

int

jscc.testing.checks.validate_metadata_presence(*args, allow_missing=<function _false>)[source]#

Warns and returns the number of errors relating to metadata in a JSON Schema.

The root schema and each field must have “type”, “title” and “description” properties, unless it has a “$ref” property.

Parameters:: allow_missing (function) – a method that accepts a JSON Pointer, and returns whether the field is allowed to not have a “title” or “description” property
Returns:: the number of errors
Return type:: int

jscc.testing.checks.validate_null_type(path, data, pointer='', no_null=False, expect_null=True, allow_object_null=(), allow_no_null=(), allow_null=())[source]#

Warns and returns the number of errors relating to non-nullable optional fields and nullable required fields.

If no_null is True, then “type” properties mustn’t include “null”.

Otherwise, the “type” properties for objects and arrays of objects mustn’t include “null”. If a field is required, then its “type” property mustn’t include “null”. If a field isn’t required (and it isn’t an object or array of objects), then its “type” property must include “null”.

Parameters:

no_null (bool) – whether the standard disallows “null” in the “type” property of any field
expect_null (bool) – whether the field, in context, is expected to have “null” in its “type” property
allow_object_null (list, tuple or set) – JSON Pointers of fields whose “type” properties are allowed to include “object”, “null”
allow_no_null (list, tuple or set) – JSON Pointers of fields whose “type” properties are allowed to exclude “null”
allow_null (list, tuple or set) – JSON Pointers of fields whose “type” properties are allowed to include “null”

Returns:

the number of errors

Return type:

int

jscc.testing.checks.validate_codelist_enum(*args, fallback=None, allow_enum=<function _false>, allow_missing=<function _false>)[source]#

Warns and returns the number of errors relating to codelists in a JSON Schema.

If a field has a “codelist” property but no “type” property (e.g. if the “codelist” property is being overwritten), then its “type” is assumed to be “array” unless a fallback “type” is provided via fallback.

If the “codelist” property is set:

If the “openCodelist” property is set to true, then the “enum” property mustn’t be set.
If the “openCodelist” property is set to false, then the “enum” property must be set, its value must include null if the “type” property includes “null”, and its value must match the codes in the codelist.

If the “enum” property is set, then the “codelist” and “openCodelist” properties must be set.

Parameters:

fallback (dict) – a dict in which keys are JSON Pointers and values are lists of “type” values
allow_enum (function) – a method that accepts a JSON Pointer, and returns whether the field is allowed to set the “enum” property without setting the “codelist” property
allow_missing (function) – a method that accepts a codelist name, and returns whether the codelist file is allowed to be missing from the repository

Returns:

the number of errors

Return type:

int

jscc.testing.checks.validate_array_items(*args, allow_invalid=())[source]#

Warns and returns the number of errors relating to array fields without an “items” property.

A field whose “type” property includes “array” must set the “items” property.

Parameters:: allow_invalid (list, tuple or set) – JSON Pointers of fields whose “items” properties are allowed to be missing
Returns:: the number of errors
Return type:: int

jscc.testing.checks.validate_items_type(*args, additional_valid_types=None, allow_invalid=())[source]#

Warns and returns the number of errors relating to the “type” property under an “items” property.

The “type” property under an “items” property must only include “array” (e.g. for geometries), “number” (e.g. for coordinates) and/or “string”.

Parameters:

additional_valid_types (list, tuple or set) – additional valid values of the “type” property under an “items” property
allow_invalid (list, tuple or set) – JSON Pointers of fields whose “type” properties are allowed to include invalid values

Returns:

the number of errors

Return type:

int

jscc.testing.checks.validate_deep_properties(*args, allow_deep=())[source]#

Warns and returns the number of errors relating to deep objects.

The schema must use “definitions” or “$defs” instead of nesting “properties”.

Parameters:: allow_deep (list, tuple or set) – JSON Pointers of fields to ignore
Returns:: the number of errors
Return type:: int

jscc.testing.checks.validate_object_id(*args, allow_missing=<function _false>, allow_optional=())[source]#

Warns and returns the number of errors relating to objects within arrays lacking “id” fields.

If an array field’s “wholeListMerge” and “omitWhenMerged” properties aren’t set or are set to false or null, then the object fields under it must have an “id” field, and the “id” field must be required.

Parameters:

allow_missing (function) – a method that accepts a JSON Pointer, and returns whether the field is allowed to not have an “id” field
allow_optional (list, tuple or set) – JSON Pointers of fields whose “id” field is allowed to be optional

Returns:

the number of errors

Return type:

int

jscc.testing.checks.validate_merge_properties(*args)[source]#

Warns and returns the number of errors relating to missing or extra merge properties.

The “omitWhenMerged” and “wholeListMerge” properties mustn’t both be set, and mustn’t be set to false or null. The “wholeListMerge” property must be set on non-nullable arrays of objects only.

See https://standard.open-contracting.org/1.1/en/schema/merging/#whole-list-merge

Returns:: the number of errors
Return type:: int

jscc.testing.checks.validate_ref(path, data, **kwargs)[source]#

Warns and returns 1 if not all $ref’erences can be resolved.

Uses the jsonref module.

Returns:: 0 or 1
Return type:: int

jscc.testing.checks.validate_schema_codelists_match(path, data, top, is_extension=False, is_profile=False, external_codelists=None)[source]#

Warns and returns the number of errors relating to mismatches between codelist files and codelist references from JSON Schema.

All codelist filenames that don’t start with + or - must match codelist references. All codelist references must match codelist filenames or be in external_codelists. All codelist filenames that start with + or - must be in external_codelists.

Parameters:

top (str) – the file path of the directory tree
is_extension (bool) – whether the repository is an extension or a profile
is_profile (bool) – whether the repository is a profile (a collection of extensions)
external_codelists (list, tuple or set) – names of codelists defined by the standard

Returns:

the number of errors

Return type:

int