Skip to content
Commits on Source (7)
include gittaggers.py Makefile LICENSE.txt
include gittaggers.py Makefile LICENSE.txt requirements.txt
include schema_salad/py.typed
include schema_salad/avro/*
include schema_salad/tests/*
include schema_salad/tests/test_schema/*.md
......
......@@ -33,7 +33,7 @@ COVBASE=coverage run --branch --append --source=${MODULE} \
# Updating the Major & Minor version below?
# Don't forget to update setup.py as well
VERSION=3.0.$(shell date +%Y%m%d%H%M%S --utc --date=`git log --first-parent \
VERSION=4.5.$(shell date +%Y%m%d%H%M%S --utc --date=`git log --first-parent \
--max-count=1 --format=format:%cI`)
## all : default task
......@@ -89,13 +89,13 @@ diff_pycodestyle_report: pycodestyle_report.txt
pep257: pydocstyle
## pydocstyle : check Python code style
pydocstyle: $(PYSOURCES)
pydocstyle --ignore=D100,D101,D102,D103 $^ || true
pydocstyle --add-ignore=D100,D101,D102,D103 $^ || true
pydocstyle_report.txt: $(PYSOURCES)
pydocstyle setup.py $^ > $@ 2>&1 || true
diff_pydocstyle_report: pydocstyle_report.txt
diff-quality --violations=pycodestyle $^
diff-quality --violations=pycodestyle --fail-under=100 $^
## autopep8 : fix most Python code indentation and formatting
autopep8: $(PYSOURCES)
......@@ -199,7 +199,7 @@ jenkins: FORCE
rm -Rf env && virtualenv env
. env/bin/activate ; \
pip install -U setuptools pip wheel ; \
${MAKE} install-dep coverage.html coverage.xml pep257_report.txt \
${MAKE} install-dep coverage.html coverage.xml pydocstyle_report.txt \
sloccount.sc pycodestyle_report.txt pylint_report.txt
if ! test -d env3 ; then virtualenv -p python3 env3 ; fi
. env3/bin/activate ; \
......
Metadata-Version: 1.1
Name: schema-salad
Version: 3.0.20181206233650
Version: 4.5.20190621200723
Summary: Schema Annotations for Linked Avro Data (SALAD)
Home-page: https://github.com/common-workflow-language/schema_salad
Author: Common workflow language working group
Author-email: common-workflow-language@googlegroups.com
License: Apache 2.0
Download-URL: https://github.com/common-workflow-language/schema_salad/releases
Description: |Build Status| |Build status|
Description: |Linux Build Status| |Windows Build status| |Code coverage| |CII Best Practices|
.. |Build Status| image:: https://img.shields.io/travis/common-workflow-language/schema_salad/master.svg?label=unix%20build
.. |Linux Build Status| image:: https://img.shields.io/travis/common-workflow-language/schema_salad/master.svg?label=unix%20build
:target: https://travis-ci.org/common-workflow-language/schema_salad
.. |Build status| image:: https://img.shields.io/appveyor/ci/mr-c/schema-salad/master.svg?label=windows%20build
.. |Windows Build status| image:: https://img.shields.io/appveyor/ci/mr-c/schema-salad/master.svg?label=windows%20build
:target: https://ci.appveyor.com/project/mr-c/schema-salad/branch/master
.. |Code coverage| image:: https://codecov.io/gh/common-workflow-language/schema_salad/branch/master/graph/badge.svg
:target: https://codecov.io/gh/common-workflow-language/schema_salad
.. |CII Best Practices| image:: https://bestpractices.coreinfrastructure.org/projects/1867/badge
:target: https://bestpractices.coreinfrastructure.org/projects/1867
Schema Salad
------------
......@@ -64,6 +66,10 @@ Description: |Build Status| |Build status|
$ schema-salad-tool myschema.yml mydocument.yml
Generate HTML documentation::
$ schema-salad-tool myschema.yml > myschema.html
Get JSON-LD context::
$ schema-salad-tool --print-jsonld-context myschema.yml mydocument.yml
......@@ -81,6 +87,84 @@ Description: |Build Status| |Build status|
$ schema-salad-tool --print-inheritance-dot myschema.yml | dot -Tsvg > myschema.svg
Quick Start
-----------
Let's say you have a 'basket' record that can contain items measured either by weight or by count. Here's an example::
basket:
- product: bananas
price: 0.39
per: pound
weight: 1
- product: cucumbers
price: 0.79
per: item
count: 3
We want to validate that all the expected fields are present, the
measurement is known, and that "count" cannot be a fractional value.
Here is an example schema to do that::
- name: Product
doc: |
The base type for a product. This is an abstract type, so it
can't be used directly, but can be used to define other types.
type: record
abstract: true
fields:
product: string
price: float
- name: ByWeight
doc: |
A product, sold by weight. Products may be sold by pound or by
kilogram. Weights may be fractional.
type: record
extends: Product
fields:
per:
type:
type: enum
symbols:
- pound
- kilogram
jsonldPredicate: '#per'
weight: float
- name: ByCount
doc: |
A product, sold by count. The count must be a integer value.
type: record
extends: Product
fields:
per:
type:
type: enum
symbols:
- item
jsonldPredicate: '#per'
count: int
- name: Basket
doc: |
A basket of products. The 'documentRoot' field indicates it is a
valid starting point for a document. The 'basket' field will
validate subtypes of 'Product' (ByWeight and ByCount).
type: record
documentRoot: true
fields:
basket:
type:
type: array
items: Product
You can check the schema and document in schema_salad/tests/basket_schema.yml and schema_salad/tests/basket.yml::
$ schema-salad-tool basket_schema.yml basket.yml
Document `basket.yml` is valid
Documentation
-------------
......@@ -138,7 +222,6 @@ Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
|Build Status| |Build status|
|Linux Build Status| |Windows Build status| |Code coverage| |CII Best Practices|
.. |Build Status| image:: https://img.shields.io/travis/common-workflow-language/schema_salad/master.svg?label=unix%20build
.. |Linux Build Status| image:: https://img.shields.io/travis/common-workflow-language/schema_salad/master.svg?label=unix%20build
:target: https://travis-ci.org/common-workflow-language/schema_salad
.. |Build status| image:: https://img.shields.io/appveyor/ci/mr-c/schema-salad/master.svg?label=windows%20build
.. |Windows Build status| image:: https://img.shields.io/appveyor/ci/mr-c/schema-salad/master.svg?label=windows%20build
:target: https://ci.appveyor.com/project/mr-c/schema-salad/branch/master
.. |Code coverage| image:: https://codecov.io/gh/common-workflow-language/schema_salad/branch/master/graph/badge.svg
:target: https://codecov.io/gh/common-workflow-language/schema_salad
.. |CII Best Practices| image:: https://bestpractices.coreinfrastructure.org/projects/1867/badge
:target: https://bestpractices.coreinfrastructure.org/projects/1867
Schema Salad
------------
......@@ -55,6 +57,10 @@ Validate a document using a schema::
$ schema-salad-tool myschema.yml mydocument.yml
Generate HTML documentation::
$ schema-salad-tool myschema.yml > myschema.html
Get JSON-LD context::
$ schema-salad-tool --print-jsonld-context myschema.yml mydocument.yml
......@@ -72,6 +78,84 @@ Display inheritance relationship between classes as a graphviz 'dot' file and re
$ schema-salad-tool --print-inheritance-dot myschema.yml | dot -Tsvg > myschema.svg
Quick Start
-----------
Let's say you have a 'basket' record that can contain items measured either by weight or by count. Here's an example::
basket:
- product: bananas
price: 0.39
per: pound
weight: 1
- product: cucumbers
price: 0.79
per: item
count: 3
We want to validate that all the expected fields are present, the
measurement is known, and that "count" cannot be a fractional value.
Here is an example schema to do that::
- name: Product
doc: |
The base type for a product. This is an abstract type, so it
can't be used directly, but can be used to define other types.
type: record
abstract: true
fields:
product: string
price: float
- name: ByWeight
doc: |
A product, sold by weight. Products may be sold by pound or by
kilogram. Weights may be fractional.
type: record
extends: Product
fields:
per:
type:
type: enum
symbols:
- pound
- kilogram
jsonldPredicate: '#per'
weight: float
- name: ByCount
doc: |
A product, sold by count. The count must be a integer value.
type: record
extends: Product
fields:
per:
type:
type: enum
symbols:
- item
jsonldPredicate: '#per'
count: int
- name: Basket
doc: |
A basket of products. The 'documentRoot' field indicates it is a
valid starting point for a document. The 'basket' field will
validate subtypes of 'Product' (ByWeight and ByCount).
type: record
documentRoot: true
fields:
basket:
type:
type: array
items: Product
You can check the schema and document in schema_salad/tests/basket_schema.yml and schema_salad/tests/basket.yml::
$ schema-salad-tool basket_schema.yml basket.yml
Document `basket.yml` is valid
Documentation
-------------
......
python-schema-salad (4.5.20190621200723-1) unstable; urgency=medium
* New upstream version
* debhelper-compat 12
* Standards-Version: 4.4.0
-- Michael R. Crusoe <michael.crusoe@gmail.com> Tue, 13 Aug 2019 13:08:20 +0200
python-schema-salad (3.0.20181206233650-2) unstable; urgency=medium
* Fix the already shipped tests and enable them as autopkgtests.
......
......@@ -3,10 +3,11 @@ Maintainer: Debian Med Packaging Team <debian-med-packaging@lists.alioth.debian.
Uploaders: Michael R. Crusoe <michael.crusoe@gmail.com>
Section: python
Priority: optional
Build-Depends: debhelper (>= 11~),
Build-Depends: debhelper-compat (= 12),
dh-python,
python3,
python3-all,
python3-future,
python3-setuptools,
python3-mistune,
python3-requests,
......@@ -17,10 +18,10 @@ Build-Depends: debhelper (>= 11~),
python3-cachecontrol,
python3-lockfile,
python3-typing-extensions,
python3-pytest-xdist,
python3-pytest-runner,
python3-pytest-xdist,
python3-pytest-runner,
help2man
Standards-Version: 4.3.0
Standards-Version: 4.4.0
Vcs-Browser: https://salsa.debian.org/med-team/python-schema-salad
Vcs-Git: https://salsa.debian.org/med-team/python-schema-salad.git
Homepage: http://www.commonwl.org
......@@ -31,7 +32,7 @@ Depends: ${python3:Depends},
${misc:Depends},
libjs-bootstrap
Recommends: cwltool
Breaks: cwltool (<< 1.0.20181128175059)
Breaks: cwltool (<< 1.0.20190621)
Conflicts: python-schema-salad
Description: Schema Annotations for Linked Avro Data (SALAD)
Salad is a schema language for describing JSON or YAML structured linked data
......
From: Michael R. Crusoe <michael.crusoe@gmail.com>
Subject: Add proper "#!" header to all CWL files
Forwarded: https://github.com/common-workflow-language/schema_salad/pull/225
--- schema-salad.orig/schema_salad/tests/foreign/foreign_prop1.cwl
+++ schema-salad/schema_salad/tests/foreign/foreign_prop1.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
doc: |
Foreign properties test. This property is not part of the supplied
ontology. If strict_foreign_properties is true, it should fail. If
@@ -7,4 +8,4 @@
inputs: []
outputs: []
baseCommand: echo
-'edam:has_topic': abc
\ No newline at end of file
+'edam:has_topic': abc
--- schema-salad.orig/schema_salad/tests/foreign/foreign_prop2.cwl
+++ schema-salad/schema_salad/tests/foreign/foreign_prop2.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
doc: |
Foreign properties test. If strict_foreign_properties is true, it
should fail. If false, should pass no warnings.
@@ -8,4 +9,4 @@
inputs: []
outputs: []
baseCommand: echo
-'edam:has_topic': abc
\ No newline at end of file
+'edam:has_topic': abc
--- schema-salad.orig/schema_salad/tests/foreign/foreign_prop3.cwl
+++ schema-salad/schema_salad/tests/foreign/foreign_prop3.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
doc: |
Foreign properties test. Should pass since the edam namespace is declared,
the ontology is imported, the property is valid in the ontology, and the
@@ -11,4 +12,4 @@
inputs: []
outputs: []
baseCommand: echo
-edam:has_topic: edam:topic_0003
\ No newline at end of file
+edam:has_topic: edam:topic_0003
--- schema-salad.orig/schema_salad/tests/foreign/foreign_prop4.cwl
+++ schema-salad/schema_salad/tests/foreign/foreign_prop4.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
doc: |
Foreign properties test. This property is not part of the supplied
ontology. If strict_foreign_properties is true, it should fail. If
@@ -11,4 +12,4 @@
inputs: []
outputs: []
baseCommand: echo
-'edam:fake_property': abc
\ No newline at end of file
+'edam:fake_property': abc
--- schema-salad.orig/schema_salad/tests/foreign/foreign_prop5.cwl
+++ schema-salad/schema_salad/tests/foreign/foreign_prop5.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
doc: |
Foreign properties test. The property is supposed to cross
reference another concept node, but that node doesn't exist.
@@ -12,4 +13,4 @@
inputs: []
outputs: []
baseCommand: echo
-edam:has_topic: abc
\ No newline at end of file
+edam:has_topic: abc
--- schema-salad.orig/schema_salad/tests/foreign/foreign_prop6.cwl
+++ schema-salad/schema_salad/tests/foreign/foreign_prop6.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
doc: |
Foreign properties test. The property
references a mailto: URI
@@ -10,4 +11,4 @@
inputs: []
outputs: []
baseCommand: echo
-edam:has_topic: mailto:test@example.com
\ No newline at end of file
+edam:has_topic: mailto:test@example.com
--- schema-salad.orig/schema_salad/tests/foreign/foreign_prop7.cwl
+++ schema-salad/schema_salad/tests/foreign/foreign_prop7.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
doc: |
Foreign properties test. The property
reference an unsupported mailto: URI
@@ -10,4 +11,4 @@
inputs: []
outputs: []
baseCommand: echo
-edam:has_topic: urn:blurb
\ No newline at end of file
+edam:has_topic: urn:blurb
--- schema-salad.orig/schema_salad/tests/test_schema/test1.cwl
+++ schema-salad/schema_salad/tests/test_schema/test1.cwl
@@ -1 +1,2 @@
-class: Workflow
\ No newline at end of file
+#!/usr/bin/env cwl-runner
+class: Workflow
--- schema-salad.orig/schema_salad/tests/test_schema/test10.cwl
+++ schema-salad/schema_salad/tests/test_schema/test10.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
@@ -7,4 +8,4 @@
step1:
scatterMethod: [record]
in: []
- out: [out]
\ No newline at end of file
+ out: [out]
--- schema-salad.orig/schema_salad/tests/test_schema/test11.cwl
+++ schema-salad/schema_salad/tests/test_schema/test11.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
@@ -7,4 +8,4 @@
step1:
run: blub.cwl
in: []
- out: [out]
\ No newline at end of file
+ out: [out]
--- schema-salad.orig/schema_salad/tests/test_schema/test12.cwl
+++ schema-salad/schema_salad/tests/test_schema/test12.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
--- schema-salad.orig/schema_salad/tests/test_schema/test13.cwl
+++ schema-salad/schema_salad/tests/test_schema/test13.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
@@ -17,4 +18,4 @@
id: blah
class: CommandLineTool
inputs: []
- outputs: []
\ No newline at end of file
+ outputs: []
--- schema-salad.orig/schema_salad/tests/test_schema/test14.cwl
+++ schema-salad/schema_salad/tests/test_schema/test14.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
--- schema-salad.orig/schema_salad/tests/test_schema/test16.cwl
+++ schema-salad/schema_salad/tests/test_schema/test16.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
--- schema-salad.orig/schema_salad/tests/test_schema/test17.cwl
+++ schema-salad/schema_salad/tests/test_schema/test17.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
baseCommand: cowsay
--- schema-salad.orig/schema_salad/tests/test_schema/test18.cwl
+++ schema-salad/schema_salad/tests/test_schema/test18.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
class: CommandLineTool
cwlVersion: v1.0
baseCommand: echo
--- schema-salad.orig/schema_salad/tests/test_schema/test19.cwl
+++ schema-salad/schema_salad/tests/test_schema/test19.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
: aaa
cwlVersion: v1.0
class: CommandLineTool
--- schema-salad.orig/schema_salad/tests/test_schema/test2.cwl
+++ schema-salad/schema_salad/tests/test_schema/test2.cwl
@@ -1 +1,2 @@
-class: xWorkflow
\ No newline at end of file
+#!/usr/bin/env cwl-runner
+class: xWorkflow
--- schema-salad.orig/schema_salad/tests/test_schema/test3.cwl
+++ schema-salad/schema_salad/tests/test_schema/test3.cwl
@@ -1,6 +1,7 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
outputs:
bar: xstring
-steps: []
\ No newline at end of file
+steps: []
--- schema-salad.orig/schema_salad/tests/test_schema/test4.cwl
+++ schema-salad/schema_salad/tests/test_schema/test4.cwl
@@ -1,6 +1,7 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
outputs:
bar: 12
-steps: []
\ No newline at end of file
+steps: []
--- schema-salad.orig/schema_salad/tests/test_schema/test5.cwl
+++ schema-salad/schema_salad/tests/test_schema/test5.cwl
@@ -1,6 +1,7 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
outputs:
bar: string
-steps: [12]
\ No newline at end of file
+steps: [12]
--- schema-salad.orig/schema_salad/tests/test_schema/test6.cwl
+++ schema-salad/schema_salad/tests/test_schema/test6.cwl
@@ -1,5 +1,6 @@
+#!/usr/bin/env cwl-runner
inputs:
foo: string
outputs:
bar: string
-steps: [12]
\ No newline at end of file
+steps: [12]
--- schema-salad.orig/schema_salad/tests/test_schema/test7.cwl
+++ schema-salad/schema_salad/tests/test_schema/test7.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
@@ -7,4 +8,4 @@
step1:
scatter_method: blub
in: []
- out: [out]
\ No newline at end of file
+ out: [out]
--- schema-salad.orig/schema_salad/tests/test_schema/test8.cwl
+++ schema-salad/schema_salad/tests/test_schema/test8.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
@@ -7,4 +8,4 @@
step1:
scatterMethod: abc
in: []
- out: [out]
\ No newline at end of file
+ out: [out]
--- schema-salad.orig/schema_salad/tests/test_schema/test9.cwl
+++ schema-salad/schema_salad/tests/test_schema/test9.cwl
@@ -1,3 +1,4 @@
+#!/usr/bin/env cwl-runner
class: Workflow
inputs:
foo: string
@@ -7,4 +8,4 @@
step1:
scatterMethod: 12
in: []
- out: [out]
\ No newline at end of file
+ out: [out]
--- schema-salad.orig/schema_salad/tests/test_print_oneline.py
+++ schema-salad/schema_salad/tests/test_print_oneline.py
@@ -46,7 +46,7 @@
except RuntimeError as e:
msg = reformat_yaml_exception_message(strip_dup_lineno(six.text_type(e)))
msg = to_one_line_messages(msg)
- self.assertTrue(msg.endswith(src+":10:1: could not find expected \':\'"))
+ self.assertTrue(msg.endswith(src+":11:1: could not find expected \':\'"))
print("\n", e)
raise
@@ -62,10 +62,9 @@
six.text_type(get_data("tests/test_schema/"+src)), True)
except ValidationException as e:
msgs = to_one_line_messages(str(e)).splitlines()
- self.assertEqual(len(msgs), 3)
- self.assertTrue(msgs[0].endswith(src+":9:1: the `outputs` field is not valid because"))
- self.assertTrue(msgs[1].endswith(src+":13:5: missing required field `id`"))
- self.assertTrue(msgs[2].endswith(src+":13:5: invalid field `aa`, expected one of: 'label', 'secondaryFiles', 'format', 'streamable', 'doc', 'id', 'outputBinding', 'type'"))
+ assert len(msgs) == 2, msgs
+ self.assertTrue(msgs[0].endswith(src+":14:5: missing required field `id`"))
+ self.assertTrue(msgs[1].endswith(src+":14:5: invalid field `aa`, expected one of: 'label', 'secondaryFiles', 'format', 'streamable', 'doc', 'id', 'outputBinding', 'type'"))
print("\n", e)
raise
@@ -88,9 +87,9 @@
self.assertEqual(len(msgs), 2)
print("\n", e)
assert msgs[0].endswith(
- src + ':8:1: checking field `outputs`')
+ src + ':9:1: checking field `outputs`')
assert msgs[1].endswith(
- src + ':13:5: Field `type` references unknown identifier '
+ src + ':14:5: Field `type` references unknown identifier '
'`Filea`, tried file://%s#Filea' % (fullpath))
raise
@@ -108,8 +107,8 @@
msg = reformat_yaml_exception_message(strip_dup_lineno(six.text_type(e)))
msgs = msg.splitlines()
self.assertEqual(len(msgs), 2)
- self.assertTrue(msgs[0].endswith(src+":9:7: while scanning a simple key"))
- self.assertTrue(msgs[1].endswith(src+":10:1: could not find expected ':'"))
+ self.assertTrue(msgs[0].endswith(src+":10:7: while scanning a simple key"))
+ self.assertTrue(msgs[1].endswith(src+":11:1: could not find expected ':'"))
print("\n", e)
raise
@@ -125,7 +124,7 @@
six.text_type(get_data("tests/test_schema/"+src)), True)
except RuntimeError as e:
msg = reformat_yaml_exception_message(strip_dup_lineno(six.text_type(e)))
- self.assertTrue(msg.endswith(src+":1:1: expected <block end>, but found ':'")
- or msg.endswith(src+":1:1: expected <block end>, but found u':'"))
+ self.assertTrue(msg.endswith(src+":2:1: expected <block end>, but found ':'")
+ or msg.endswith(src+":2:1: expected <block end>, but found u':'"))
print("\n", e)
raise
typing==3.7.4 ; python_version < "3.5"
ruamel.yaml>=0.12.4, <= 0.15.96
rdflib==4.2.2
rdflib-jsonld==0.4.0
mistune>=0.8.1,<0.9
CacheControl==0.11.7
lockfile==0.12.2
typing-extensions
future
Metadata-Version: 1.1
Name: schema-salad
Version: 3.0.20181206233650
Version: 4.5.20190621200723
Summary: Schema Annotations for Linked Avro Data (SALAD)
Home-page: https://github.com/common-workflow-language/schema_salad
Author: Common workflow language working group
Author-email: common-workflow-language@googlegroups.com
License: Apache 2.0
Download-URL: https://github.com/common-workflow-language/schema_salad/releases
Description: |Build Status| |Build status|
Description: |Linux Build Status| |Windows Build status| |Code coverage| |CII Best Practices|
.. |Build Status| image:: https://img.shields.io/travis/common-workflow-language/schema_salad/master.svg?label=unix%20build
.. |Linux Build Status| image:: https://img.shields.io/travis/common-workflow-language/schema_salad/master.svg?label=unix%20build
:target: https://travis-ci.org/common-workflow-language/schema_salad
.. |Build status| image:: https://img.shields.io/appveyor/ci/mr-c/schema-salad/master.svg?label=windows%20build
.. |Windows Build status| image:: https://img.shields.io/appveyor/ci/mr-c/schema-salad/master.svg?label=windows%20build
:target: https://ci.appveyor.com/project/mr-c/schema-salad/branch/master
.. |Code coverage| image:: https://codecov.io/gh/common-workflow-language/schema_salad/branch/master/graph/badge.svg
:target: https://codecov.io/gh/common-workflow-language/schema_salad
.. |CII Best Practices| image:: https://bestpractices.coreinfrastructure.org/projects/1867/badge
:target: https://bestpractices.coreinfrastructure.org/projects/1867
Schema Salad
------------
......@@ -64,6 +66,10 @@ Description: |Build Status| |Build status|
$ schema-salad-tool myschema.yml mydocument.yml
Generate HTML documentation::
$ schema-salad-tool myschema.yml > myschema.html
Get JSON-LD context::
$ schema-salad-tool --print-jsonld-context myschema.yml mydocument.yml
......@@ -81,6 +87,84 @@ Description: |Build Status| |Build status|
$ schema-salad-tool --print-inheritance-dot myschema.yml | dot -Tsvg > myschema.svg
Quick Start
-----------
Let's say you have a 'basket' record that can contain items measured either by weight or by count. Here's an example::
basket:
- product: bananas
price: 0.39
per: pound
weight: 1
- product: cucumbers
price: 0.79
per: item
count: 3
We want to validate that all the expected fields are present, the
measurement is known, and that "count" cannot be a fractional value.
Here is an example schema to do that::
- name: Product
doc: |
The base type for a product. This is an abstract type, so it
can't be used directly, but can be used to define other types.
type: record
abstract: true
fields:
product: string
price: float
- name: ByWeight
doc: |
A product, sold by weight. Products may be sold by pound or by
kilogram. Weights may be fractional.
type: record
extends: Product
fields:
per:
type:
type: enum
symbols:
- pound
- kilogram
jsonldPredicate: '#per'
weight: float
- name: ByCount
doc: |
A product, sold by count. The count must be a integer value.
type: record
extends: Product
fields:
per:
type:
type: enum
symbols:
- item
jsonldPredicate: '#per'
count: int
- name: Basket
doc: |
A basket of products. The 'documentRoot' field indicates it is a
valid starting point for a document. The 'basket' field will
validate subtypes of 'Product' (ByWeight and ByCount).
type: record
documentRoot: true
fields:
basket:
type:
type: array
items: Product
You can check the schema and document in schema_salad/tests/basket_schema.yml and schema_salad/tests/basket.yml::
$ schema-salad-tool basket_schema.yml basket.yml
Document `basket.yml` is valid
Documentation
-------------
......@@ -138,7 +222,6 @@ Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
......@@ -3,6 +3,7 @@ MANIFEST.in
Makefile
README.rst
gittaggers.py
requirements.txt
setup.cfg
setup.py
schema_salad/__init__.py
......@@ -14,6 +15,7 @@ schema_salad/jsonld_context.py
schema_salad/main.py
schema_salad/makedoc.py
schema_salad/metaschema.py
schema_salad/py.typed
schema_salad/python_codegen.py
schema_salad/python_codegen_support.py
schema_salad/ref_resolver.py
......@@ -52,6 +54,10 @@ schema_salad/metaschema/map_res_src.yml
schema_salad/metaschema/metaschema.yml
schema_salad/metaschema/metaschema_base.yml
schema_salad/metaschema/salad.md
schema_salad/metaschema/sfdsl_res.yml
schema_salad/metaschema/sfdsl_res_proc.yml
schema_salad/metaschema/sfdsl_res_schema.yml
schema_salad/metaschema/sfdsl_res_src.yml
schema_salad/metaschema/typedsl_res.yml
schema_salad/metaschema/typedsl_res_proc.yml
schema_salad/metaschema/typedsl_res_schema.yml
......@@ -69,9 +75,14 @@ schema_salad/tests/cwl-pre.yml
schema_salad/tests/frag.yml
schema_salad/tests/hello.txt
schema_salad/tests/hellofield.yml
schema_salad/tests/inject-id1.yml
schema_salad/tests/inject-id2.yml
schema_salad/tests/inject-id3.yml
schema_salad/tests/list.json
schema_salad/tests/matcher.py
schema_salad/tests/metaschema-pre.yml
schema_salad/tests/mixin.yml
schema_salad/tests/multidoc.yml
schema_salad/tests/pt.yml
schema_salad/tests/test_cg.py
schema_salad/tests/test_cli_args.py
......
setuptools
requests>=1.0
ruamel.yaml<0.16,>=0.12.4
ruamel.yaml<=0.15.96,>=0.12.4
rdflib<4.3.0,>=4.2.2
rdflib-jsonld<0.5.0,>=0.3.0
mistune<0.9,>=0.8.1
......@@ -8,6 +8,7 @@ CacheControl<0.12,>=0.11.7
lockfile>=0.9
six>=1.8.0
typing-extensions
future
[:python_version<"3.7"]
typing>=3.6.4
[:python_version<"3.5"]
typing>=3.7.4
......@@ -196,7 +196,7 @@ class Names(object):
"""Track name set and default namespace during parsing."""
def __init__(self, default_namespace=None):
# type: (Optional[Text]) -> None
# type: (Optional[Text]) -> None
self.names = {} # type: Dict
self.default_namespace = default_namespace
......
......@@ -2,12 +2,10 @@
from __future__ import absolute_import, print_function
import argparse
import itertools
import logging
import os
import re
import sys
from typing import (Any, Dict, List, Mapping, MutableSequence, Pattern, Tuple,
from typing import (Any, Dict, List, Mapping, MutableSequence,
Union, cast)
import pkg_resources # part of setuptools
......@@ -21,9 +19,10 @@ from typing_extensions import Text # pylint: disable=unused-import
from . import codegen, jsonld_context, schema, validate
from .ref_resolver import Loader, file_uri
from .sourceline import strip_dup_lineno
from .sourceline import strip_dup_lineno, to_one_line_messages, reformat_yaml_exception_message
from .utils import json_dumps
from .avro.schema import SchemaParseException
from .makedoc import makedoc
register('json-ld', Parser, 'rdflib_jsonld.parser', 'JsonLDParser')
_logger = logging.getLogger("salad")
......@@ -38,84 +37,6 @@ def printrdf(workflow, # type: str
g = jsonld_context.makerdf(workflow, wf, ctx)
print(g.serialize(format=sr, encoding='utf-8').decode('utf-8')) # type: ignore
def regex_chunk(lines, regex):
# type: (List[str], Pattern[str]) -> List[List[str]]
lst = list(itertools.dropwhile(lambda x: not regex.match(x), lines))
arr = []
while lst:
ret = [lst[0]]+list(itertools.takewhile(lambda x: not regex.match(x),
lst[1:]))
arr.append(ret)
lst = list(itertools.dropwhile(lambda x: not regex.match(x),
lst[1:]))
return arr
def chunk_messages(message): # type: (str) -> List[Tuple[int, str]]
file_regex = re.compile(r'^(.+:\d+:\d+:)(\s+)(.+)$')
item_regex = re.compile(r'^\s*\*\s+')
arr = []
for chun in regex_chunk(message.splitlines(), file_regex):
fst = chun[0]
mat = file_regex.match(fst)
if mat:
place = mat.group(1)
indent = len(mat.group(2))
lst = [mat.group(3)]+chun[1:]
if [x for x in lst if item_regex.match(x)]:
for item in regex_chunk(lst, item_regex):
msg = re.sub(item_regex, '', "\n".join(item))
arr.append((indent, place+' '+re.sub(
r'[\n\s]+', ' ', msg)))
else:
msg = re.sub(item_regex, '', "\n".join(lst))
arr.append((indent, place+' '+re.sub(
r'[\n\s]+', ' ', msg)))
return arr
def to_one_line_messages(message): # type: (str) -> str
ret = []
max_elem = (0, '')
for (indent, msg) in chunk_messages(message):
if indent > max_elem[0]:
max_elem = (indent, msg)
else:
ret.append(max_elem[1])
max_elem = (indent, msg)
ret.append(max_elem[1])
return "\n".join(ret)
def reformat_yaml_exception_message(message): # type: (str) -> str
line_regex = re.compile(r'^\s+in "(.+)", line (\d+), column (\d+)$')
fname_regex = re.compile(r'^file://'+re.escape(os.getcwd())+'/')
msgs = message.splitlines()
ret = []
if len(msgs) == 3:
msgs = msgs[1:]
nblanks = 0
elif len(msgs) == 4:
c_msg = msgs[0]
match = line_regex.match(msgs[1])
if match:
c_file, c_line, c_column = match.groups()
c_file = re.sub(fname_regex, '', c_file)
ret.append("%s:%s:%s: %s" % (c_file, c_line, c_column, c_msg))
msgs = msgs[2:]
nblanks = 2
p_msg = msgs[0]
match = line_regex.match(msgs[1])
if match:
p_file, p_line, p_column = match.groups()
p_file = re.sub(fname_regex, '', p_file)
ret.append("%s:%s:%s:%s %s" % (p_file, p_line, p_column, ' '*nblanks, p_msg))
return "\n".join(ret)
def main(argsl=None): # type: (List[str]) -> int
if argsl is None:
......@@ -156,6 +77,9 @@ def main(argsl=None): # type: (List[str]) -> int
exgroup.add_argument("--print-oneline", action="store_true",
help="Print each error message in oneline")
exgroup.add_argument("--print-doc", action="store_true",
help="Print HTML schema documentation page")
exgroup = parser.add_mutually_exclusive_group()
exgroup.add_argument("--strict", action="store_true", help="Strict validation (unrecognized or out of place fields are error)",
default=True, dest="strict")
......@@ -170,6 +94,12 @@ def main(argsl=None): # type: (List[str]) -> int
exgroup.add_argument("--debug", action="store_true",
help="Print even more logging")
parser.add_argument('--only', action='append', help="Use with --print-doc, document only listed types")
parser.add_argument('--redirect', action='append', help="Use with --print-doc, override default link for type")
parser.add_argument('--brand', help="Use with --print-doc, set the 'brand' text in nav bar")
parser.add_argument('--brandlink', help="Use with --print-doc, set the link for 'brand' in nav bar")
parser.add_argument('--primtype', default="#PrimitiveType", help="Use with --print-doc, link to use for primitive types (string, int etc)")
parser.add_argument("schema", type=str, nargs="?", default=None)
parser.add_argument("document", type=str, nargs="?", default=None)
parser.add_argument("--version", "-v", action="store_true",
......@@ -210,15 +140,19 @@ def main(argsl=None): # type: (List[str]) -> int
schema_raw_doc, schema_uri)
except (validate.ValidationException) as e:
_logger.error("Schema `%s` failed link checking:\n%s",
args.schema, e, exc_info=(True if args.debug else False))
args.schema, Text(e), exc_info=(True if args.debug else False))
_logger.debug("Index is %s", list(metaschema_loader.idx.keys()))
_logger.debug("Vocabulary is %s", list(metaschema_loader.vocab.keys()))
return 1
except (RuntimeError) as e:
_logger.error("Schema `%s` read error:\n%s",
args.schema, e, exc_info=(True if args.debug else False))
args.schema, Text(e), exc_info=(True if args.debug else False))
return 1
if args.print_doc:
makedoc(args)
return 0
# Optionally print the schema after ref resolution
if not args.document and args.print_pre:
print(json_dumps(schema_doc, indent=4))
......@@ -234,7 +168,7 @@ def main(argsl=None): # type: (List[str]) -> int
metaschema_loader, args.strict)
except validate.ValidationException as e:
_logger.error("While validating schema `%s`:\n%s",
args.schema, str(e))
args.schema, Text(e))
return 1
# Get the json-ld context and RDFS representation from the schema
......@@ -262,7 +196,7 @@ def main(argsl=None): # type: (List[str]) -> int
try:
avsc_names = schema.make_avro_schema_from_avro(avsc_obj)
except SchemaParseException as err:
_logger.error("Schema `%s` error:\n%s", args.schema, err,
_logger.error("Schema `%s` error:\n%s", args.schema, Text(err),
exc_info=(
(type(err), err, None) if args.debug else None))
if args.print_avro:
......
......@@ -55,7 +55,7 @@ class MyRenderer(mistune.Renderer):
self.options = {}
def header(self, text, level, raw=None): # type: (Text, int, Any) -> Text
return """<h%i id="%s">%s</h%i>""" % (level, to_id(text), text, level)
return """<h%i id="%s" class="section">%s <a href="#%s">&sect;</a></h%i>""" % (level, to_id(text), text, to_id(text), level)
def table(self, header, body): # type: (Text, Text) -> Text
return (
......@@ -253,14 +253,14 @@ class RenderType(object):
if len(ar) > 40:
ar += "<br>"
ar += "map&lt;<code>%s</code>,&nbsp;<code>%s</code> | %s&gt" % (
ar += "<a href='#map'>map</a>&lt;<code>%s</code>,&nbsp;<code>%s</code> | %s&gt" % (
jsonldPredicate["mapSubject"], jsonldPredicate["mapPredicate"],
self.typefmt(tp["items"], redirects))
else:
ar += " | "
if len(ar) > 40:
ar += "<br>"
ar += "map&lt;<code>%s</code>,&nbsp;%s&gt" % (
ar += "<a href='#map'>map</a>&lt;<code>%s</code>,&nbsp;%s&gt" % (
jsonldPredicate["mapSubject"],
self.typefmt(tp["items"], redirects))
return ar
......@@ -271,6 +271,8 @@ class RenderType(object):
return """<a href="%s">%s</a>""" % (redirects[tp["name"]], frg)
if tp["name"] in self.typemap:
return """<a href="#%s">%s</a>""" % (to_id(frg), frg)
if tp["type"] == "https://w3id.org/cwl/salad#enum" and len(tp["symbols"]) == 1:
return "constant value <code>%s</code>" % schema.avro_name(tp["symbols"][0])
return frg
if isinstance(tp["type"], MutableMapping):
return self.typefmt(tp["type"], redirects)
......@@ -371,8 +373,14 @@ class RenderType(object):
if f["type"] == "record":
doc += "<h3>Fields</h3>"
doc += """<table class="table table-striped">"""
doc += "<tr><th>field</th><th>type</th><th>required</th><th>description</th></tr>"
doc += """
<div class="responsive-table">
<div class="row responsive-table-header">
<div class="col-xs-3 col-lg-2">field</div>
<div class="col-xs-2 col-lg-1">required</div>
<div class="col-xs-7 col-lg-3">type</div>
<div class="col-xs-12 col-lg-6 description-header">description</div>
</div>"""
required = []
optional = []
for i in f.get("fields", []):
......@@ -388,19 +396,24 @@ class RenderType(object):
# desc = "%s _Inherited from %s_" % (desc, linkto(i["inherited_from"]))
rfrg = schema.avro_name(i["name"])
tr = "<td><code>%s</code></td><td>%s</td><td>%s</td>"\
"<td>%s</td>" % (
rfrg, self.typefmt(tp, self.redirects,
jsonldPredicate=i.get("jsonldPredicate")),
opt,
mistune.markdown(desc))
tr = """
<div class="row responsive-table-row">
<div class="col-xs-3 col-lg-2"><code>%s</code></div>
<div class="col-xs-2 col-lg-1">%s</div>
<div class="col-xs-7 col-lg-3">%s</div>
<div class="col-xs-12 col-lg-6 description-col">%s</div>
</div>""" % (rfrg, "required" if opt else "optional",
self.typefmt(tp, self.redirects,
jsonldPredicate=i.get("jsonldPredicate")),
mistune.markdown(desc))
if opt:
required.append(tr)
else:
optional.append(tr)
for i in required + optional:
doc += "<tr>" + i + "</tr>"
doc += """</table>"""
doc += i
doc += """</div>"""
elif f["type"] == "enum":
doc += "<h3>Symbols</h3>"
doc += """<table class="table table-striped">"""
......@@ -449,7 +462,12 @@ def avrold_doc(j, # type: List[Dict[Text, Any]]
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css" integrity="sha384-604wwakM23pEysLJAhja8Lm42IIwYrJ0dEAqzFsj9pJ/P5buiujjywArgPCi8eoz" crossorigin="anonymous">
<script>
// Picture element HTML5 shiv
document.createElement( "picture" );
</script>
<script src="https://cdn.rawgit.com/scottjehl/picturefill/3.0.2/dist/picturefill.min.js" integrity="sha384-ZJsVW8YHHxQHJ+SJDncpN90d0EfAhPP+yA94n+EhSRzhcxfo84yMnNk+v37RGlWR" crossorigin="anonymous" async></script>
""")
outdoc.write("<title>%s</title>" % (rt.title))
......@@ -470,6 +488,52 @@ def avrold_doc(j, # type: List[Dict[Text, Any]]
margin-left: 2em;
margin-right: 2em;
}
.section a {
visibility: hidden;
}
.section:hover a {
visibility: visible;
color: rgb(201, 201, 201);
}
.responsive-table-header {
text-align: left;
padding: 8px;
vertical-align: top;
font-weight: bold;
border-top-color: rgb(221, 221, 221);
border-top-style: solid;
border-top-width: 1px;
background-color: #f9f9f9
}
.responsive-table > .responsive-table-row {
text-align: left;
padding: 8px;
vertical-align: top;
border-top-color: rgb(221, 221, 221);
border-top-style: solid;
border-top-width: 1px;
}
@media (min-width: 0px), print {
.description-header {
display: none;
}
.description-col {
margin-top: 1em;
margin-left: 1.5em;
}
}
@media (min-width: 1170px) {
.description-header {
display: inline;
}
.description-col {
margin-top: 0px;
margin-left: 0px;
}
}
.responsive-table-row:nth-of-type(odd) {
background-color: #f9f9f9
}
</style>
</head>
<body>
......@@ -529,6 +593,10 @@ def main(): # type: () -> None
args = parser.parse_args()
makedoc(args)
def makedoc(args): # type: (argparse.Namespace) -> None
s = [] # type: List[Dict[Text, Any]]
a = args.schema
with open(a, encoding='utf-8') as f:
......
......@@ -5,16 +5,33 @@
specific objects. Processing must resolve relative identifiers to absolute
identifiers using the following rules:
* If an identifier URI is prefixed with `#` it is a URI relative
fragment identifier. It is resolved relative to the base URI by setting
or replacing the fragment portion of the base URI.
* If an identifier URI does not contain a scheme and is not prefixed `#` it
is a parent relative fragment identifier. It is resolved relative to the
base URI by the following rule: if the base URI does not contain a
document fragment, set the fragment portion of the base URI. If the base
URI does contain a document fragment, append a slash `/` followed by the
identifier field to the fragment portion of the base URI.
* If an identifier URI begins with `#` it is a current document
fragment identifier. It is resolved relative to the base URI by
setting or replacing the fragment portion of the base URI.
* If an identifier URI contains `#` in some other position is a
relative URI with fragment identifier. It is resolved relative
to the base URI by stripping the last path segment from the base
URI and adding the identifier followed by the fragment.
* If an identifier URI does not contain a scheme and does not
contain `#` it is a parent relative fragment identifier.
* If an identifier URI is a parent relative fragment identifier
and the base URI does not contain a document fragment, set the
document fragment on the base URI.
* If an identifier URI is a parent relative fragment identifier
and the object containing this identifier is assigned to a
parent object field defined with `subscope` in
`jsonldPredicate`, append a slash `/` to the base URI fragment
followed by the value of the parent field `subscope`. Then
append the identifier as described in the next rule.
* If an identifier URI is a parent relative fragment identifier
and the base URI contains a document fragment, append a slash
`/` to the fragment followed by the identifier field to the
fragment portion of the base URI.
* If an identifier URI begins with a namespace prefix declared in
`$namespaces` followed by a colon `:`, the prefix and colon must be
......
......@@ -14,7 +14,12 @@
},
{
"id": "http://example.com/acid#six",
},
{
"subscopeField": {
"id": "http://example.com/base#one/thisIsASubscope/seven"
}
}
]
],
}
}
......@@ -9,6 +9,15 @@
"name": "id",
"type": "string",
"jsonldPredicate": "@id"
}]}, {
"name": "SubscopeType",
"type": "record",
"fields": [{
"name": "subscopeField",
"type": "ExampleType",
"jsonldPredicate": {
"subscope": "thisIsASubscope"
}
}]
}]
}
......@@ -14,7 +14,12 @@
},
{
"id": "acid:six",
},
{
"subscopeField": {
"id": "seven"
}
}
]
],
}
}