CHANGES.txt

 Changes
 =======
 
+v1.2 - july 7, 2018
+-------------------
+
+See closed issues in Milestone 1.2: https://github.com/geopython/stetl/milestone/8?closed=1
+
+Most important changes are related to deployment in Docker and Kubernetes environments, dealing
+with (env) variables, Stetl arguments and logging, for example:
+
+- issue #71: Allow Environment vars to substitute/override config template arg-variables
+- issue #72: Allow multiple -a args for Stetl main prog. Allowing multiple -a arguments allows
+  for more simpler overriding of for example default options.
+- #68 Stetl should not output passwords and other particular data in its log
+
 v1.1.1 - november 7, 2017
 -------------------------
 

PKG-INFO

-Metadata-Version: 1.1
+Metadata-Version: 1.2
 Name: Stetl
-Version: 1.1
-Summary: Stetl provides transformation for spatial data
+Version: 1.2
+Summary: Transformation and conversion framework (ETL) mainly for geospatial data
 Home-page: http://github.com/geopython/stetl
 Author: Just van den Broecke
 Author-email: justb4@gmail.com
+Maintainer: Just van den Broecke
+Maintainer-email: justb4@gmail.com
 License: GNU GPL v3
 Description: # Stetl - Streaming ETL
         
@@ -12,7 +14,7 @@ Description: # Stetl - Streaming ETL
         
         [![Build Status](https://travis-ci.org/geopython/stetl.png)](https://travis-ci.org/geopython/stetl)
         [![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)](http://stetl.readthedocs.org/en/latest)
-        [![Gitter Chat](http://img.shields.io/badge/chat-online-brightgreen.svg)](https://gitter.im/justb4/stetl)
+        [![Gitter Chat](http://img.shields.io/badge/chat-online-brightgreen.svg)](https://gitter.im/geopython/stetl)
         
         Notice: the Stetl GH repo is now at the [GeoPython GH organization](https://github.com/geopython).
         
@@ -54,7 +56,7 @@ Description: # Stetl - Streaming ETL
         
         Most of the data conversions within the [Dutch NLExtract Project](https://github.com/nlextract/NLExtract) apply Stetl.
         
-        Stetl also proved to be very effective in [IoT-related transformations involving the SensorWeb/SOS](https://github.com/Geonovum/smartemission).
+        Stetl also proved to be very effective in [IoT-related transformations involving the SensorWeb/SOS](https://github.com/smartemission).
         
         ## Examples
         
@@ -63,7 +65,7 @@ Description: # Stetl - Streaming ETL
         
         ## Installation
          
-        Stetl can be installed via PyPi `pip install stetl` and recently as a [Stetl Docker image](https://hub.docker.com/r/justb4/stetl).
+        Stetl can be installed via PyPi `pip install stetl` and recently as a [Stetl Docker image](https://hub.docker.com/r/geopython/stetl).
         More on [installation in the documentation](http://www.stetl.org/en/latest/install.html).
         
         ## Contributing
@@ -80,7 +82,7 @@ Description: # Stetl - Streaming ETL
         Stetl originated in the INSPIRE-FOSS project: [2009-2013 now archived](https://github.com/justb4/inspire-foss). 
         Since then Stetl evolved into a wider use like
         transforming [Dutch GML-based Open Datasets](https://github.com/nlextract/NLExtract) such as IMGEO/BGT (Large Scale Topography) 
-        and IMKAD/BRK (Cadastral Data).
+        and IMKAD/BRK (Cadastral Data) and [Sensor Data Transformation and Calibration](https://github.com/smartemission/docker-se-stetl).
         
         ## Finally
         
@@ -96,6 +98,19 @@ Description: # Stetl - Streaming ETL
         Changes
         =======
         
+        v1.2 - july 7, 2018
+        -------------------
+        
+        See closed issues in Milestone 1.2: https://github.com/geopython/stetl/milestone/8?closed=1
+        
+        Most important changes are related to deployment in Docker and Kubernetes environments, dealing
+        with (env) variables, Stetl arguments and logging, for example:
+        
+        - issue #71: Allow Environment vars to substitute/override config template arg-variables
+        - issue #72: Allow multiple -a args for Stetl main prog. Allowing multiple -a arguments allows
+          for more simpler overriding of for example default options.
+        - #68 Stetl should not output passwords and other particular data in its log
+        
         v1.1.1 - november 7, 2017
         -------------------------
         
@@ -206,9 +221,9 @@ Description: # Stetl - Streaming ETL
         
         Thanks to Tom Kralidis for helping out to move from personal repo to https://github.com/geopython organization.
         
-Keywords: etl xsl gdal gis vector feature data
+Keywords: etl xsl gdal gis vector feature data gml xml
 Platform: UNKNOWN
-Classifier: Development Status :: 4 - Beta
+Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research

README.md

@@ -4,7 +4,7 @@ Stetl, streaming ETL, pronounced "staedl", is a lightweight ETL-framework for ge
 
 [![Build Status](https://travis-ci.org/geopython/stetl.png)](https://travis-ci.org/geopython/stetl)
 [![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)](http://stetl.readthedocs.org/en/latest)
-[![Gitter Chat](http://img.shields.io/badge/chat-online-brightgreen.svg)](https://gitter.im/justb4/stetl)
+[![Gitter Chat](http://img.shields.io/badge/chat-online-brightgreen.svg)](https://gitter.im/geopython/stetl)
 
 Notice: the Stetl GH repo is now at the [GeoPython GH organization](https://github.com/geopython).
 
@@ -46,7 +46,7 @@ of GML/XML-based National geo-datasets to for example PostGIS.
 
 Most of the data conversions within the [Dutch NLExtract Project](https://github.com/nlextract/NLExtract) apply Stetl.
 
-Stetl also proved to be very effective in [IoT-related transformations involving the SensorWeb/SOS](https://github.com/Geonovum/smartemission).
+Stetl also proved to be very effective in [IoT-related transformations involving the SensorWeb/SOS](https://github.com/smartemission).
 
 ## Examples
 
@@ -55,7 +55,7 @@ Best is to start with the [basic examples](examples/basics)
 
 ## Installation
  
-Stetl can be installed via PyPi `pip install stetl` and recently as a [Stetl Docker image](https://hub.docker.com/r/justb4/stetl).
+Stetl can be installed via PyPi `pip install stetl` and recently as a [Stetl Docker image](https://hub.docker.com/r/geopython/stetl).
 More on [installation in the documentation](http://www.stetl.org/en/latest/install.html).
 
 ## Contributing
@@ -72,7 +72,7 @@ review the [guidelines for contributing](CONTRIBUTING.md).
 Stetl originated in the INSPIRE-FOSS project: [2009-2013 now archived](https://github.com/justb4/inspire-foss). 
 Since then Stetl evolved into a wider use like
 transforming [Dutch GML-based Open Datasets](https://github.com/nlextract/NLExtract) such as IMGEO/BGT (Large Scale Topography) 
-and IMKAD/BRK (Cadastral Data).
+and IMKAD/BRK (Cadastral Data) and [Sensor Data Transformation and Calibration](https://github.com/smartemission/docker-se-stetl).
 
 ## Finally
 

VERSION.txt

-1.1
\ No newline at end of file
+1.2
\ No newline at end of file

bin/stetl

@@ -8,6 +8,7 @@
 from stetl.main import parse_args
 from stetl.etl import ETL
 from stetl.util import Util
+import sys
 
 log = Util.get_log('main')
 
@@ -18,12 +19,12 @@ def main():
     Args:
        -c  --config <config_file>  the Stetl config file.
        -s  --section <section_name> the section in the Stetl config (ini) file to execute (default is [etl]).
-       -a  --args <arglist> substitutable args for symbolic, {arg}, values in Stetl config file, in format "arg1=foo arg2=bar" etc.
+       -a  --args <arglist> zero or more substitutable args for symbolic, {arg}, values in Stetl config file, in format -a arg1=foo -a arg2=bar etc.
        -h  --help <subject> Get component documentation like its configuration parameters, e.g. stetl doc stetl.inputs.fileinput.FileInput
 
     """
 
-    args = parse_args()
+    args = parse_args(sys.argv[1:])
 
     if args.config_file:
         # Do the ETL

debian/changelog

+python-stetl (1.2+ds-1~bpo9+1) stretch-backports; urgency=medium
+
+  * Rebuild for stretch-backports.
+
+ -- Bas Couwenberg <sebastic@debian.org>  Tue, 10 Jul 2018 07:14:36 +0200
+
+python-stetl (1.2+ds-1) unstable; urgency=medium
+
+  * New upstream release.
+  * Update copyright-format URL to use HTTPS.
+  * Fix deprecated source override location.
+  * Update Vcs-* URLs for Salsa.
+  * Bump Standards-Version to 4.1.5, no changes.
+  * Add module import tests to autopkgtest configuration.
+  * Add override for dependency-on-python-version-marked-for-end-of-life.
+  * Strip trailing whitespace from control & rules files.
+  * Drop reproducible_build.patch, applied upstream.
+
+ -- Bas Couwenberg <sebastic@debian.org>  Sat, 07 Jul 2018 19:30:07 +0200
+
 python-stetl (1.1+ds-2~bpo9+1) stretch-backports; urgency=medium
 
   * Rebuild for stretch-backports.

debian/control

@@ -23,9 +23,9 @@ Build-Depends: debhelper (>= 9),
                docbook-xsl,
                docbook-xml,
                xsltproc
-Standards-Version: 4.1.1
-Vcs-Browser: https://anonscm.debian.org/cgit/pkg-grass/python-stetl.git
-Vcs-Git: https://anonscm.debian.org/git/pkg-grass/python-stetl.git -b stretch-backports
+Standards-Version: 4.1.5
+Vcs-Browser: https://salsa.debian.org/debian-gis-team/python-stetl
+Vcs-Git: https://salsa.debian.org/debian-gis-team/python-stetl.git -b stretch-backports
 Homepage: http://stetl.org/
 
 Package: python-stetl
@@ -79,4 +79,3 @@ Description: Streaming ETL - Commandline utility
  file or even WFS-T a geospatial protocol to publish GML to a server).
  .
  This package contains the stetl utility.
-

debian/copyright

-Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: Stetl
 Upstream-Contact: Just van den Broecke <just@justobjects.nl>
 Source: http://pypi.debian.net/Stetl/

debian/patches/reproducible_build.patch

-Description: Make the build reproducible
-Author: Chris Lamb <lamby@debian.org>
-Last-Update: 2017-11-08
-Bug-Debian: https://bugs.debian.org/881217
-Forwarded: https://github.com/geopython/stetl/pull/64
-
---- a/stetl/filters/templatingfilter.py
-+++ b/stetl/filters/templatingfilter.py
-@@ -139,7 +139,7 @@ class Jinja2TemplatingFilter(TemplatingF
-     # Applying Decorator pattern with the Config class to provide
-     # read-only config values from the configured properties.
- 
--    @Config(ptype=str, default=[os.getcwd()], required=False)
-+    @Config(ptype=str, default=None, required=False)
-     def template_search_paths(self):
-         """
-         List of directories where to search for templates, default is current working directory only.
-@@ -207,7 +207,7 @@ class Jinja2TemplatingFilter(TemplatingF
-                     raise e
- 
-         # Load and Init Template once
--        loader = FileSystemLoader(self.template_search_paths)
-+        loader = FileSystemLoader(self.template_search_paths or [os.getcwd()])
-         self.jinja2_env = Environment(loader=loader, extensions=['jinja2.ext.do'], lstrip_blocks=True, trim_blocks=True)
- 
-         # Register additional Filters on the template environment by updating the filters dict:

debian/patches/series

-reproducible_build.patch

debian/rules

@@ -50,4 +50,3 @@ override_dh_installexamples:
 
 	# Remove executable bit
 	find debian/*/usr/share/doc/*/examples/ -type f -exec chmod 644 {} \;
-

debian/stetl.lintian-overrides

+# Python 3 is not supported yet: https://github.com/geopython/stetl/issues/23
+stetl: dependency-on-python-version-marked-for-end-of-life (Depends: python)
+

debian/tests/control

 # Test installability
 Depends: @
 Test-Command: /bin/true
+
+# Test module import (Python 2)
+Depends: python-all, python-stetl
+Test-Command: set -e ; for py in $(pyversions -r 2>/dev/null) ; do cd "$ADTTMP" ; echo "Testing with $py:" ; $py -c "import stetl; print(stetl)" ; done

docs/code.rst

@@ -111,6 +111,10 @@ Components: Filters
    :members:
    :show-inheritance:
 
+.. automodule:: stetl.filters.sieve
+   :members:
+   :show-inheritance:
+
 .. automodule:: stetl.filters.stringfilter
    :members:
    :show-inheritance:
@@ -131,6 +135,30 @@ Components: Filters
    :members:
    :show-inheritance:
 
+.. automodule:: stetl.filters.execfilter
+   :members:
+   :show-inheritance:
+
+.. automodule:: stetl.filters.nullfilter
+   :members:
+   :show-inheritance:
+
+.. automodule:: stetl.filters.packetbuffer
+   :members:
+   :show-inheritance:
+
+.. automodule:: stetl.filters.packetwriter
+   :members:
+   :show-inheritance:
+
+.. automodule:: stetl.filters.regexfilter
+   :members:
+   :show-inheritance:
+
+.. automodule:: stetl.filters.zipfileextractor
+   :members:
+   :show-inheritance:
+
 Components: Outputs
 -------------------
 

docs/conf.py

@@ -52,9 +52,9 @@ copyright = u'2013+, Just van den Broecke'
 # built documents.
 #
 # The short X.Y version.
-version = '1.1'
+version = '1.2-dev'
 # The full version, including alpha/beta/rc tags.
-release = '1.1'
+release = '1.2-dev'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/contact.rst

@@ -9,5 +9,4 @@ All development is done via GitHub: see https://github.com/geopython/stetl.
 
 Contact the main author Just van den Broecke via email at just@justobjects.nl.
 
-
-
+Online chat via Gitter: https://gitter.im/geopython/stetl

docs/using.rst

@@ -234,9 +234,15 @@ or even OGC WPS servers (planned).
 
 Reusable Stetl Configs
 ----------------------
-What we saw in the last example is that it is hard to reuse this `etl.cfg` when we have for example a different input file
-or want to map to different output files. For this Stetl supports `parameter substitution`. Here command line parameters are substituted
-for variables in `etl.cfg`. A variable is declared between curly brackets like `{out_xml}`. See
+What we saw in the last example is that it is hard to reuse this `etl.cfg`
+when we have for example a different input file
+or want to map to different output files.
+For this Stetl supports `config parameter substitution`.
+
+Dynamic or secret (e.g. database credentials) parameters in `etl.cfg` are declared
+symbolically and substituted at runtime via the commandline or the OS environment.
+
+A variable is declared between curly brackets like `{out_xml}`. See
 example `6_cmdargs <https://github.com/geopython/stetl/tree/master/examples/basics/6_cmdargs>`_. ::
 
 	[etl]
@@ -254,12 +260,13 @@ example `6_cmdargs <https://github.com/geopython/stetl/tree/master/examples/basi
 	class = outputs.fileoutput.FileOutput
 	file_path = {out_xml}
 
-Note the symbolic input, xsl and output files. We can now perform this ETL using the `stetl -a option` in two ways.
+Note the symbolic input, xsl and output files. We can now perform
+the ETL using the `stetl -a option` in two basic ways.
 One, passing the arguments on the commandline, like ::
 
 	stetl -c etl.cfg -a "in_xml=input/cities.xml in_xsl=cities2gml.xsl out_xml=output/gmlcities.gml"
 
-Two, passing the arguments in a properties file, here called `etl.args` (the name of the suffix .args is not significant). ::
+Two, passing the arguments in a properties file, here called `etl.args` (the name of the suffix .args is not significant, could be .env as well). ::
 
 	stetl -c etl.cfg -a etl.args
 
@@ -270,7 +277,41 @@ Where the content of the `etl.args` properties file is: ::
 	in_xsl=cities2gml.xsl
 	out_xml=output/gmlcities.gml
 
-This makes an ETL chain highly reusable. A very elaborate Stetl config with parameter substitution can be seen in the
+It is also possible to specify **multiple -a arguments**. This provides for situations
+where a `default.args` contains all default arguments and a `my.args` or explicit `-a` settings
+that override the default values in `default.args`. Overriding is determined by the order of
+the `-a` arguments. Examples: ::
+
+	stetl -c etl.cfg -a default.args -a my.args
+	stetl -c etl.cfg -a default.args -a "db_user=docker db_password=pass"
+	stetl -c etl.cfg -a default.args -a db_user=docker -a db_password=pass
+
+It is also possible to pass these key/value pairs via OS Environment variables.
+This is especially handy in Docker-based deployments like Docker Compose and Kubernetes.
+In this case the variable names need to be prepended with `STETL_` or `stetl_` as
+to not mix-up with other non-related OS-env vars. A mixture of commandline args (file)
+and environment vars is possible. The rule is that
+*OS Environment variables always override/overrule arguments specified with -a option(s)*.
+
+For example, the above args could also be passed as follows: ::
+
+	export stetl_in_xml="input/cities.xml"
+	export stetl_in_xsl="cities2gml.xsl"
+	export stetl_out_xml="output/gmlcities.gml"
+	stetl -c etl.cfg
+
+or only override the input file name `in_xml` from `etl.args`: ::
+
+	export stetl_in_xml="input/cities2.xml"
+	stetl -c etl.cfg -a etl.args
+
+or even with multiple `-a args`: ::
+
+	export stetl_in_xml="input/cities2.xml"
+	stetl -c etl.cfg -a etl.args -a my.args
+
+This makes an ETL chain highly reusable.
+A very elaborate Stetl config with parameter substitution can be seen in the
 `Top10NL ETL <https://github.com/geopython/stetl/blob/master/examples/top10nl/etl-top10nl.cfg>`_.
 
 Connection Compatibility

examples/basics/17_sieve/etl.cfg

+# Trivial example Sieve filter.
+# The input data is in input/cities.csv.
+# We sieve out (passthrough) all records where city attr value
+# matches "amsterdam" or "otterlo".
+
+[etl]
+chains = input_csv|attr_value_sieve|output_std,
+		input_csv|attr_value_sieve|output_file
+
+[input_csv]
+class = inputs.fileinput.CsvFileInput
+file_path = input/cities.csv
+output_format = record_array
+
+[attr_value_sieve]
+class = filters.sieve.AttrValueRecordSieve
+input_format = record_array
+output_format = record_array
+attr_name = city
+attr_values = amsterdam,otterlo
+
+[output_std]
+class = outputs.standardoutput.StandardOutput
+
+[output_file]
+class = outputs.fileoutput.FileOutput
+file_path = output/cities_sieved.txt
+