.travis.yml

@@ -5,7 +5,7 @@ env:
         - PYTHON_VERSION=$TRAVIS_PYTHON_VERSION
         - NUMPY_VERSION=stable
         - MAIN_CMD='python setup.py'
-        - CONDA_DEPENDENCIES='xarray dask toolz Cython sphinx cartopy pillow matplotlib scipy pyyaml pyproj pyresample coveralls coverage codecov behave netcdf4 h5py h5netcdf gdal rasterio imageio pyhdf mock libtiff pycoast pydecorate'
+        - CONDA_DEPENDENCIES='xarray dask toolz Cython sphinx cartopy pillow matplotlib scipy pyyaml pyproj pyresample coveralls coverage codecov behave netcdf4 h5py h5netcdf gdal rasterio<1.0.14 imageio pyhdf mock libtiff pycoast pydecorate'
         - PIP_DEPENDENCIES='trollsift trollimage pyspectral pyorbital libtiff'
         - SETUP_XVFB=False
         - EVENT_TYPE='push pull_request'

CHANGELOG.md

+## Version 0.11.2 (2019/01/28)
+
+### Issues Closed
+
+* [Issue 584](https://github.com/pytroll/satpy/issues/584) - DayNightCompositor does not work with eg overview_sun as the day part ([PR 593](https://github.com/pytroll/satpy/pull/593))
+* [Issue 577](https://github.com/pytroll/satpy/issues/577) - Creation of composites using `sunz_corrected` modifier fails with VIIRS SDR data
+* [Issue 569](https://github.com/pytroll/satpy/issues/569) - Can not show or save ABI true color image (RuntimeWarning: invalid value encountered in log)
+* [Issue 531](https://github.com/pytroll/satpy/issues/531) - Mask space pixels in AHI HSD reader ([PR 592](https://github.com/pytroll/satpy/pull/592))
+* [Issue 106](https://github.com/pytroll/satpy/issues/106) - Warnings
+
+In this release 5 issues were closed.
+
+### Pull Requests Merged
+
+#### Bugs fixed
+
+* [PR 594](https://github.com/pytroll/satpy/pull/594) - Fix VIIRS L1B reader not using standard 'y' and 'x' dimension names
+* [PR 593](https://github.com/pytroll/satpy/pull/593) - Fix sunz_corrected modifier adding unnecessary x and y coordinates ([587](https://github.com/pytroll/satpy/issues/587), [584](https://github.com/pytroll/satpy/issues/584))
+* [PR 592](https://github.com/pytroll/satpy/pull/592) - Fix masking of AHI HSD space pixels ([531](https://github.com/pytroll/satpy/issues/531))
+* [PR 589](https://github.com/pytroll/satpy/pull/589) - Fix dask not importing sharedict automatically in dask 1.1+
+* [PR 588](https://github.com/pytroll/satpy/pull/588) - Fix start_time type in seviri_l1b_nc reader
+* [PR 585](https://github.com/pytroll/satpy/pull/585) - Fix geotiff writer not using fill_value from writer YAML config
+* [PR 572](https://github.com/pytroll/satpy/pull/572) - Fix VIIRS SDR masking and distracting colors in composites
+* [PR 570](https://github.com/pytroll/satpy/pull/570) - Fix CF epoch for xarray compat
+* [PR 563](https://github.com/pytroll/satpy/pull/563) - Fix StopIteration and python 3.7 compatibility issue in MultiScene
+* [PR 554](https://github.com/pytroll/satpy/pull/554) - Fix AreaDefinition usage to work with newer versions of pyresample
+
+#### Features added
+
+* [PR 561](https://github.com/pytroll/satpy/pull/561) - Add AHI HRIT B07 files for high resolution night data
+
+#### Documentation changes
+
+* [PR 590](https://github.com/pytroll/satpy/pull/590) - Add FAQ page to docs
+* [PR 575](https://github.com/pytroll/satpy/pull/575) - Add page for data download resources
+* [PR 574](https://github.com/pytroll/satpy/pull/574) - Add code of conduct
+
+In this release 14 pull requests were closed.
+
+
 ## Version 0.11.1 (2018/12/27)
 
 ### Pull Requests Merged

CODE_OF_CONDUCT.md

+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at <pytroll-conduct@lists.wisc.edu>. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.rst

@@ -84,6 +84,7 @@ hurt readability. SatPy follows
 for all code API documentation. When in doubt use the existing code as a
 guide for how coding should be done.
 
+.. _dev_help:
 
 How do I get help?
 ==================
@@ -105,3 +106,32 @@ After talking to the SatPy developers any additional work like code or
 documentation changes can be provided as a GitHub
 `Pull Request <https://help.github.com/articles/creating-a-pull-request/>`_.
 
+Code of Conduct
+===============
+
+SatPy follows the same code of conduct as the PyTroll project. For reference
+it is copied to this repository in
+`CODE_OF_CONDUCT.md <https://github.com/pytroll/satpy/blob/master/CODE_OF_CONDUCT.md>`_.
+
+As stated in the PyTroll home page, this code of conduct applies to the
+project space (GitHub) as well as the public space online and offline when
+an individual is representing the project or the community. Online examples
+of this include the PyTroll Slack team, mailing list, and the PyTroll twitter
+account. This code of conduct also applies to in-person situations like
+PyTroll Contributor Weeks (PCW), conference meet-ups, or any other time when
+the project is being represented.
+
+Any violations of this code of conduct will be handled by the core maintainers
+of the project including David Hoese, Martin Raspaud, and Adam Dybbroe.
+If you wish to report one of the maintainers for a violation and are
+not comfortable with them seeing it, please contact one or more of the other
+maintainers to report the violation. Responses to violations will be
+determined by the maintainers and may include one or more of the following:
+
+- Verbal warning
+- Ask for public apology
+- Temporary or permanent ban from in-person events
+- Temporary or permanent ban from online communication (Slack, mailing list, etc)
+
+For details see the official
+`code of conduct document <https://github.com/pytroll/satpy/blob/master/CODE_OF_CONDUCT.md>`_.

README.rst

@@ -24,3 +24,49 @@ to resample data to different uniform areas or grids.
 
 The documentation is available at
 http://satpy.readthedocs.org/.
+
+Installation
+------------
+
+SatPy can be installed from PyPI with pip:
+
+.. code-block:: bash
+
+    pip install satpy
+
+
+It is also available from `conda-forge` for conda installations:
+
+.. code-block:: bash
+
+    conda install -c conda-forge satpy
+
+Code of Conduct
+---------------
+
+SatPy follows the same code of conduct as the PyTroll project. For reference
+it is copied to this repository in CODE_OF_CONDUCT.md_.
+
+As stated in the PyTroll home page, this code of conduct applies to the
+project space (GitHub) as well as the public space online and offline when
+an individual is representing the project or the community. Online examples
+of this include the PyTroll Slack team, mailing list, and the PyTroll twitter
+account. This code of conduct also applies to in-person situations like
+PyTroll Contributor Weeks (PCW), conference meet-ups, or any other time when
+the project is being represented.
+
+Any violations of this code of conduct will be handled by the core maintainers
+of the project including David Hoese, Martin Raspaud, and Adam Dybbroe.
+If you wish to report one of the maintainers for a violation and are
+not comfortable with them seeing it, please contact one or more of the other
+maintainers to report the violation. Responses to violations will be
+determined by the maintainers and may include one or more of the following:
+
+- Verbal warning
+- Ask for public apology
+- Temporary or permanent ban from in-person events
+- Temporary or permanent ban from online communication (Slack, mailing list, etc)
+
+For details see the official CODE_OF_CONDUCT.md_.
+
+.. _CODE_OF_CONDUCT.md: ./CODE_OF_CONDUCT.md

appveyor.yml

@@ -3,7 +3,7 @@ environment:
     PYTHON: "C:\\conda"
     MINICONDA_VERSION: "latest"
     CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\ci-helpers\\appveyor\\windows_sdk.cmd"
-    CONDA_DEPENDENCIES: "xarray dask toolz Cython sphinx cartopy pillow matplotlib scipy pyyaml pyproj pyresample coverage netcdf4 h5py h5netcdf gdal rasterio imageio pyhdf mock libtiff pycoast pydecorate"
+    CONDA_DEPENDENCIES: "xarray dask toolz Cython sphinx cartopy pillow matplotlib scipy pyyaml pyproj pyresample coverage netcdf4 h5py h5netcdf gdal 'rasterio<1.0.14' imageio pyhdf mock libtiff pycoast pydecorate"
     PIP_DEPENDENCIES: "trollsift trollimage pyspectral pyorbital libtiff"
     CONDA_CHANNELS: "conda-forge"
 

debian/changelog

-satpy (0.11.1-3) UNRELEASED; urgency=medium
+satpy (0.11.2-1) UNRELEASED; urgency=medium
 
+  * New upstream release (Closes: #919566).
   * debian/patches:
-    - new 0005-Reproducible-build.patch (Closes: #919566)
+    - drop 0002-Compatibility-with-Python-3.7.patch and 0004-Fix-cf-epoch.patch:
+      applied upstream
+    - drop 0003-Skip-tests-that-require-pydecorate.patch: no longer necessary
+      (pydecorate is now available)
 
  -- Antonio Valentino <antonio.valentino@tiscali.it>  Fri, 18 Jan 2019 07:45:38 +0000
 

debian/patches/0002-Compatibility-with-Python-3.7.patch

-From: Antonio Valentino <antonio.valentino@tiscali.it>
-Date: Sat, 22 Dec 2018 18:29:46 +0100
-Subject: Compatibility with Python 3.7
-
----
- satpy/multiscene.py | 5 ++++-
- 1 file changed, 4 insertions(+), 1 deletion(-)
-
-diff --git a/satpy/multiscene.py b/satpy/multiscene.py
-index 8602357..1b0bf22 100644
---- a/satpy/multiscene.py
-+++ b/satpy/multiscene.py
-@@ -122,7 +122,10 @@ class _SceneGenerator(object):
-         self._dataset_idx[ds_id] = idx = 0
-         while True:
-             if idx >= len(self._scene_cache):
--                scn = next(self._self_iter)
-+                try:
-+                    scn = next(self._self_iter)
-+                except StopIteration:
-+                    return
-             else:
-                 scn = self._scene_cache[idx]
-             yield scn.get(ds_id)

debian/patches/0003-Skip-tests-that-require-pydecorate.patch

-From: Antonio Valentino <antonio.valentino@tiscali.it>
-Date: Tue, 25 Dec 2018 19:27:25 +0000
-Subject: Skip tests that require pydecorate
-
----
- satpy/tests/test_writers.py | 7 +++++++
- 1 file changed, 7 insertions(+)
-
-diff --git a/satpy/tests/test_writers.py b/satpy/tests/test_writers.py
-index 741776e..81bed45 100644
---- a/satpy/tests/test_writers.py
-+++ b/satpy/tests/test_writers.py
-@@ -36,6 +36,12 @@ except ImportError:
-     import mock
- 
- 
-+try:
-+    import pydecorate
-+except ImportError:
-+    pydecorate = False
-+
-+
- def mkdir_p(path):
-     """Make directories."""
-     if not path or path == '.':
-@@ -549,6 +555,7 @@ class TestBaseWriter(unittest.TestCase):
-         self.assertTrue(os.path.isfile(os.path.join(self.base_dir, exp_fn)))
- 
- 
-+@unittest.skipIf(not pydecorate, 'pydecorate not available')
- class TestOverlays(unittest.TestCase):
-     """Tests for add_overlay and add_decorate functions."""
- 

debian/patches/0004-Fix-cf-epoch.patch

-From: Antonio Valentino <antonio.valentino@tiscali.it>
-Date: Tue, 15 Jan 2019 06:26:23 +0000
-Subject: Fix cf epoch
-
-Fix CF epoch for xarray compat.
-See also upstream commit 58bbb0108ecb7abb99e0688a9342512e8ba65709
-(https://github.com/pytroll/satpy).
----
- satpy/writers/cf_writer.py | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/satpy/writers/cf_writer.py b/satpy/writers/cf_writer.py
-index c8243c5..dcf2f52 100644
---- a/satpy/writers/cf_writer.py
-+++ b/satpy/writers/cf_writer.py
-@@ -33,7 +33,7 @@ from satpy.writers import Writer
- 
- logger = logging.getLogger(__name__)
- 
--EPOCH = u"seconds since 1970-01-01 00:00:00 +00:00"
-+EPOCH = u"seconds since 1970-01-01 00:00:00"
- 
- 
- def omerc2cf(area):

debian/patches/0005-Reproducible-build.patch

-From: Antonio Valentino <antonio.valentino@tiscali.it>
-Date: Fri, 18 Jan 2019 07:44:58 +0000
-Subject: Reproducible build
-
----
- satpy/composites/__init__.py | 4 +++-
- satpy/config.py              | 4 +++-
- satpy/readers/__init__.py    | 8 ++++++--
- satpy/scene.py               | 4 +++-
- 4 files changed, 15 insertions(+), 5 deletions(-)
-
-diff --git a/satpy/composites/__init__.py b/satpy/composites/__init__.py
-index b02dd86..4c5ede0 100644
---- a/satpy/composites/__init__.py
-+++ b/satpy/composites/__init__.py
-@@ -60,7 +60,9 @@ class IncompatibleTimes(Exception):
- class CompositorLoader(object):
-     """Read composites using the configuration files on disk."""
- 
--    def __init__(self, ppp_config_dir=CONFIG_PATH):
-+    def __init__(self, ppp_config_dir=None):
-+        if ppp_config_dir is None:
-+            ppp_config_dir = CONFIG_PATH
-         self.modifiers = {}
-         self.compositors = {}
-         self.ppp_config_dir = ppp_config_dir
-diff --git a/satpy/config.py b/satpy/config.py
-index c36566b..ed7cf1e 100644
---- a/satpy/config.py
-+++ b/satpy/config.py
-@@ -40,7 +40,9 @@ BASE_PATH = os.path.dirname(os.path.realpath(__file__))
- PACKAGE_CONFIG_PATH = os.path.join(BASE_PATH, 'etc')
- 
- 
--def get_environ_config_dir(default=PACKAGE_CONFIG_PATH):
-+def get_environ_config_dir(default=None):
-+    if default is None:
-+        default = PACKAGE_CONFIG_PATH
-     return os.environ.get('PPP_CONFIG_DIR', default)
- 
- 
-diff --git a/satpy/readers/__init__.py b/satpy/readers/__init__.py
-index 15423a4..7a2dcc5 100644
---- a/satpy/readers/__init__.py
-+++ b/satpy/readers/__init__.py
-@@ -515,7 +515,7 @@ def available_readers(as_dict=False):
- 
- 
- def find_files_and_readers(start_time=None, end_time=None, base_dir=None,
--                           reader=None, sensor=None, ppp_config_dir=get_environ_config_dir(),
-+                           reader=None, sensor=None, ppp_config_dir=None,
-                            filter_parameters=None, reader_kwargs=None):
-     """Find on-disk files matching the provided parameters.
- 
-@@ -553,6 +553,8 @@ def find_files_and_readers(start_time=None, end_time=None, base_dir=None,
-     Returns: Dictionary mapping reader name string to list of filenames
- 
-     """
-+    if ppp_config_dir is None:
-+        ppp_config_dir = get_environ_config_dir()
-     reader_files = {}
-     reader_kwargs = reader_kwargs or {}
-     filter_parameters = filter_parameters or reader_kwargs.get('filter_parameters', {})
-@@ -595,7 +597,7 @@ def find_files_and_readers(start_time=None, end_time=None, base_dir=None,
- 
- 
- def load_readers(filenames=None, reader=None, reader_kwargs=None,
--                 ppp_config_dir=get_environ_config_dir()):
-+                 ppp_config_dir=None):
-     """Create specified readers and assign files to them.
- 
-     Args:
-@@ -612,6 +614,8 @@ def load_readers(filenames=None, reader=None, reader_kwargs=None,
-     """
-     reader_instances = {}
-     reader_kwargs = reader_kwargs or {}
-+    if ppp_config_dir is None:
-+        ppp_config_dir = get_environ_config_dir()
- 
-     if not filenames and not reader:
-         # used for an empty Scene
-diff --git a/satpy/scene.py b/satpy/scene.py
-index 31041bb..8f06d9d 100644
---- a/satpy/scene.py
-+++ b/satpy/scene.py
-@@ -72,7 +72,7 @@ class Scene(MetadataObject):
-     """
- 
-     def __init__(self, filenames=None, reader=None, filter_parameters=None, reader_kwargs=None,
--                 ppp_config_dir=get_environ_config_dir(),
-+                 ppp_config_dir=None,
-                  base_dir=None,
-                  sensor=None,
-                  start_time=None,
-@@ -107,6 +107,8 @@ class Scene(MetadataObject):
- 
-         """
-         super(Scene, self).__init__()
-+        if ppp_config_dir is None:
-+            ppp_config_dir = get_environ_config_dir()
-         # Set the PPP_CONFIG_DIR in the environment in case it's used elsewhere in pytroll
-         LOG.debug("Setting 'PPP_CONFIG_DIR' to '%s'", ppp_config_dir)
-         os.environ["PPP_CONFIG_DIR"] = self.ppp_config_dir = ppp_config_dir

debian/patches/series

 0001-Fix-pyhdf.patch
-0002-Compatibility-with-Python-3.7.patch
-0003-Skip-tests-that-require-pydecorate.patch
-0004-Fix-cf-epoch.patch
-0005-Reproducible-build.patch

doc/source/data_download.rst

+Downloading Data
+================
+
+One of the main features of SatPy is its ability to read various satellite
+data formats. However, it does not currently provide any functionality for
+downloading data from any remote sources. SatPy assumes all data is available
+through the local system, either as a local directory or network
+mounted file systems. Certain readers that use ``xarray`` to open data files
+may be able to load files from remote systems by using OpenDAP or similar
+protocols.
+
+As a user there are two options for getting access to data:
+
+1. Download data to your local machine.
+2. Connect to a remote system that already has access to data.
+
+The most common case of a remote system having access to data is with a cloud
+computing service like Google Cloud Platform (GCP) or Amazon Web
+Services (AWS). Another possible case is an organization having direct
+broadcast antennas where they receive data directly from the satellite or
+satellite mission organization (NOAA, NASA, EUMETSAT, etc). In these cases
+data is usually available as a mounted network file system and can be accessed
+like a normal local path (with the added latency of network communications).
+
+Below are some data sources that provide data that can be read by SatPy. If
+you know of others please let us know by either creating a GitHub issue or
+pull request.
+
+NOAA GOES on Amazon Web Services
+--------------------------------
+
+* `Resource Description <https://registry.opendata.aws/noaa-goes/>`__
+* `Data Browser <http://noaa-goes16.s3.amazonaws.com/index.html>`__
+* Associated Readers: ``abi_l1b``
+
+In addition ot the pages above, Brian Blaylock has prepared some instructions
+for using the ``rclone`` tool for downloading AWS data to a local machine. The
+instructions can be found
+`here <https://github.com/blaylockbk/pyBKB_v3/blob/master/rclone_howto.md>`_.
+
+NOAA GOES on Google Cloud Platform
+----------------------------------
+
+GOES-16
+^^^^^^^
+
+* `Resource Description <https://console.cloud.google.com/marketplace/details/noaa-public/goes-16>`__
+* `Data Browser <https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16>`__
+* Associated Readers: ``abi_l1b``
+
+GOES-17
+^^^^^^^
+
+* `Resource Description <https://console.cloud.google.com/marketplace/details/noaa-public/goes-17>`__
+* `Data Browser <https://console.cloud.google.com/storage/browser/gcp-public-data-goes-17>`__
+* Associated Readers: ``abi_l1b``
+
+NOAA CLASS
+----------
+
+* `Data Ordering <https://www.class.ncdc.noaa.gov>`__
+* Associated Readers: ``viirs_sdr``
+
+NASA VIIRS Atmosphere SIPS
+--------------------------
+
+* `Resource Description <https://sips.ssec.wisc.edu/>`__
+* Associated Readers: ``viirs_l1b``
+
+EUMETSAT Data Center
+--------------------
+
+* `Data Ordering <https://eoportal.eumetsat.int>`__

doc/source/faq.rst

+FAQ
+===
+
+Below you'll find frequently asked questions, performance tips, and other
+topics that don't really fit in to the rest of the SatPy documentation.
+
+If you have any other questions that aren't answered here feel free to make
+an issue on GitHub or talk to us on the Slack team or mailing list. See the
+:ref:`contributing <dev_help>` documentation for more information.
+
+.. contents:: Topics
+    :depth: 1
+    :local:
+
+Why is SatPy slow on my powerful machine?
+-----------------------------------------
+
+SatPy depends heavily on the dask library for its performance. However,
+on some systems dask's default settings can actually hurt performance.
+By default dask will create a "worker" for each logical core on your
+system. In most systems you have twice as many logical cores
+(also known as threaded cores) as physical cores. Managing and communicating
+with all of these workers can slow down dask, especially when they aren't all
+being used by most SatPy calculations. One option is to limit the number of
+workers by doing the following at the **top** of your python code:
+
+.. code-block:: python
+
+    import dask
+    from multiprocessing.pool import ThreadPool
+    dask.config.set(pool=ThreadPool(8))
+    # all other SatPy imports and code
+
+This will limit dask to using 8 workers. Typically numbers between 4 and 8
+are good starting points.
+
+Similarly, if you have many workers processing large chunks of data you may
+be using much more memory than you expect. If you limit the number of workers
+*and* the size of the data chunks being processed by each worker you can
+reduce the overall memory usage. Default chunk size can be configured in SatPy
+by setting the following environment variable:
+
+.. code-block:: bash
+
+    export PYTROLL_CHUNK_SIZE=2048
+
+This could also be set inside python using ``os.environ``, but must be set
+**before** SatPy is imported. This value defaults to 4096, meaning each
+chunk of data will be 4096 rows by 4096 columns. In the future setting this
+value will change to be easier to set in python.
+
+How do I avoid memory errors?
+-----------------------------
+
+If your environment is using many dask workers, it may be using more memory
+than it needs to be using. See the "Why is SatPy slow on my powerful machine?"
+question above for more information on changing SatPy's memory usage.

doc/source/index.rst

@@ -33,6 +33,7 @@ installation.
 
     overview
     install
+    data_download
     examples
     quickstart
     readers
@@ -41,8 +42,12 @@ installation.
     writers
     multiscene
     dev_guide/index
-    SatPy API <api/satpy>
 
+.. toctree::
+    :maxdepth: 1
+
+    SatPy API <api/satpy>
+    faq
 
 .. _reader_table:
 

satpy/composites/__init__.py

@@ -60,7 +60,9 @@ class IncompatibleTimes(Exception):
 class CompositorLoader(object):
     """Read composites using the configuration files on disk."""
 
-    def __init__(self, ppp_config_dir=CONFIG_PATH):
+    def __init__(self, ppp_config_dir=None):
+        if ppp_config_dir is None:
+            ppp_config_dir = CONFIG_PATH
         self.modifiers = {}
         self.compositors = {}
         self.ppp_config_dir = ppp_config_dir
@@ -336,19 +338,24 @@ class SunZenithCorrectorBase(CompositeBase):
         key = (vis.attrs["start_time"], area_name)
         tic = time.time()
         LOG.debug("Applying sun zen correction")
-        if len(projectables) == 1:
         coszen = self.coszen.get(key)
-            if coszen is None:
+        if coszen is None and len(projectables) == 1:
+            # we were not given SZA, generate SZA then calculate cos(SZA)
             from pyorbital.astronomy import cos_zen
             LOG.debug("Computing sun zenith angles.")
             lons, lats = vis.attrs["area"].get_lonlats_dask(CHUNK_SIZE)
 
+            coords = {}
+            if 'y' in vis.coords and 'x' in vis.coords:
+                coords['y'] = vis['y']
+                coords['x'] = vis['x']
             coszen = xr.DataArray(cos_zen(vis.attrs["start_time"], lons, lats),
-                                      dims=['y', 'x'], coords=[vis['y'], vis['x']])
+                                  dims=['y', 'x'], coords=coords)
             if self.max_sza is not None:
                 coszen = coszen.where(coszen >= self.max_sza_cos)
             self.coszen[key] = coszen
-        else:
+        elif coszen is None:
+            # we were given the SZA, calculate the cos(SZA)
             coszen = np.cos(np.deg2rad(projectables[1]))
             self.coszen[key] = coszen
 
@@ -691,9 +698,17 @@ class GenericCompositor(CompositeBase):
 
     modes = {1: 'L', 2: 'LA', 3: 'RGB', 4: 'RGBA'}
 
-    def _concat_datasets(self, projectables, mode):
-        projectables = self.check_areas(projectables)
+    def __init__(self, name, common_channel_mask=True, **kwargs):
+        """Collect custom configuration values.
 
+        Args:
+            common_channel_mask (bool): If True, mask all the channels with
+                a mask that combines all the invalid areas of the given data.
+        """
+        self.common_channel_mask = common_channel_mask
+        super(GenericCompositor, self).__init__(name, **kwargs)
+
+    def _concat_datasets(self, projectables, mode):
         try:
             data = xr.concat(projectables, 'bands', coords='minimal')
             data['bands'] = list(mode)
@@ -726,7 +741,11 @@ class GenericCompositor(CompositeBase):
             # num may not be in `self.modes` so only check if we need to
             mode = self.modes[num]
         if len(projectables) > 1:
+            projectables = self.check_areas(projectables)
             data = self._concat_datasets(projectables, mode)
+            # Skip masking if user wants it or a specific alpha channel is given.
+            if self.common_channel_mask and mode[-1] != 'A':
+                data = data.where(data.notnull().all(dim='bands'))
         else:
             data = projectables[0]
 
@@ -1179,6 +1198,7 @@ class RatioSharpenedRGB(GenericCompositor):
             raise ValueError("RatioSharpenedRGB.high_resolution_band must "
                              "be one of ['red', 'green', 'blue', None]. Not "
                              "'{}'".format(self.high_resolution_band))
+        kwargs.setdefault('common_channel_mask', False)
         super(RatioSharpenedRGB, self).__init__(*args, **kwargs)
 
     def _get_band(self, high_res, low_res, color, ratio):

satpy/config.py

@@ -40,7 +40,9 @@ BASE_PATH = os.path.dirname(os.path.realpath(__file__))
 PACKAGE_CONFIG_PATH = os.path.join(BASE_PATH, 'etc')
 
 
-def get_environ_config_dir(default=PACKAGE_CONFIG_PATH):
+def get_environ_config_dir(default=None):
+    if default is None:
+        default = PACKAGE_CONFIG_PATH
     return os.environ.get('PPP_CONFIG_DIR', default)
 
 

satpy/etc/readers/ahi_hrit.yaml

@@ -47,6 +47,15 @@ file_types:
         - 'IMG_DK{area:02d}B06_{start_time:%Y%m%d%H%M}'
 
     hrit_b07:
+      file_reader: !!python/name:satpy.readers.hrit_jma.HRITJMAFileHandler
+        # B07 are high resolution versions of IR4 at night
+        # See section 1.3 of
+      # https://www.data.jma.go.jp/mscweb/en/himawari89/himawari_cast/note/HimawariCast_dataset_20150624_en.pdf
+      file_patterns:
+        - 'IMG_DK{area:02d}B07_{start_time:%Y%m%d%H%M}_{segment:03d}'
+        - 'IMG_DK{area:02d}B07_{start_time:%Y%m%d%H%M}'
+
+    hrit_b07_ir4:
         file_reader: !!python/name:satpy.readers.hrit_jma.HRITJMAFileHandler
         file_patterns:
         - 'IMG_DK{area:02d}IR4_{start_time:%Y%m%d%H%M}_{segment:03d}'
@@ -139,7 +148,7 @@ datasets:
     name: B03
     sensor: ahi
     wavelength: [0.62,0.64,0.66]
-    resolution: 500
+    resolution: 1000
     calibration:
       reflectance:
         standard_name: toa_bidirectional_reflectance
@@ -153,7 +162,7 @@ datasets:
     name: B04
     sensor: ahi
     wavelength: [0.83, 0.85, 0.87]
-    resolution: 1000
+    resolution: 4000
     calibration:
       reflectance:
         standard_name: toa_bidirectional_reflectance
@@ -167,7 +176,7 @@ datasets:
     name: B05
     sensor: ahi
     wavelength: [1.5, 1.6, 1.7]
-    resolution: 2000
+    resolution: 4000
     calibration:
       reflectance:
         standard_name: toa_bidirectional_reflectance
@@ -181,7 +190,7 @@ datasets:
     name: B06
     sensor: ahi
     wavelength: [2.2, 2.3, 2.4]
-    resolution: 2000
+    resolution: 4000
     calibration:
       reflectance:
         standard_name: toa_bidirectional_reflectance
@@ -191,11 +200,12 @@ datasets:
         units: 1
     file_type: hrit_b06
 
-  B07:
+  B07_low_res:
     name: B07
+    resolution: 4000
+#    resolution: 2000
     sensor: ahi
     wavelength: [3.7, 3.9, 4.1]
-    resolution: 2000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -203,13 +213,28 @@ datasets:
       counts:
         standard_name: counts
         units: 1
-    file_type: hrit_b07
+    # FUTURE: Split this in to multiple resolutions so each can be loaded
+    file_type: [hrit_b07, hrit_b07_ir4]
+
+#  B07_high_res:
+#    name: B07
+#    resolution: 2000
+#    sensor: ahi
+#    wavelength: [3.7, 3.9, 4.1]
+#    calibration:
+#      brightness_temperature:
+#        standard_name: toa_brightness_temperature
+#        units: "K"
+#      counts:
+#        standard_name: counts
+#        units: 1
+#    file_type: hrit_b07
 
   B08:
     name: B08
     sensor: ahi
     wavelength: [6.0, 6.2, 6.4]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -223,7 +248,7 @@ datasets:
     name: B09
     sensor: ahi
     wavelength: [6.7, 6.9, 7.1]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -237,7 +262,7 @@ datasets:
     name: B10
     sensor: ahi
     wavelength: [7.1, 7.3, 7.5]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -251,7 +276,7 @@ datasets:
     name: B11
     sensor: ahi
     wavelength: [8.4, 8.6, 8.8]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -265,7 +290,7 @@ datasets:
     name: B12
     sensor: ahi
     wavelength: [9.4, 9.6, 9.8]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -279,7 +304,7 @@ datasets:
     name: B13
     sensor: ahi
     wavelength: [10.2, 10.4, 10.6]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -293,7 +318,7 @@ datasets:
     name: B14
     sensor: ahi
     wavelength: [11.0, 11.2, 11.4]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -307,7 +332,7 @@ datasets:
     name: B15
     sensor: ahi
     wavelength: [12.2, 12.4, 12.6]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature
@@ -321,7 +346,7 @@ datasets:
     name: B16
     sensor: ahi
     wavelength: [13.1, 13.3, 13.5]
-    resolution: 2000
+    resolution: 4000
     calibration:
       brightness_temperature:
         standard_name: toa_brightness_temperature

satpy/multiscene.py

@@ -32,6 +32,13 @@ from satpy.scene import Scene
 from satpy.writers import get_enhanced_image
 from itertools import chain
 
+try:
+    # new API
+    from dask.highlevelgraph import HighLevelGraph
+except ImportError:
+    # old API
+    import dask.sharedict as HighLevelGraph
+
 try:
     from itertools import zip_longest
 except ImportError:
@@ -77,7 +84,7 @@ def cascaded_compute(callback, arrays, batch_size=None, optimize=True):
             # optimize Dask graph over all objects
             dsk = da.Array.__dask_optimize__(
                 # combine all Dask Array graphs
-                dask.sharedict.merge(*[e.__dask_graph__() for e in batch_arrs]),
+                HighLevelGraph.merge(*[e.__dask_graph__() for e in batch_arrs]),
                 # get Dask Array keys in result
                 list(dask.core.flatten([e.__dask_keys__() for e in batch_arrs]))
             )
@@ -122,7 +129,10 @@ class _SceneGenerator(object):
         self._dataset_idx[ds_id] = idx = 0
         while True:
             if idx >= len(self._scene_cache):
+                try:
                     scn = next(self._self_iter)
+                except StopIteration:
+                    return
             else:
                 scn = self._scene_cache[idx]
             yield scn.get(ds_id)

satpy/readers/__init__.py

@@ -515,7 +515,7 @@ def available_readers(as_dict=False):
 
 
 def find_files_and_readers(start_time=None, end_time=None, base_dir=None,
-                           reader=None, sensor=None, ppp_config_dir=get_environ_config_dir(),
+                           reader=None, sensor=None, ppp_config_dir=None,
                            filter_parameters=None, reader_kwargs=None):
     """Find on-disk files matching the provided parameters.
 
@@ -553,6 +553,8 @@ def find_files_and_readers(start_time=None, end_time=None, base_dir=None,
     Returns: Dictionary mapping reader name string to list of filenames
 
     """
+    if ppp_config_dir is None:
+        ppp_config_dir = get_environ_config_dir()
     reader_files = {}
     reader_kwargs = reader_kwargs or {}
     filter_parameters = filter_parameters or reader_kwargs.get('filter_parameters', {})
@@ -595,7 +597,7 @@ def find_files_and_readers(start_time=None, end_time=None, base_dir=None,
 
 
 def load_readers(filenames=None, reader=None, reader_kwargs=None,
-                 ppp_config_dir=get_environ_config_dir()):
+                 ppp_config_dir=None):
     """Create specified readers and assign files to them.
 
     Args:
@@ -612,6 +614,8 @@ def load_readers(filenames=None, reader=None, reader_kwargs=None,
     """
     reader_instances = {}
     reader_kwargs = reader_kwargs or {}
+    if ppp_config_dir is None:
+        ppp_config_dir = get_environ_config_dir()
 
     if not filenames and not reader:
         # used for an empty Scene