Commit 3843d22b authored by Ole Streicher's avatar Ole Streicher

New upstream version 0.9

parent 1966cacb
.. _gammapy_0p9_release:
0.9 (2018-11-29)
----------------
Summary
+++++++
- Released on November 29, 2018 (`Gammapy 0.9 on PyPI <https://pypi.org/project/gammapy/0.9>`__)
- 9 contributors (3 new)
- 2 months of work
- 88 pull requests (not all listed below)
**What's new?**
Gammapy v0.9 comes just two months after v0.8. This is following the `Gammapy
1.0 roadmap`_, Gammapy will from now on have bi-monthly releases, as we work
towards the Gammapy 1.0 release in fall 2019.
Gammapy v0.9 contains many fixes, and a few new features. Big new features
like observation event and time filters, background model classes, as well as
support for fitting joint datasets will come in spring 2019.
The ``FluxPointEstimator`` has been rewritten, and the option to compute
spectral likelihood profiles has been added. The background and diffuse model
interpolation in energy has been improved to be more accurate. The
``gammapy.utils.fitting`` backend is under heavy development, most of the
functionality of MINUIT (covariance, confidence intervals, profiles, contours)
can now be obtained from any ``Fit`` class (spectral or map analysis). Maps now
support arithmetic operators, so that you can e.g. write ``residual = counts -
model`` if ``counts`` and ``model`` are maps containing observed and model
counts.
Gammapy v0.9 now requires Astropy 2.0 or later, and Scipy was changed from
status of optional to required dependency, since currently it is required for
most analysis tasks (e.g. using interpolation when evaluating instrument
responses). Please also note that we have a `plan to drop Python 2.7 support`_
in Gammapy v0.11 in March 2019. If you have any questions or concerns about
moving your scripts and notebooks to Python 3, or need Python 2 support with
later Gammapy releases in 2019, please let us know!
.. _Gammapy 1.0 roadmap: https://github.com/gammapy/gammapy/pull/1841
.. _plan to drop Python 2.7 support: https://github.com/gammapy/gammapy/pull/1278
**Contributors:**
- Atreyee Sinha
- Axel Donath
- Brigitta Sipocz
- Christoph Deil
- Daniel Morcuende (new)
- David Fidalgo
- Ignacio Minaya (new)
- José Enrique Ruiz
- José Luis Contreras (new)
- Régis Terrier
Pull requests
+++++++++++++
This list is incomplete. Small improvements and bug fixes are not listed here.
See the complete `Gammapy 0.9 merged pull requests list on Github <https://github.com/gammapy/gammapy/pulls?q=is%3Apr+milestone%3A0.9+is%3Aclosed>`__.
- [#1949] Add fit minos_contour method (Christoph Deil)
- [#1937] No copy of input and result model in fit (Christoph Deil)
- [#1934] Improve FluxPointEstimator test and docs (Axel Donath)
- [#1933] Add likelihood profiles to FluxPointEstimator (Axel Donath)
- [#1930] Add sections in documentation navigation bar (José Enrique Ruiz)
- [#1929] Rewrite FluxpointEstimator (Axel Donath)
- [#1927] Improve Fit class, add confidence method (Christoph Deil)
- [#1926] Fix MapAxis interpolation FITS serialisation (Atreyee Sinha)
- [#1922] Add Fit.covar method (Christoph Deil)
- [#1921] Use and improve ScaledRegularGridInterpolator (Axel Donath)
- [#1919] Add Scipy as core dependency (Axel Donath)
- [#1918] Add parameters correlation matrix property (Christoph Deil)
- [#1912] Add ObservationFilter class (David Fidalgo)
- [#1909] Clean up irf/io.py and add load_cta_irf function (Régis Terrier)
- [#1908] Take observation time from GTI table (David Fidalgo)
- [#1904] Fix parameter limit handling in fitting (Christoph Deil)
- [#1903] Improve flux points class (Axel Donath)
- [#1898] Review and unify quantity handling (Axel Donath)
- [#1895] Rename obs_list to observations (David Fidalgo)
- [#1894] Improve Background3D energy axis integration (Axel Donath)
- [#1893] Add MapGeom equality operator (Régis Terrier)
- [#1891] Add arithmetic operators for maps (Régis Terrier)
- [#1890] Change map quantity to view instead of copy (Régis Terrier)
- [#1888] Change ObservationList class to Observations (David Fidalgo)
- [#1884] Improve analysis3d tutorial notebook (Ignacio Minaya)
- [#1883] Fix fit parameter bug for very large numbers (Christoph Deil)
- [#1871] Fix TableModel and ConstantModel output dimension (Régis Terrier)
- [#1862] Move make_psf, make_mean_psf and make_mean_edisp (David Fidalgo)
- [#1861] Change from live to on time in background computation (Christoph Deil)
- [#1859] Fix in MapFit energy dispersion apply (Régis Terrier)
- [#1857] Modify image_fitting_with_sherpa to use DC1 runs (Atreyee Sinha)
- [#1855] Add ScaledRegularGridInterpolator (Axel Donath)
- [#1854] Add FluxPointProfiles class (Christoph Deil)
- [#1846] Allow different true and reco energy in map analysis (Atreyee Sinha)
- [#1845] Improve first steps with Gammapy tutorial (Daniel Morcuende)
- [#1837] Add method to compute energy-weighted 2D PSF kernel (Atreyee Sinha)
- [#1836] Fix gammapy download for Python 2 (José Enrique Ruiz)
- [#1807] Change map smooth widths to match Astropy (Atreyee Sinha)
- [#1849] Improve gammapy.stats documentation page (José Luis Contreras)
- [#1766] Add gammapy jupyter CLI for developers (José Enrique Ruiz)
- [#1763] Improve gammapy download (José Enrique Ruiz)
- [#1710] Clean up TableModel implementation (Axel Donath)
- [#1419] PIG 4 - Setup for tutorial notebooks and data (José Enrique Ruiz and Christoph Deil)
.. _gammapy_0p8_release:
0.8 (2018-09-23)
......@@ -197,6 +305,7 @@ See the complete `Gammapy 0.8 merged pull requests list on Github <https://githu
- [#1374] Add units to gammapy.maps (Régis Terrier)
- [#1373] Improve 3D analysis code using gammapy.maps (Christoph Deil)
- [#1372] Add 3D analysis functions using gammapy.maps (Régis Terrier)
- [#1369] Add gammapy download command (José Enrique Ruiz)
- [#1367] Add first draft of LightCurve model class (Christoph Deil)
- [#1362] Fix map sum_over_axes (Christoph Deil)
- [#1360] Sphinx RTD responsive theme for documentation (José Enrique Ruiz)
......@@ -247,8 +356,8 @@ Installation:
Documentation:
- We have created a separate project webpage at http://gammapy.org .
The http://docs.gammapy.org page is not just for the Gammapy documentation.
- We have created a separate project webpage at https://gammapy.org .
The https://docs.gammapy.org page is not just for the Gammapy documentation.
- A lot of new tutorials were added in the form of Jupyter notebooks. To make the content of the
notebooks easier to navigate and search, a rendered static version of the notebooks was integrated
in the Sphinx-based documentation (the one you are looking at) at :ref:`tutorials`.
......@@ -256,7 +365,7 @@ Documentation:
service. There is a "launch in binder" link at the top of each tutorial in the docs,
see e.g. here: `CTA data analysis with Gammapy <notebooks/cta_data_analysis.html>`__
- A page was created to collect the information for CTA members how to get started with Gammapy
and with contact / support channels: http://gammapy.org/cta.html
and with contact / support channels: https://gammapy.org/cta.html
Gammapy Python package:
......@@ -291,12 +400,12 @@ Command line interface:
Organisation:
- A webpage at http://gammapy.org/ was set up, separate from the Gammapy
documentation page http://docs.gammapy.org/ .
- A webpage at https://gammapy.org/ was set up, separate from the Gammapy
documentation page https://docs.gammapy.org/ .
- The Gammapy project and team organisation was set up with clear roles and
responsibilities, in a way to help the Gammapy project grow, and to support
astronomers and projects like CTA using Gammapy better. This is described at
http://gammapy.org/team.html .
https://gammapy.org/team.html .
- To improve the quality of Gammapy, we have set up a proposal-driven process
for major improvements for Gammapy, described in :ref:`pig-001`. We are now
starting to use this to design a better low-level analysis code (`PIG 2`_) and
......
* Code: https://github.com/gammapy/gammapy
* Docs: http://docs.gammapy.org/
* Docs: https://docs.gammapy.org/
**Gammapy** is an open source (BSD licensed) gamma-ray astronomy Python package.
......
Metadata-Version: 2.1
Name: gammapy
Version: 0.8
Version: 0.9
Summary: A Python package for gamma-ray astronomy
Home-page: https://github.com/gammapy/gammapy
Author: The Gammapy developers
......@@ -8,7 +8,7 @@ Author-email: gammapy@googlegroups.com
License: BSD
Description:
* Code: https://github.com/gammapy/gammapy
* Docs: http://docs.gammapy.org/
* Docs: https://docs.gammapy.org/
**Gammapy** is an open source (BSD licensed) gamma-ray astronomy Python package.
......@@ -36,4 +36,5 @@ Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Development Status :: 3 - Alpha
Provides-Extra: analysis
Provides-Extra: test
Provides-Extra: plotting
......@@ -3,22 +3,26 @@ Gammapy
A Python Package for Gamma-ray Astronomy.
* Webpage: http://gammapy.org
* Webpage: https://gammapy.org
* Code: https://github.com/gammapy/gammapy
* Documentation: http://docs.gammapy.org/
* Documentation: https://docs.gammapy.org/
* License: BSD-3
.. image:: http://img.shields.io/badge/powered%20by-AstroPy-orange.svg?style=flat
:target: http://www.astropy.org/
.. image:: http://mybinder.org/badge.svg
:target: https://mybinder.org/v2/gh/gammapy/gammapy/v0.8?urlpath=lab/tree/first_steps.ipynb
:target: https://mybinder.org/v2/gh/gammapy/gammapy/11d0142234f8be6552466c2d2cf70b9dc5ce46af?urlpath=lab/tree/first_steps.ipynb
Status shields
++++++++++++++
(mostly useful for developers)
* .. image:: https://dev.azure.com/gammapy/gammapy/_apis/build/status/gammapy.gammapy
:target: https://dev.azure.com/gammapy/gammapy/_build/latest?definitionId=1
:alt: Build Status
* .. image:: http://img.shields.io/travis/gammapy/gammapy.svg?branch=master
:target: https://travis-ci.org/gammapy/gammapy
:alt: Test Status Travis-CI
......
# Autogenerated by Astropy-affiliated package astropy_helpers's setup.py on 2018-09-23 22:21:48
# Autogenerated by Astropy-affiliated package astropy_helpers's setup.py on 2018-11-15 12:10:37
from __future__ import unicode_literals
import datetime
......@@ -11,7 +11,7 @@ minor = 0
bugfix = 6
release = True
timestamp = datetime.datetime(2018, 9, 23, 22, 21, 48)
timestamp = datetime.datetime(2018, 11, 15, 12, 10, 37)
debug = False
try:
......
......@@ -19,11 +19,11 @@ If you use conda or pip, you can upgrade to this latest stable version using:
To learn more about Gammapy:
* http://gammapy.org/ (project information, contacts, news)
* https://gammapy.org/ (project information, contacts, news)
* http://docs.gammapy.org/stable/ (documentation)
* http://docs.gammapy.org/stable/tutorials.html (tutorials; best place to get started)
If you have any questions or issues, please see let us know!
( see http://gammapy.org/contact.html )
( see https://gammapy.org/contact.html )
Christoph for the Gammapy team
......@@ -25,11 +25,11 @@ If you use conda or pip, you can upgrade to this latest stable version using:
To learn more about Gammapy:
* http://gammapy.org/ (project information, contacts, news)
* https://gammapy.org/ (project information, contacts, news)
* http://docs.gammapy.org/stable/ (documentation)
* http://docs.gammapy.org/stable/tutorials.html (tutorials; best place to get started)
If you have any questions or issues, please see let us know!
( see http://gammapy.org/contact.html )
( see https://gammapy.org/contact.html )
Christoph for the Gammapy team
$(document).ready(function() {
// javascript code to enable download links of notebooks and scripts *.py
// https://stackoverflow.com/questions/2304941
if (document.readyState == "interactive") {
document.getElementsByClassName("last")[0].children[0].children[0].setAttribute("target", "_blank")
document.getElementsByClassName("last")[0].children[2].children[1].setAttribute("download", "")
document.getElementsByClassName("last")[0].children[2].children[2].setAttribute("download", "")
}
// add info note for past releases
var verFile = new XMLHttpRequest();
verFile.open("GET", "https://docs.gammapy.org/stable/index.html", true);
verFile.onreadystatechange = function() {
if (verFile.readyState === 4) { // makes sure the document is ready to parse.
if (verFile.status === 200) { // makes sure it's found the file.
var allText = verFile.responseText;
var match = allText.match(/url=\.\.\/(.*)"/i);
var version = match[1];
var note = '<div class="admonition note"><p class="first admonition-title">Note</p>'
note += '<p class="last">You are not reading the most up to date version of Gammapy '
note += 'documentation.<br/>Access the <a href="https://docs.gammapy.org/'
note += version
note += '/">latest stable version v'
note += version
note += '</a> or the <a href="https://gammapy.org/news.html#releases">list of Gammapy releases</a>.</p></div>'
var url = window.location.href
if (url.includes("/dev/") == false && url.includes(version) == false ) {
var divbody = document.querySelectorAll('[role="main"]');
var divnote = document.createElement("div");
divnote.innerHTML = note;
divbody[0].insertBefore(divnote, divbody[0].firstChild);
}
}
}
}
verFile.send(null);
});
/*
* linksdl.js
* ~~~~~~~~~~~
*
* javascript code to enable download links of notebooks and scripts *.py
* https://stackoverflow.com/questions/2304941
*
*/
document.onreadystatechange = function () {
if (document.readyState == "interactive") {
document.getElementsByClassName("last")[0].children[0].children[0].setAttribute("target", "_blank")
document.getElementsByClassName("last")[0].children[2].children[1].setAttribute("download", "")
document.getElementsByClassName("last")[0].children[2].children[2].setAttribute("download", "")
}
}
......@@ -7,7 +7,7 @@ from gammapy.astro.population import FaucherSpiral
from gammapy.utils.coordinates import polar, cartesian
catalog = simulate.make_base_catalog_galactic(
n_sources=int(1e4), rad_dis="YK04", vel_dis="H05", max_age=Quantity(1E6, "yr")
n_sources=int(1e4), rad_dis="YK04", vel_dis="H05", max_age=Quantity(1e6, "yr")
)
spiral = FaucherSpiral()
......
......@@ -11,6 +11,6 @@ pulsar = Pulsar(P_0=Quantity(0.01, "s"), logB=12)
plt.plot(t.value, 1 / pulsar.period(t).cgs.value)
plt.xlabel("time [years]")
plt.ylabel("frequency [1/s]")
plt.ylim(1E0, 1E2)
plt.ylim(1e0, 1e2)
plt.loglog()
plt.show()
......@@ -9,7 +9,7 @@ t = Quantity(np.logspace(1, 5, 100), "yr")
n_ISM = Quantity(1, "cm^-3")
snr = SNRTrueloveMcKee(m_ejecta=8 * M_sun, n_ISM=n_ISM)
pwn = PWN(snr=snr)
pwn.pulsar.L_0 = Quantity(1E40, "erg/s")
pwn.pulsar.L_0 = Quantity(1e40, "erg/s")
plt.plot(t.value, pwn.radius(t).to("pc").value, label="Radius PWN")
plt.plot(t.value, snr.radius_reverse_shock(t).to("pc").value, label="Reverse Shock SNR")
......
......@@ -11,11 +11,11 @@ for density in densities:
snr = SNR(n_ISM=density)
F = snr.luminosity_tev(t) / (4 * np.pi * Quantity(1, "kpc") ** 2)
plt.plot(t.value, F.to("cm-2 s-1").value, label="n_ISM = {0}".format(density.value))
plt.vlines(snr.sedov_taylor_begin.to("yr").value, 1E-13, 1E-10, linestyle="--")
plt.vlines(snr.sedov_taylor_end.to("yr").value, 1E-13, 1E-10, linestyle="--")
plt.vlines(snr.sedov_taylor_begin.to("yr").value, 1e-13, 1e-10, linestyle="--")
plt.vlines(snr.sedov_taylor_end.to("yr").value, 1e-13, 1e-10, linestyle="--")
plt.xlim(1E2, 1E5)
plt.ylim(1E-13, 1E-10)
plt.xlim(1e2, 1e5)
plt.ylim(1e-13, 1e-10)
plt.xlabel("time [years]")
plt.ylabel("flux @ 1kpc [s^-1 cm^-2]")
plt.legend(loc=4)
......
......@@ -11,6 +11,6 @@ We don't list every pull request.
Maintenance and cleanup changes (e.g. to update astropy-helpers or fix the docs build)
are not of interest to users and are not listed here.
A complete list of Gammapy contributors is at http://gammapy.org/team.html .
A complete list of Gammapy contributors is at https://gammapy.org/team.html .
.. include:: ../CHANGES.rst
......@@ -198,12 +198,14 @@ gammapy_sphinx_notebooks(setup_cfg)
def setup(app):
app.add_stylesheet("gammapy.css")
app.add_javascript("copybutton.js")
app.add_javascript("gammapy.js")
# copybutton.js provides hide/show button for python prompts >>>
# slightly modified to work on RTD theme from javascript file in easydev package
# https://github.com/cokelaer/easydev/blob/master/easydev/share/copybutton.js
html_favicon = os.path.join(html_static_path[0], "gammapy_logo.ico")
# -- Options for LaTeX output --------------------------------------------------
......
......@@ -761,19 +761,18 @@ Gammapy has a class for generic n-dimensional data arrays,
should use this class. The goal is to reuse code for interpolation
and have an coherent I/O interface, mainly in `~gammapy.irf`.
A usage example can be found in :gp-extra-notebooks:``nddata_demo``.
A usage example can be found in :gp-extra-notebook:`nddata_demo`.
Also, consult :ref:`interpolation-extrapolation` if you are not sure how to
setup your interpolator.
Sphinx docs build
-----------------
Generating the HTML docs for Gammapy is straight-forward::
python setup.py build_docs
open docs/_build/html/index.html
make docs-all
make docs-show
Generating the PDF docs is more complex.
This should work::
......@@ -786,17 +785,31 @@ This should work::
You need a bunch or LaTeX stuff, specifically ``texlive-fonts-extra`` is needed.
Jupyter notebooks present in the ``gammapy-extra`` repository are by default copied
to the ``docs/notebooks`` and ``docs/_static/notebooks`` folders during
the process of generating HTML docs. This triggers its conversion to Sphinx
formatted HTML files and .py scripts. The Sphinx formatted versions of the notebooks provide access to the raw .ipynb Jupyter files and .py script versions stored in ``docs/_static/notebooks`` folder.
Jupyter notebooks stripped of output cells are present in the ``tutorials`` folder.
They are by default tested, executed, and copied to the ``docs/notebooks`` and
``docs/_static/notebooks`` folders during the process of generating HTML docs. This
triggers its conversion to Sphinx formatted HTML files and ``.py`` scripts. The Sphinx
formatted versions of the notebooks provide links to the raw ``.ipynb`` Jupyter files
and ``.py`` script versions stored in ``docs/_static/notebooks`` folder.
Once the documentation built you can optimize the speed of re-building processes,
for example in case you are modifying or creating new docs and you would like to check
these changes are displayed nicely. For that purpose, if your modified doc file
does not contain links to notebooks, you may set the flag ``build_notebooks`` to False
in the ``setup.cfg`` file, so they are not re-written again by Sphinx.
these changes are displayed nicely. For that purpose, if your modified RST file
does not contain links to notebooks, you may run ``make docs-all nbs=False`` so
that notebooks are not executed during the docs build.
In the case one single notebook is modified or added to the documentation, you can
execute the build doc process with the ``src`` parameter with value the name of the
considered notebook. i.e. ``make docs-all src=tutorials/my-notebook.ipynb``
Each *fixed-text* Sphinx formatted notebook present in the documentation has its
own link pointing to its specific space in Gammapy Binder. Since notebooks are
evolving with Gammapy functionalities and documentation, it is possible to link
the different versions of the notebooks to the same versions built in Gammapy Binder.
For stable releases, it is useful to use the ``release`` parameter with the value
of the release label tag used in Github. This value will be used to build the links
to Binder for that specific stable release for each of the tutorials published in
the :ref:`tutorials` section. i.e. ``make docs-all release=v0.8``
Documentation guidelines
------------------------
......@@ -895,19 +908,6 @@ so many getters and setters. We could start using descriptors.
TODO: make a decision on this and describe the issue / solution here.
Different versions of notebooks in Binder
-----------------------------------------
Jupyter notebooks may be accessed and executed on-line in the
`Gammapy Binder <http://mybinder.org/repo/gammapy/gammapy-extra>`__ space. Each *fixed-text* sphinx
formatted notebook present in the documentation has its own link pointing to its specific space in
Gammapy Binder. Since notebooks are evolving with Gammapy functionalities and documentation, it is
possible to link the different versions of the notebooks stored in GitHub repository ``gammapy-extra``
to the same versions built in Gammapy Binder. For this purpose just edit the variable **git_commit**
in ``setup.cfg`` file and provide the branch, tag or commit of GitHub repository ``gammapy-extra``
that will be used to access the same version of the notebook in Gammapy Binder.
Link to a notebook in gammapy-extra from the docs
-------------------------------------------------
......@@ -1025,7 +1025,6 @@ rendering of the plot, which can rasie errore as well. Here is a short example:
from gammapy.utils.testing import mpl_plot_check
def test_plot():
import matplotlib.pyplot as plt
with mpl_plot_check():
plt.plot([1., 2., 3., 4., 5.])
......@@ -1033,4 +1032,4 @@ rendering of the plot, which can rasie errore as well. Here is a short example:
With this approach we make sure that the plotting code is at least executed once
and runs completely (up to saving the plot to file) without errors. In future we
will maybe change to something like https://github.com/matplotlib/pytest-mpl
to ensure that correct plots are produced.
\ No newline at end of file
to ensure that correct plots are produced.
......@@ -166,7 +166,7 @@ more details here: :ref:`setup_cython`. If you want remove the generated files
run ``make clean``.
For the development it is also convenient to fork and set up the
:ref:`dev_gammapy-extra`:
:ref:`dev_gammapy-extra`, as well as declaring some environment variables:
.. code-block:: bash
......@@ -175,6 +175,19 @@ For the development it is also convenient to fork and set up the
git clone https://github.com/[your-github-username]/gammapy-extra.git
export GAMMAPY_EXTRA=$PWD/gammapy-extra
`$GAMMAPY_EXTRA`` is mainly used for testing purposes, on the contrary the datasets
provided in ``gammapy.catalog`` and those used in the tutorial Jupyter notebooks
should be in a different path declared in a `$GAMMAPY_DATA` environment variable.
You can download these datasets with `gammapy download datasets` and then point
your `$GAMMAPY_DATA` to the local path you have chosen.
.. code-block:: bash
# Download GAMMAPY_DATA
cd code
gammapy download datasets --out GAMMAPY_DATA
export GAMMAPY_DATA=$PWD/GAMMAPY_DATA
* install dependencies
* git clone dev version
* run tests
......@@ -201,6 +214,13 @@ Integrate the code in Gammapy
* check tests locally
* check docs locally
Contribute with Jupyter notebooks
=================================
* check tests with user tutorials environment: `gammapy jupyter test --tutor`
* strip the output cells and format code: `gammapy jupyter strip` `gammapy jupyter black`
* diff stripped notebooks: `git diff mynotbeook.pynb`
Make a pull request
===================
......@@ -217,4 +237,3 @@ Close the loop
==============
tbd
......@@ -19,5 +19,6 @@ label`_ .
pig-001
pig-002
pig-004
.. _pull requests with the "pig" label: https://github.com/gammapy/gammapy/issues?q=label%3Apig
.. include:: ../../references.txt
.. _pig-004:
*********************************************
PIG 4 - Setup for tutorial notebooks and data
*********************************************
* Author: José Enrique Ruiz, Christoph Deil
* Created: May 16, 2018
* Accepted: Oct 4, 2018
* Status: accepted
* Discussion: `GH 1419`_
Abstract
========
For the past years, we have had tutorial notebooks and example datasets in a
second ``gammapy-extra`` repository, as well as others example datasets placed in
differente repositories like ``gamma-cat`` and ``gammapy-fermi-lat-data``.
The motivation was to keep the main ``gammapy`` code repository small.
But we always had problems with code, tutorials and data changing and versions
not being linked.
We propose to move the notebooks to the ``gammapy`` repository so that code and
tutorials can be version-coupled, and to only use stable datasets in tutorials to
mostly save the versioning issues. The datasets will remain in ``gammapy-extra``
repository.
To ship tutorials and datasets to users, we propose to add a ``gammapy download``
command. The ``gammapy-extra`` repository will remain as a repository for
developers and as one place where datasets can be put, but it will not be mentioned
to users.
What we have
============
We have the `gammapy`_ repository for code, and the `gammapy-extra`_ repository
for tutorial notebooks, example datasets and a few other things.
The ``gammapy`` repository currently is 12 MB and the ``gammapy-extra`` repository
is 161 MB. In ``gammapy-extra/notebooks``, we have ~ 30 tutorial notebooks, each
20 kB to 1 MB in size, i.e. a few MB in total. Most of the size comes from PNG
output images in the notebooks, and they usually change on re-run, i.e. even
though git compresses a bit, the repo grows by up to 1 MB every time a notebook
is edited. The datasets we access from the tutorials are maybe 20 or 30 MB, a
lot of the datasets we have there are old and should be cleaned up / removed.
The reason the notebooks and datasets were split out from the code was to keep the
code repository small and avoid it growing to 100 MB or even 1 GB over time.
This separation of code vs. notebooks and datasets has been a problem in Gammapy
for years.
Given that Gammapy code changes (and probably always will, even if the pace will
become slower and slower over the years), the tutorials have to be version-coupled to
the code. A related question is how tutorial notebooks, datasets used in tutorials and
other datasets not used in the tutorials are shipped to users. Some related discussions
may be found in the following references, see e.g. `GH 1237`_, `GH 1369`_,
`GH 700`_, `GH 431`_, `GH 405`_, `GH 228`_, `GH 1131`_ (probably missed a few).
Proposal
========
This proposal is limited. It outlines a few key changes in the setup for
notebooks and example data that mostly solve the versioning and shipping issue
of tutorials and datasets. Other related issues that may appear will be faced
iteratively with or without an extra PIG.
To solve the versioning issue for notebooks, we propose to move the notebooks
from ``gammapy-extra/notebooks`` to ``gammapy/notebooks``. We propose to store
the notebooks in the repository without output cells filled. Removing output
cells before committing has the advantage that the files are small, and that
the diff in the pull request is also small and review becomes possible.
On the other hand, the output is not directly visible on Github. Note that in
any case, a rendered version of the notebooks will be available via the docs,
that is already in place. We count on developer documentation and code review
to guarantee empty-output notebooks stored in ``gammapy/notebooks``, though we
can also explore `nbstripout`_ for a potential implementation of an automated
mechamism to remove outputs from notebooks in the Github repository.
In the process of documentation building the notebooks will be texted and executed
automatically, so the static HTML-formatted notebooks will contain output cells
rendered in the the documentation. On the contrary, links to Binder notebooks and
download links to .ipynb files will point to empty-output notebooks.
To solve the versioning issue for datasets, we propose to only use stable
example datasets. Examples are `gammapy-fermi-lat-data`_ or
`gammapy-extra/datasets/cta-1dc`_ or the upcoming `HESS DL3 DR1`_ or
``joint-crab`` datasets. Datasets can be in ``gammapy-extra`` or at any other
URL, but even if they are in ``gammapy-extra``, they should not be ''live''
datasets. If an issue is found or something is improved, a separate new dataset
should be added, instead of changing the existing one. So versioning of example
datasets is not needed.
To ship notebooks and example data to users, we propose to introduce a ``gammapy
download`` command. This work and discussion how it should work in detail has
started in `GH 1369`_. Roughly, the idea is that users will use ``gammapy
download`` to download a version of the notebooks matching the version of the
Gammapy code, by fetching the files from Github. A ``gammapy download tutorials``
command will download all notebooks and the input datasets related. Not output
datasets from the notebooks will be downloaded. All files will be copied into a