Commit 1071b994 authored by Wesley Wiedenmeier's avatar Wesley Wiedenmeier Committed by Scott Moser

Improve module documentation and doc cleanup.

This adds lots of config module documentation in a standard format.
It will greatly improve the content at readthedocs.
Additionally:
 * Add a 'doc' env to tox.ini
 * Changed default highlight language for sphinx conf from python to yaml
   most examples in documentation are yaml configs
 * Updated datasource examples to highlight sh code properly
parent 02f6c4bb
......@@ -18,6 +18,213 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Apt Configure
-------------
**Summary:** configure apt
This module handles both configuration of apt options and adding source lists.
There are configuration options such as ``apt_get_wrapper`` and
``apt_get_command`` that control how cloud-init invokes apt-get.
These configuration options are handled on a per-distro basis, so consult
documentation for cloud-init's distro support for instructions on using
these config options.
.. note::
To ensure that apt configuration is valid yaml, any strings containing
special characters, especially ``:`` should be quoted.
.. note::
For more information about apt configuration, see the
``Additional apt configuration`` example.
**Preserve sources.list:**
By default, cloud-init will generate a new sources list in
``/etc/apt/sources.list.d`` based on any changes specified in cloud config.
To disable this behavior and preserve the sources list from the pristine image,
set ``preserve_sources_list`` to ``true``.
.. note::
The ``preserve_sources_list`` option overrides all other config keys that
would alter ``sources.list`` or ``sources.list.d``, **except** for
additional sources to be added to ``sources.list.d``.
**Disable source suites:**
Entries in the sources list can be disabled using ``disable_suites``, which
takes a list of suites to be disabled. If the string ``$RELEASE`` is present in
a suite in the ``disable_suites`` list, it will be replaced with the release
name. If a suite specified in ``disable_suites`` is not present in
``sources.list`` it will be ignored. For convenience, several aliases are
provided for ``disable_suites``:
- ``updates`` => ``$RELEASE-updates``
- ``backports`` => ``$RELEASE-backports``
- ``security`` => ``$RELEASE-security``
- ``proposed`` => ``$RELEASE-proposed``
- ``release`` => ``$RELEASE``
.. note::
When a suite is disabled using ``disable_suites``, its entry in
``sources.list`` is not deleted; it is just commented out.
**Configure primary and security mirrors:**
The primary and security archive mirrors can be specified using the ``primary``
and ``security`` keys, respectively. Both the ``primary`` and ``security`` keys
take a list of configs, allowing mirrors to be specified on a per-architecture
basis. Each config is a dictionary which must have an entry for ``arches``,
specifying which architectures that config entry is for. The keyword
``default`` applies to any architecture not explicitly listed. The mirror url
can be specified with the ``url`` key, or a list of mirrors to check can be
provided in order, with the first mirror that can be resolved being selected.
This allows the same configuration to be used in different environment, with
different hosts used for a local apt mirror. If no mirror is provided by uri or
search, ``search_dns`` may be used to search for dns names in the format
``<distro>-mirror`` in each of the following:
- fqdn of this host per cloud metadata
- localdomain
- domains listed in ``/etc/resolv.conf``
If there is a dns entry for ``<distro>-mirror``, then it is assumed that there
is a distro mirror at ``http://<distro>-mirror.<domain>/<distro>``. If the
``primary`` key is defined, but not the ``security`` key, then then
configuration for ``primary`` is also used for ``security``. If ``search_dns``
is used for the ``security`` key, the search pattern will be.
``<distro>-security-mirror``.
If no mirrors are specified, or all lookups fail, then default mirrors defined
in the datasource are used. If none are present in the datasource either the
following defaults are used:
- primary: ``http://archive.ubuntu.com/ubuntu``
- security: ``http://security.ubuntu.com/ubuntu``
**Specify sources.list template:**
A custom template for rendering ``sources.list`` can be specefied with
``sources_list``. If no ``sources_list`` template is given, cloud-init will
use sane default. Within this template, the following strings will be replaced
with the appropriate values:
- ``$MIRROR``
- ``$RELEASE``
- ``$PRIMARY``
- ``$SECURITY``
**Pass configuration to apt:**
Apt configuration can be specified using ``conf``. Configuration is specified
as a string. For multiline apt configuration, make sure to follow yaml syntax.
**Configure apt proxy:**
Proxy configuration for apt can be specified using ``conf``, but proxy config
keys also exist for convenience. The proxy config keys, ``http_proxy``,
``ftp_proxy``, and ``https_proxy`` may be used to specify a proxy for http, ftp
and https protocols respectively. The ``proxy`` key also exists as an alias for
``http_proxy``. Proxy url is specified in the format
``<protocol>://[[user][:pass]@]host[:port]/``.
**Add apt repos by regex:**
All source entries in ``apt-sources`` that match regex in
``add_apt_repo_match`` will be added to the system using
``add-apt-repository``. If ``add_apt_repo_match`` is not specified, it defaults
to ``^[\w-]+:\w``
**Add source list entries:**
Source list entries can be specified as a dictionary under the ``sources``
config key, with key in the dict representing a different source file. The key
The key of each source entry will be used as an id that can be referenced in
other config entries, as well as the filename for the source's configuration
under ``/etc/apt/sources.list.d``. If the name does not end with ``.list``,
it will be appended. If there is no configuration for a key in ``sources``, no
file will be written, but the key may still be referred to as an id in other
``sources`` entries.
Each entry under ``sources`` is a dictionary which may contain any of the
following optional keys:
- ``source``: a sources.list entry (some variable replacements apply)
- ``keyid``: a key to import via shortid or fingerprint
- ``key``: a raw PGP key
- ``keyserver``: alternate keyserver to pull ``keyid`` key from
The ``source`` key supports variable replacements for the following strings:
- ``$MIRROR``
- ``$PRIMARY``
- ``$SECURITY``
- ``$RELEASE``
**Internal name:** ``cc_apt_configure``
**Module frequency:** per instance
**Supported distros:** ubuntu, debian
**Config keys**::
apt:
preserve_sources_list: <true/false>
disable_suites:
- $RELEASE-updates
- backports
- $RELEASE
- mysuite
primary:
- arches:
- amd64
- i386
- default
uri: "http://us.archive.ubuntu.com/ubuntu"
search:
- "http://cool.but-sometimes-unreachable.com/ubuntu"
- "http://us.archive.ubuntu.com/ubuntu"
search_dns: <true/false>
- arches:
- s390x
- arm64
uri: "http://archive-to-use-for-arm64.example.com/ubuntu"
security:
- arches:
- default
search_dns: true
sources_list: |
deb $MIRROR $RELEASE main restricted
deb-src $MIRROR $RELEASE main restricted
deb $PRIMARY $RELEASE universe restricted
deb $SECURITY $RELEASE-security multiverse
conf: |
APT {
Get {
Assume-Yes "true";
Fix-Broken "true";
}
}
proxy: "http://[[user][:pass]@]host[:port]/"
http_proxy: "http://[[user][:pass]@]host[:port]/"
ftp_proxy: "ftp://[[user][:pass]@]host[:port]/"
https_proxy: "https://[[user][:pass]@]host[:port]/"
sources:
source1:
keyid: "keyid"
keyserver: "keyserverurl"
source: "deb http://<url>/ xenial main"
source2:
source "ppa:<ppa-name>"
source3:
source "deb $MIRROR $RELEASE multiverse"
key: |
------BEGIN PGP PUBLIC KEY BLOCK-------
<key data>
------END PGP PUBLIC KEY BLOCK-------
"""
import glob
import os
import re
......
......@@ -16,6 +16,31 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Apt Pipelining
--------------
**Summary:** configure apt pipelining
This module configures apt's ``Acquite::http::Pipeline-Depth`` option, whcih
controls how apt handles HTTP pipelining. It may be useful for pipelining to be
disabled, because some web servers, such as S3 do not pipeline properly (LP:
#948461). The ``apt_pipelining`` config key may be set to ``false`` to disable
pipelining altogether. This is the default behavior. If it is set to ``none``,
``unchanged``, or ``os``, no change will be made to apt configuration and the
default setting for the distro will be used. The pipeline depth can also be
manually specified by setting ``apt_pipelining`` to a number. However, this is
not recommended.
**Internal name:** ``cc_apt_pipelining``
**Module frequency:** per instance
**Supported distros:** ubuntu, debian
**Config keys**::
apt_pipelining: <false/none/unchanged/os/number>
"""
from cloudinit.settings import PER_INSTANCE
from cloudinit import util
......
......@@ -18,6 +18,34 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Bootcmd
-------
**Summary:** run commands early in boot process
This module runs arbitrary commands very early in the boot process,
only slightly after a boothook would run. This is very similar to a
boothook, but more user friendly. The environment variable ``INSTANCE_ID``
will be set to the current instance id for all run commands. Commands can be
specified either as lists or strings. For invocation details, see ``runcmd``.
.. note::
bootcmd should only be used for things that could not be done later in the
boot process.
**Internal name:** ``cc_bootcmd``
**Module frequency:** per always
**Supported distros:** all
**Config keys**::
bootcmd:
- echo 192.168.1.130 us.archive.ubuntu.com > /etc/hosts
- [ cloud-nit-per, once, mymkfs, mkfs, /dev/vdb ]
"""
import os
from cloudinit.settings import PER_ALWAYS
......
......@@ -18,6 +18,39 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Byobu
-----
**Summary:** enable/disable byobu system wide and for default user
This module controls whether byobu is enabled or disabled system wide and for
the default system user. If byobu is to be enabled, this module will ensure it
is installed. Likewise, if it is to be disabled, it will be removed if
installed.
Valid configuration options for this module are:
- ``enable-system``: enable byobu system wide
- ``enable-user``: enable byobu for the default user
- ``disable-system``: disable byobu system wide
- ``disable-user``: disable byobu for the default user
- ``enable``: enable byobu both system wide and for default user
- ``disable``: disable byobu for all users
- ``user``: alias for ``enable-user``
- ``system``: alias for ``enable-system``
**Internal name:** ``cc_byobu``
**Module frequency:** per instance
**Supported distros:** ubuntu, debian
**Config keys**::
byobu_by_default: <user/system>
"""
# Ensure this is aliased to a name not 'distros'
# since the module attribute 'distros'
# is a list of distros that are supported, not a sub-module
......
......@@ -14,6 +14,38 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
CA Certs
--------
**Summary:** add ca certificates
This module adds CA certificates to ``/etc/ca-certificates.conf`` and updates
the ssl cert cache using ``update-ca-certificates``. The default certificates
can be removed from the system with the configuration option
``remove-defaults``.
.. note::
certificates must be specified using valid yaml. in order to specify a
multiline certificate, the yaml multiline list syntax must be used
**Internal name:** ``cc_ca_certs``
**Module frequency:** per instance
**Supporte distros:** ubuntu, debian
**Config keys**::
ca-certs:
remove-defaults: <true/false>
trusted:
- <single line cert>
- |
-----BEGIN CERTIFICATE-----
YOUR-ORGS-TRUSTED-CA-CERT-HERE
-----END CERTIFICATE-----
"""
import os
from cloudinit import util
......
......@@ -19,9 +19,11 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Chef
----
**Summary:** module that configures, starts and installs chef.
**Description:** This module enables chef to be installed (from packages or
This module enables chef to be installed (from packages or
from gems, or from omnibus). Before this occurs chef configurations are
written to disk (validation.pem, client.pem, firstboot.json, client.rb),
and needed chef folders/directories are created (/etc/chef and /var/log/chef
......@@ -33,7 +35,13 @@ chef will have forked into its own process) then a post run function can
run that can do finishing activities (such as removing the validation pem
file).
It can be configured with the following option structure::
**Internal name:** ``cc_chef``
**Module frequency:** per always
**Supported distros:** all
**Config keys**::
chef:
directories: (defaulting to /etc/chef, /var/log/chef, /var/lib/chef,
......
......@@ -15,22 +15,28 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Debug
-----
**Summary:** helper to debug cloud-init *internal* datastructures.
**Description:** This module will enable for outputting various internal
information that cloud-init sources provide to either a file or to the output
console/log location that this cloud-init has been configured with when
running.
This module will enable for outputting various internal information that
cloud-init sources provide to either a file or to the output console/log
location that this cloud-init has been configured with when running.
It can be configured with the following option structure::
.. note::
Log configurations are not output.
debug:
verbose: (defaulting to true)
output: (location to write output, defaulting to console + log)
**Internal name:** ``cc_debug``
.. note::
**Module frequency:** per instance
Log configurations are not output.
**Supported distros:** all
**Config keys**::
debug:
verbose: true/false (defaulting to true)
output: (location to write output, defaulting to console + log)
"""
import copy
......
......@@ -18,6 +18,26 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Disable EC2 Metadata
--------------------
**Summary:** disable aws ec2 metadata
This module can disable the ec2 datasource by rejecting the route to
``169.254.169.254``, the usual route to the datasource. This module is disabled
by default.
**Internal name:** ``cc_disable_ec2_metadata``
**Module frequency:** per always
**Supported distros:** all
**Config keys**::
disable_ec2_metadata: <true/false>
"""
from cloudinit import util
from cloudinit.settings import PER_ALWAYS
......
......@@ -16,6 +16,96 @@
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Disk Setup
----------
**Summary:** configure partitions and filesystems
This module is able to configure simple partition tables and filesystems.
.. note::
for more detail about configuration options for disk setup, see the disk
setup example
For convenience, aliases can be specified for disks using the
``device_aliases`` config key, which takes a dictionary of alias: path
mappings. There are automatic aliases for ``swap`` and ``ephemeral<X>``, where
``swap`` will always refer to the active swap partition and ``ephemeral<X>``
will refer to the block device of the ephemeral image.
Disk partitioning is done using the ``disk_setup`` directive. This config
directive accepts a dictionary where each key is either a path to a block
device or an alias specified in ``device_aliases``, and each value is the
configuration options for the device. The ``table_type`` option specifies the
partition table type, either ``mbr`` or ``gpt``. The ``layout`` option
specifies how partitions on the device are to be arranged. If ``layout`` is set
to ``true``, a single partition using all the space on the device will be
created. If set to ``false``, no partitions will be created. Partitions can be
specified by providing a list to ``layout``, where each entry in the list is
either a size or a list containing a size and the numerical value for a
partition type. The size for partitions is specified in **percentage** of disk
space, not in bytes (e.g. a size of 33 would take up 1/3 of the disk space).
The ``overwrite`` option controls whether this module tries to be safe about
writing partition talbes or not. If ``overwrite: false`` is set, the device
will be checked for a partition table and for a file system and if either is
found, the operation will be skipped. If ``overwrite: true`` is set, no checks
will be performed.
.. note::
Using ``overwrite: true`` is dangerous and can lead to data loss, so double
check that the correct device has been specified if using this option.
File system configuration is done using the ``fs_setup`` directive. This config
directive accepts a list of filesystem configs. The device to create the
filesystem on may be specified either as a path or as an alias in the format
``<alias name>.<y>`` where ``<y>`` denotes the partition number on the device.
The partition can also be specified by setting ``partition`` to the desired
partition number. The ``partition`` option may also be set to ``auto``, in
which this module will search for the existance of a filesystem matching the
``label``, ``type`` and ``device`` of the ``fs_setup`` entry and will skip
creating the filesystem if one is found. The ``partition`` option may also be
set to ``any``, in which case any file system that matches ``type`` and
``device`` will cause this module to skip filesystem creation for the
``fs_setup`` entry, regardless of ``label`` matching or not. To write a
filesystem directly to a device, use ``partition: none``. A label can be
specified for the filesystem using ``label``, and the filesystem type can be
specified using ``filesystem``.
.. note::
If specifying device using the ``<device name>.<partition number>`` format,
the value of ``partition`` will be overwritten.
.. note::
Using ``overwrite: true`` for filesystems is dangerous and can lead to data
loss, so double check the entry in ``fs_setup``.
**Internal name:** ``cc_disk_setup``
**Module frequency:** per instance
**Supported distros:** all
**Config keys**::
device_aliases:
<alias name>: <device path>
disk_setup:
<alias name/path>:
table_type: <'mbr'/'gpt'>
layout:
- [33,82]
- 66
overwrite: <true/false>
fs_setup:
- label: <label>
filesystem: <filesystem type>
device: <device>
partition: <"auto"/"any"/"none"/<partition number>>
overwrite: <true/false>
replace_fs: <filesystem type>
"""
from cloudinit.settings import PER_INSTANCE
from cloudinit import util
import logging
......@@ -40,7 +130,7 @@ LOG = logging.getLogger(__name__)
def handle(_name, cfg, cloud, log, _args):
"""
See doc/examples/cloud-config_disk-setup.txt for documentation on the
See doc/examples/cloud-config-disk-setup.txt for documentation on the
format.
"""
disk_setup = cfg.get("disk_setup")
......
......@@ -18,6 +18,21 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Emit Upstart
------------
**Summary:** emit upstart configuration
Emit upstart configuration for cloud-init modules on upstart based systems. No
user configuration should be required.
**Internal name:** ``cc_emit_upstart``
**Module frequency:** per always
**Supported distros:** ubuntu, debian
"""
import os
from cloudinit import log as logging
......
......@@ -15,25 +15,38 @@
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
fan module allows configuration of Ubuntu Fan
https://wiki.ubuntu.com/FanNetworking
Example config:
#cloud-config
fan:
config: |
# fan 240
10.0.0.0/8 eth0/16 dhcp
10.0.0.0/8 eth1/16 dhcp off
# fan 241
241.0.0.0/8 eth0/16 dhcp
config_path: /etc/network/fan
If cloud-init sees a 'fan' entry in cloud-config it will
a.) write 'config_path' with the contents
b.) install the package 'ubuntu-fan' if it is not installed
c.) ensure the service is started (or restarted if was previously running)
Fan
---
**Summary:** configure ubuntu fan networking
This module installs, configures and starts the ubuntu fan network system. For
more information about Ubuntu Fan, see:
``https://wiki.ubuntu.com/FanNetworking``.
If cloud-init sees a ``fan`` entry in cloud-config it will:
- write ``config_path`` with the contents of the ``config`` key
- install the package ``ubuntu-fan`` if it is not installed
- ensure the service is started (or restarted if was previously running)
**Internal name:** ``cc_fan``
**Module frequency:** per instance
**Supported distros:** ubuntu
**Config keys**::
fan:
config: |
# fan 240
10.0.0.0/8 eth0/16 dhcp
10.0.0.0/8 eth1/16 dhcp off
# fan 241
241.0.0.0/8 eth0/16 dhcp
config_path: /etc/network/fan
"""
from cloudinit import log as logging
......
......@@ -18,6 +18,31 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Final Message
-------------
**Summary:** output final message when cloud-init has finished
This module configures the final message that cloud-init writes. The message is
specified as a jinja template with the following variables set:
- ``version``: cloud-init version
- ``timestamp``: time at cloud-init finish
- ``datasource``: cloud-init data source
- ``uptime``: system uptime
**Internal name:** ``cc_final_message``
**Module frequency:** per always
**Supported distros:** all
**Config keys**::
final_message: <message>
"""
from cloudinit import templater
from cloudinit import util
from cloudinit import version
......
......@@ -18,6 +18,20 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Foo
---
**Summary:** example module
Example to show module structure. Does not do anything.
**Internal name:** ``cc_foo``
**Module frequency:** per instance
**Supported distros:** all
"""
from cloudinit.settings import PER_INSTANCE
# Modules are expected to have the following attributes.
......@@ -35,7 +49,7 @@ from cloudinit.settings import PER_INSTANCE
# Typically those are from module configuration where the module
# is defined with some extra configuration that will eventually
# be translated from yaml into arguments to this module.
# 2. A optional 'frequency' that defines how often this module should be ran.
# 2. A optional 'frequency' that defines how often this module should be run.
# Typically one of PER_INSTANCE, PER_ALWAYS, PER_ONCE. If not
# provided PER_INSTANCE will be assumed.
# See settings.py for these constants.
......
......@@ -18,6 +18,63 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
Growpart
--------
**Summary:** grow partitions
Growpart resizes partitions to fill the available disk space.
This is useful for cloud instances with a larger amount of disk space available
than the pristine image uses, as it allows the instance to automatically make
use of the extra space.
The devices run growpart on are specified as a list under the ``devices`` key.
Each entry in the devices list can be either the path to the device's
mountpoint in the filesystem or a path to the block device in ``/dev``.
The utility to use for resizing can be selected using the ``mode`` config key.
If ``mode`` key is set to ``auto``, then any available utility (either
``growpart`` or ``gpart``) will be used. If neither utility is available, no
error will be raised. If ``mode`` is set to ``growpart``, then the ``growpart``
utility will be used. If this utility is not available on the system, this will
result in an error. If ``mode`` is set to ``off`` or ``false``, then
``cc_growpart`` will take no action.
There is some functionality overlap between this module and the ``growroot``
functionality of ``cloud-initramfs-tools``. However, there are some situations