Commit 127f0f50 authored by Scott Moser's avatar Scott Moser

doc: make the RST files consistently formated and other improvements.

The biggest things here are:
 * move doc/sources/*/README.rst to doc/rtd/topics/datasources
   This gives each datasource a page in the rtd docs, which make
   it easier to read.
 * consistently use the same header style throughout.
   As suggested at
   http://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html
   use:
     # with overline, for parts
     * with overline, for chapters
     =, for sections
     -, for subsections
     ^, for subsubsections
     “, for paragraphs

Also, move and re-format vendor-data documentation to rtd.
parent 25c218e5
=====================
*********************
Hacking on cloud-init
=====================
*********************
This document describes how to contribute changes to cloud-init.
It assumes you have a `Launchpad`_ account, and refers to your launchpad user
as ``LP_USER`` throughout.
Do these things once
--------------------
====================
* To contribute, you must sign the Canonical `contributor license agreement`_
......@@ -21,11 +21,12 @@ Do these things once
There is more information on Launchpad as a git hosting site in
`Launchpad git documentation`_.
* Create a new remote pointing to your personal Launchpad repository
* Create a new remote pointing to your personal Launchpad repository.
This is equivalent to 'fork' on github.
This is equivalent to 'fork' on github::
.. code:: sh
git remote add LP_USER git+ssh://LP_USER@git.launchpad.net/~LP_USER/cloud-init
git remote add LP_USER ssh://LP_USER@git.launchpad.net/~LP_USER/cloud-init
git push LP_USER master
.. _repository: https://git.launchpad.net/cloud-init
......@@ -34,7 +35,7 @@ Do these things once
.. _Launchpad git documentation: https://help.launchpad.net/Code/Git
Do these things for each feature or bug
---------------------------------------
=======================================
* Create a new topic branch for your work::
......
Overview
--------
========
This was implemented because it has been a common feature request that there be
a way to specify how cloud-config yaml "dictionaries" provided as user-data are
......@@ -52,7 +52,7 @@ into a more useful list, thus reducing duplication that would have had to
occur in the previous method to accomplish the same result.
Customizability
---------------
===============
Since the above merging algorithm may not always be the desired merging
algorithm (like how the previous merging algorithm was not always the preferred
......@@ -96,41 +96,45 @@ An example of one of these merging classes is the following:
merged[k] = v
return merged
As you can see there is a '_on_dict' method here that will be given a source value
and a value to merge with. The result will be the merged object. This code itself
is called by another merging class which 'directs' the merging to happen by
analyzing the types of the objects to merge and attempting to find a know object
that will merge that type. I will avoid pasting that here, but it can be found
in the `mergers/__init__.py` file (see `LookupMerger` and `UnknownMerger`).
So following the typical cloud-init way of allowing source code to be downloaded
and used dynamically, it is possible for users to inject there own merging files
to handle specific types of merging as they choose (the basic ones included will
handle lists, dicts, and strings). Note how each merge can have options associated
with it which affect how the merging is performed, for example a dictionary merger
can be told to overwrite instead of attempt to merge, or a string merger can be
told to append strings instead of discarding other strings to merge with.
As you can see there is a '_on_dict' method here that will be given a source
value and a value to merge with. The result will be the merged object. This
code itself is called by another merging class which 'directs' the merging to
happen by analyzing the types of the objects to merge and attempting to find a
know object that will merge that type. I will avoid pasting that here, but it
can be found in the `mergers/__init__.py` file (see `LookupMerger` and
`UnknownMerger`).
So following the typical cloud-init way of allowing source code to be
downloaded and used dynamically, it is possible for users to inject there own
merging files to handle specific types of merging as they choose (the basic
ones included will handle lists, dicts, and strings). Note how each merge can
have options associated with it which affect how the merging is performed, for
example a dictionary merger can be told to overwrite instead of attempt to
merge, or a string merger can be told to append strings instead of discarding
other strings to merge with.
How to activate
---------------
===============
There are a few ways to activate the merging algorithms, and to customize them
for your own usage.
1. The first way involves the usage of MIME messages in cloud-init to specify
multipart documents (this is one way in which multiple cloud-config is joined
together into a single cloud-config). Two new headers are looked for, both
of which can define the way merging is done (the first header to exist wins).
These new headers (in lookup order) are 'Merge-Type' and 'X-Merge-Type'. The value
should be a string which will satisfy the new merging format defintion (see
below for this format).
multipart documents (this is one way in which multiple cloud-config is
joined together into a single cloud-config). Two new headers are looked
for, both of which can define the way merging is done (the first header to
exist wins). These new headers (in lookup order) are 'Merge-Type' and
'X-Merge-Type'. The value should be a string which will satisfy the new
merging format defintion (see below for this format).
2. The second way is actually specifying the merge-type in the body of the
cloud-config dictionary. There are 2 ways to specify this, either as a string
or as a dictionary (see format below). The keys that are looked up for this
definition are the following (in order), 'merge_how', 'merge_type'.
cloud-config dictionary. There are 2 ways to specify this, either as a
string or as a dictionary (see format below). The keys that are looked up
for this definition are the following (in order), 'merge_how',
'merge_type'.
String format
*************
-------------
The string format that is expected is the following.
......@@ -142,14 +146,15 @@ The class name there will be connected to class names used when looking for the
class that can be used to merge and options provided will be given to the class
on construction of that class.
For example, the default string that is used when none is provided is the following:
For example, the default string that is used when none is provided is the
following:
::
list()+dict()+str()
Dictionary format
*****************
-----------------
In cases where a dictionary can be used to specify the same information as the
string format (ie option #2 of above) it can be used, for example.
......@@ -164,7 +169,7 @@ This would be the equivalent format for default string format but in dictionary
form instead of string form.
Specifying multiple types and its effect
----------------------------------------
========================================
Now you may be asking yourself, if I specify a merge-type header or dictionary
for every cloud-config that I provide, what exactly happens?
......@@ -174,13 +179,13 @@ first one on that stack is the default merging classes, this set of mergers
will be used when the first cloud-config is merged with the initial empty
cloud-config dictionary. If the cloud-config that was just merged provided a
set of merging classes (via the above formats) then those merging classes will
be pushed onto the stack. Now if there is a second cloud-config to be merged then
the merging classes from the cloud-config before the first will be used (not the
default) and so on. This way a cloud-config can decide how it will merge with a
cloud-config dictionary coming after it.
be pushed onto the stack. Now if there is a second cloud-config to be merged
then the merging classes from the cloud-config before the first will be used
(not the default) and so on. This way a cloud-config can decide how it will
merge with a cloud-config dictionary coming after it.
Other uses
----------
==========
In addition to being used for merging user-data sections, the default merging
algorithm for merging 'conf.d' yaml files (which form an initial yaml config
......@@ -192,3 +197,5 @@ merging, for example).
Note, however, that merge algorithms are not used *across* types of
configuration. As was the case before merging was implemented,
user-data will overwrite conf.d configuration without merging.
.. vi: textwidth=78
.. _index:
=====================
.. http://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html
.. As suggested at link above for headings use:
.. # with overline, for parts
.. * with overline, for chapters
.. =, for sections
.. -, for subsections
.. ^, for subsubsections
.. “, for paragraphs
#############
Documentation
=====================
#############
.. rubric:: Everything about cloud-init, a set of **python** scripts and utilities to make your cloud images be all they can be!
.. rubric:: Everything about cloud-init, a set of **python** scripts and
utilities to make your cloud images be all they can be!
*******
Summary
-----------------
`Cloud-init`_ is the *defacto* multi-distribution package that handles early initialization of a cloud instance.
*******
`Cloud-init`_ is the *defacto* multi-distribution package that handles early
initialization of a cloud instance.
----
.. toctree::
:maxdepth: 2
topics/capabilities
topics/availability
topics/format
topics/dir_layout
topics/examples
topics/datasources
topics/logging
topics/modules
topics/merging
topics/moreinfo
topics/hacking
topics/capabilities.rst
topics/availability.rst
topics/format.rst
topics/dir_layout.rst
topics/examples.rst
topics/datasources.rst
topics/logging.rst
topics/modules.rst
topics/merging.rst
topics/vendordata.rst
topics/moreinfo.rst
topics/hacking.rst
.. _Cloud-init: https://launchpad.net/cloud-init
.. vi: textwidth=78
============
************
Availability
============
************
It is currently installed in the `Ubuntu Cloud Images`_ and also in the official `Ubuntu`_ images available on EC2.
It is currently installed in the `Ubuntu Cloud Images`_ and also in the official `Ubuntu`_ images available on EC2, Azure, GCE and many other clouds.
Versions for other systems can be (or have been) created for the following distributions:
......@@ -18,3 +18,4 @@ So ask your distribution provider where you can obtain an image with it built-in
.. _Ubuntu Cloud Images: http://cloud-images.ubuntu.com/
.. _Ubuntu: http://www.ubuntu.com/
.. vi: textwidth=78
=====================
************
Capabilities
=====================
************
- Setting a default locale
- Setting a instance hostname
......@@ -9,16 +9,18 @@ Capabilities
- Setting up ephemeral mount points
User configurability
--------------------
====================
`Cloud-init`_ 's behavior can be configured via user-data.
User-data can be given by the user at instance launch time.
This is done via the ``--user-data`` or ``--user-data-file`` argument to ec2-run-instances for example.
This is done via the ``--user-data`` or ``--user-data-file`` argument to
ec2-run-instances for example.
* Check your local clients documentation for how to provide a `user-data` string
or `user-data` file for usage by cloud-init on instance creation.
* Check your local clients documentation for how to provide a `user-data`
string or `user-data` file for usage by cloud-init on instance creation.
.. _Cloud-init: https://launchpad.net/cloud-init
.. vi: textwidth=78
.. _datasources:
===========
***********
Datasources
===========
----------------------
What is a datasource?
----------------------
***********
Datasources are sources of configuration data for cloud-init that typically come
from the user (aka userdata) or come from the stack that created the configuration
drive (aka metadata). Typical userdata would include files, yaml, and shell scripts
while typical metadata would include server name, instance id, display name and other
cloud specific details. Since there are multiple ways to provide this data (each cloud
solution seems to prefer its own way) internally a datasource abstract class was
created to allow for a single way to access the different cloud systems methods
to provide this data through the typical usage of subclasses.
What is a datasource?
=====================
Datasources are sources of configuration data for cloud-init that typically
come from the user (aka userdata) or come from the stack that created the
configuration drive (aka metadata). Typical userdata would include files,
yaml, and shell scripts while typical metadata would include server name,
instance id, display name and other cloud specific details. Since there are
multiple ways to provide this data (each cloud solution seems to prefer its
own way) internally a datasource abstract class was created to allow for a
single way to access the different cloud systems methods to provide this data
through the typical usage of subclasses.
The current interface that a datasource object must provide is the following:
......@@ -70,131 +71,28 @@ The current interface that a datasource object must provide is the following:
def get_package_mirror_info(self)
---
EC2
---
The EC2 datasource is the oldest and most widely used datasource that cloud-init
supports. This datasource interacts with a *magic* ip that is provided to the
instance by the cloud provider. Typically this ip is ``169.254.169.254`` of which
at this ip a http server is provided to the instance so that the instance can make
calls to get instance userdata and instance metadata.
Metadata is accessible via the following URL:
::
GET http://169.254.169.254/2009-04-04/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
reservation-id
security-groups
Userdata is accessible via the following URL:
::
GET http://169.254.169.254/2009-04-04/user-data
1234,fred,reboot,true | 4512,jimbo, | 173,,,
Note that there are multiple versions of this data provided, cloud-init
by default uses **2009-04-04** but newer versions can be supported with
relative ease (newer versions have more data exposed, while maintaining
backward compatibility with the previous versions).
To see which versions are supported from your cloud provider use the following URL:
::
GET http://169.254.169.254/
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
...
latest
------------
Config Drive
------------
.. include:: ../../sources/configdrive/README.rst
----------
OpenNebula
----------
.. include:: ../../sources/opennebula/README.rst
---------
Alt cloud
---------
.. include:: ../../sources/altcloud/README.rst
--------
No cloud
--------
.. include:: ../../sources/nocloud/README.rst
----
MAAS
----
*TODO*
For now see: http://maas.ubuntu.com/
----------
CloudStack
----------
.. include:: ../../sources/cloudstack/README.rst
---
OVF
---
*TODO*
For now see: https://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/sources/ovf/
---------
OpenStack
---------
.. include:: ../../sources/openstack/README.rst
-------------
Fallback/None
-------------
This is the fallback datasource when no other datasource can be selected. It is
the equivalent of a *empty* datasource in that it provides a empty string as userdata
and a empty dictionary as metadata. It is useful for testing as well as for when
you do not have a need to have an actual datasource to meet your instance
requirements (ie you just want to run modules that are not concerned with any
external data). It is typically put at the end of the datasource search list
so that if all other datasources are not matched, then this one will be so that
the user is not left with an inaccessible instance.
**Note:** the instance id that this datasource provides is ``iid-datasource-none``.
.. _boto: http://docs.pythonboto.org/en/latest/
Datasource Documentation
========================
The following is a list of the implemented datasources.
Follow for more information.
.. toctree::
:maxdepth: 2
datasources/altcloud.rst
datasources/azure.rst
datasources/cloudsigma.rst
datasources/cloudstack.rst
datasources/configdrive.rst
datasources/digitalocean.rst
datasources/ec2.rst
datasources/maas.rst
datasources/nocloud.rst
datasources/opennebula.rst
datasources/openstack.rst
datasources/ovf.rst
datasources/smartos.rst
datasources/fallback.rst
.. vi: textwidth=78
Alt Cloud
=========
The datasource altcloud will be used to pick up user data on `RHEVm`_ and `vSphere`_.
RHEVm
~~~~~~
-----
For `RHEVm`_ v3.0 the userdata is injected into the VM using floppy
injection via the `RHEVm`_ dashboard "Custom Properties".
......@@ -38,7 +41,7 @@ data to it using the Delta Cloud.
For more information on Delta Cloud see: http://deltacloud.apache.org
vSphere
~~~~~~~~
-------
For VMWare's `vSphere`_ the userdata is injected into the VM as an ISO
via the cdrom. This can be done using the `vSphere`_ dashboard
......@@ -53,7 +56,7 @@ ISO on the data store.
For example, to pass the same ``simple_script.bash`` to vSphere:
Create the ISO
-----------------
^^^^^^^^^^^^^^
.. sourcecode:: sh
......@@ -67,7 +70,7 @@ NOTE: The file name on the ISO must be: ``user-data.txt``
% genisoimage -o user-data.iso -r my-iso
Verify the ISO
-----------------
^^^^^^^^^^^^^^
.. sourcecode:: sh
......@@ -85,3 +88,4 @@ For more information on Delta Cloud see: http://deltacloud.apache.org
.. _RHEVm: https://www.redhat.com/virtualization/rhev/desktop/rhevm/
.. _vSphere: https://www.vmware.com/products/datacenter-virtualization/vsphere/overview.html
.. vi: textwidth=78
================
Azure Datasource
================
Azure
=====
This datasource finds metadata and user-data from the Azure cloud platform.
......@@ -44,7 +43,7 @@ following things:
- generate a x509 certificate and send that to the endpoint
waagent.conf config
~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^
in order to use waagent.conf with cloud-init, the following settings are recommended. Other values can be changed or set to the defaults.
::
......@@ -71,7 +70,7 @@ That agent command will take affect as if it were specified in system config.
Example:
.. code::
.. sourcecode:: xml
<wa:ProvisioningSection>
<wa:Version>1.0</wa:Version>
......@@ -111,7 +110,7 @@ hostname is set, and will have the 'interface' in its environment. If
An example might be:
command: ["sh", "-c", "killall dhclient; dhclient $interface"]
.. code::
.. code:: yaml
datasource:
agent_command
......@@ -126,7 +125,6 @@ An example might be:
# the method 'bounce' command.
command: "builtin"
hostname_command: "hostname"
}
hostname
--------
......@@ -153,3 +151,5 @@ cloud-init handles this by setting the hostname in the DataSource's 'get_data'
method via '``hostname $HostName``', and then bouncing the interface. This
behavior can be configured or disabled in the datasource config. See
'Configuration' above.
.. vi: textwidth=78
=====================
CloudSigma Datasource
=====================
CloudSigma
==========
This datasource finds metadata and user-data from the `CloudSigma`_ cloud platform.
Data transfer occurs through a virtual serial port of the `CloudSigma`_'s VM and the
presence of network adapter is **NOT** a requirement,
See `server context`_ in the public documentation for more information.
This datasource finds metadata and user-data from the `CloudSigma`_ cloud
platform. Data transfer occurs through a virtual serial port of the
`CloudSigma`_'s VM and the presence of network adapter is **NOT** a
requirement, See `server context`_ in the public documentation for more
information.
Setting a hostname
~~~~~~~~~~~~~~~~~~
By default the name of the server will be applied as a hostname on the first boot.
------------------
By default the name of the server will be applied as a hostname on the first
boot.
Providing user-data
~~~~~~~~~~~~~~~~~~~
-------------------
You can provide user-data to the VM using the dedicated `meta field`_ in the `server context`_
``cloudinit-user-data``. By default *cloud-config* format is expected there and the ``#cloud-config``
header could be omitted. However since this is a raw-text field you could provide any of the valid
`config formats`_.
You can provide user-data to the VM using the dedicated `meta field`_ in the
`server context`_ ``cloudinit-user-data``. By default *cloud-config* format is
expected there and the ``#cloud-config`` header could be omitted. However
since this is a raw-text field you could provide any of the valid `config
formats`_.
You have the option to encode your user-data using Base64. In order to do that you have to add the
``cloudinit-user-data`` field to the ``base64_fields``. The latter is a comma-separated field with
all the meta fields whit base64 encoded values.
You have the option to encode your user-data using Base64. In order to do that
you have to add the ``cloudinit-user-data`` field to the ``base64_fields``.
The latter is a comma-separated field with all the meta fields whit base64
encoded values.
If your user-data does not need an internet connection you can create a
`meta field`_ in the `server context`_ ``cloudinit-dsmode`` and set "local" as value.
If this field does not exist the default value is "net".
If your user-data does not need an internet connection you can create a `meta
field`_ in the `server context`_ ``cloudinit-dsmode`` and set "local" as
value. If this field does not exist the default value is "net".
.. _CloudSigma: http://cloudsigma.com/
.. _server context: http://cloudsigma-docs.readthedocs.org/en/latest/server_context.html
.. _meta field: http://cloudsigma-docs.readthedocs.org/en/latest/meta.html
.. _config formats: http://cloudinit.readthedocs.org/en/latest/topics/format.html
.. vi: textwidth=78
CloudStack
==========
`Apache CloudStack`_ expose user-data, meta-data, user password and account
sshkey thru the Virtual-Router. For more details on meta-data and user-data,
refer the `CloudStack Administrator Guide`_.
......@@ -12,7 +15,7 @@ is the Virtual Router IP:
http://10.1.1.1/latest/meta-data/{metadata type}
Configuration
~~~~~~~~~~~~~
-------------
Apache CloudStack datasource can be configured as follows:
......@@ -26,4 +29,6 @@ Apache CloudStack datasource can be configured as follows:
.. _Apache CloudStack: http://cloudstack.apache.org/
.. _CloudStack Administrator Guide: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/virtual_machines.html#user-data-and-meta-data
\ No newline at end of file
.. _CloudStack Administrator Guide: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/latest/virtual_machines.html#user-data-and-meta-data
.. vi: textwidth=78
The configuration drive datasource supports the `OpenStack`_ configuration drive disk.
Config Drive
============
The configuration drive datasource supports the `OpenStack`_ configuration
drive disk.
See `the config drive extension`_ and `introduction`_ in the public
documentation for more information.
......@@ -6,14 +10,14 @@ The configuration drive datasource supports the `OpenStack`_ configuration drive
By default, cloud-init does *always* consider this source to be a full-fledged
datasource. Instead, the typical behavior is to assume it is really only
present to provide networking information. Cloud-init will copy off the
network information, apply it to the system, and then continue on. The
"full" datasource could then be found in the EC2 metadata service. If this is
not the case then the files contained on the located drive must provide equivalents
to what the EC2 metadata service would provide (which is typical of the version
2 support listed below)
network information, apply it to the system, and then continue on. The "full"
datasource could then be found in the EC2 metadata service. If this is not the
case then the files contained on the located drive must provide equivalents to
what the EC2 metadata service would provide (which is typical of the version 2
support listed below)
Version 1
~~~~~~~~~
---------
The following criteria are required to as a config drive:
......@@ -31,8 +35,8 @@ The following criteria are required to as a config drive:
This file is laid down by nova in order to pass static networking
information to the guest. Cloud-init will copy it off of the config-drive
and into /etc/network/interfaces (or convert it to RH format) as soon as it can,
and then attempt to bring up all network interfaces.
and into /etc/network/interfaces (or convert it to RH format) as soon as
it can, and then attempt to bring up all network interfaces.
``/root/.ssh/authorized_keys``
......@@ -46,7 +50,7 @@ The following criteria are required to as a config drive:
formatted.
Version 2
~~~~~~~~~
---------
The following criteria are required to as a config drive:
......@@ -70,9 +74,10 @@ The following criteria are required to as a config drive:
- meta-data.json (not mandatory)
Keys and values
~~~~~~~~~~~~~~~
---------------
Cloud-init's behavior can be modified by keys found in the meta.js (version 1 only) file in the following ways.
Cloud-init's behavior can be modified by keys found in the meta.js (version 1
only) file in the following ways.
::
......@@ -121,3 +126,4 @@ what all can be present here.
.. _iso9660: https://en.wikipedia.org/wiki/ISO_9660
.. _vfat: https://en.wikipedia.org/wiki/File_Allocation_Table
.. _the config drive extension: http://docs.openstack.org/user-guide/content/config-drive.html
.. vi: textwidth=78
The `DigitalOcean`_ datasource consumes the content served from DigitalOcean's `metadata service`_. This
metadata service serves information about the running droplet via HTTP over the link local address
169.254.169.254. The metadata API endpoints are fully described at
`https://developers.digitalocean.com/metadata/ <https://developers.digitalocean.com/metadata/>`_.
Digital Ocean
=============
The `DigitalOcean`_ datasource consumes the content served from DigitalOcean's
`metadata service`_. This metadata service serves information about the
running droplet via HTTP over the link local address 169.254.169.254. The
metadata API endpoints are fully described at
`https://developers.digitalocean.com/metadata/
<https://developers.digitalocean.com/metadata/>`_.
Configuration
~~~~~~~~~~~~~
-------------
DigitalOcean's datasource can be configured as follows:
......@@ -19,3 +24,5 @@ DigitalOcean's datasource can be configured as follows:
.. _DigitalOcean: http://digitalocean.com/
.. _metadata service: https://developers.digitalocean.com/metadata/
.. _Full documentation: https://developers.digitalocean.com/metadata/
.. vi: textwidth=78
Amazon EC2
==========
The EC2 datasource is the oldest and most widely used datasource that
cloud-init supports. This datasource interacts with a *magic* ip that is
provided to the instance by the cloud provider. Typically this ip is
``169.254.169.254`` of which at this ip a http server is provided to the
instance so that the instance can make calls to get instance userdata and
instance metadata.
Metadata is accessible via the following URL:
::
GET http://169.254.169.254/2009-04-04/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4