README.md 12.2 KB
Newer Older
1 2
# Librarian-puppet

3
[![Build Status](https://travis-ci.org/voxpupuli/librarian-puppet.png?branch=master)](https://travis-ci.org/voxpupuli/librarian-puppet)
PeteMS's avatar
PeteMS committed
4

5 6
## Introduction

Tim Sharpe's avatar
Tim Sharpe committed
7
Librarian-puppet is a bundler for your puppet infrastructure.  You can use
8
librarian-puppet to manage the puppet modules your infrastructure depends on,
9
whether the modules come from the [Puppet Forge](https://forge.puppet.com/),
Dan Mindru's avatar
Dan Mindru committed
10
Git repositories or just a path.
11

12
* Librarian-puppet can reuse the dependencies listed in your `Modulefile` or `metadata.json`
13
* Forge modules can be installed from [Puppetlabs Forge](https://forge.puppet.com/) or an internal Forge such as [Pulp](http://www.pulpproject.org/)
14
* Git modules can be installed from a branch, tag or specific commit, optionally using a path inside the repository
15
* Modules can be installed from GitHub using tarballs, without needing Git installed
16
* Modules can be installed from a filesystem path
17
* Module dependencies are resolved transitively without needing to list all the modules explicitly
18

Tim Sharpe's avatar
Tim Sharpe committed
19 20 21 22

Librarian-puppet manages your `modules/` directory for you based on your
`Puppetfile`.  Your `Puppetfile` becomes the authoritative source for what
modules you require and at what version, tag or branch.
23

Tim Sharpe's avatar
Tim Sharpe committed
24 25 26
Once using Librarian-puppet you should not modify the contents of your `modules`
directory.  The individual modules' repos should be updated, tagged with a new
release and the version bumped in your Puppetfile.
27

28 29 30 31
It is based on [Librarian](https://github.com/applicationsonline/librarian), a
framework for writing bundlers, which are tools that resolve, fetch, install,
and isolate a project's dependencies.

32 33
## Versions

34 35 36
Librarian-Puppet 3.0.0 and newer requires Ruby >= 2.0. Use version 2.2.4 if you need support for Puppet 3.7 or earlier, or Ruby 1.9 or earlier. Note that [Puppet 4.10 and newer require Ruby 2.1](https://puppet.com/docs/puppet/4.10/system_requirements.html#prerequisites) or newer.

Librarian-Puppet 2.0.0 and newer requires Ruby >= 1.9 and uses Puppet Forge API v3. For Ruby 1.8 use 1.5.0.
37 38 39

See the [Changelog](Changelog.md) for more details.

40 41
## The Puppetfile

42 43 44 45
Every Puppet repository that uses Librarian-puppet may have a file named
`Puppetfile`, `metadata.json` or `Modulefile` in the root directory of that repository.
The full specification
for which modules your puppet infrastructure repository depends goes in here.
46

47
### Simple usage
48

49 50 51
If no Puppetfile is present, `librarian-puppet` will download all the dependencies
listed in your `metadata.json` or `Modulefile` from the Puppet Forge,
as if the Puppetfile contained
52

53
    forge "https://forgeapi.puppetlabs.com"
54

55
    metadata
56 57


58 59
### Example Puppetfile

60
    forge "https://forgeapi.puppetlabs.com"
Tim Sharpe's avatar
Tim Sharpe committed
61

62 63
    mod 'puppetlabs-razor'
    mod 'puppetlabs-ntp', "0.0.3"
Tim Sharpe's avatar
Tim Sharpe committed
64

65
    mod 'puppetlabs-apt',
66 67
      :git => "git://github.com/puppetlabs/puppetlabs-apt.git"

68
    mod 'puppetlabs-stdlib',
69 70
      :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"

71
    mod 'puppetlabs-apache', '0.6.0',
72 73
      :github_tarball => 'puppetlabs/puppetlabs-apache'

74 75
    mod 'acme-mymodule', :path => './some_folder'

76 77
    exclusion 'acme-bad_module'

78

79
### Recursive module dependency resolution
80

81
When fetching a module all dependencies specified in its
82
`Modulefile`, `metadata.json` and `Puppetfile` will be resolved and installed.
83

84 85
### Puppetfile Breakdown

86
    forge "https://forgeapi.puppetlabs.com"
Tim Sharpe's avatar
Tim Sharpe committed
87 88 89 90 91

This declares that we want to use the official Puppet Labs Forge as our default
source when pulling down modules.  If you run your own local forge, you may
want to change this.

92 93 94 95
    metadata

Download all the dependencies listed in your `metadata.json` or `Modulefile` from the Puppet Forge.

96
    mod 'puppetlabs-razor'
Tim Sharpe's avatar
Tim Sharpe committed
97 98 99 100

Pull in the latest version of the Puppet Labs Razor module from the default
source.

101
    mod 'puppetlabs-ntp', "0.0.3"
Tim Sharpe's avatar
Tim Sharpe committed
102 103 104

Pull in version 0.0.3 of the Puppet Labs NTP module from the default source.

105
    mod 'puppetlabs-apt',
106 107
      :git => "git://github.com/puppetlabs/puppetlabs-apt.git"

Tim Sharpe's avatar
Tim Sharpe committed
108 109
Our puppet infrastructure repository depends on the `apt` module from the
Puppet Labs GitHub repos and checks out the `master` branch.
110

111
    mod 'puppetlabs-apt',
112
      :git => "git://github.com/puppetlabs/puppetlabs-apt.git",
113 114
      :ref => '0.0.3'

Tim Sharpe's avatar
Tim Sharpe committed
115 116
Our puppet infrastructure repository depends on the `apt` module from the
Puppet Labs GitHub repos and checks out a tag of `0.0.3`.
117

118
    mod 'puppetlabs-apt',
119
      :git => "git://github.com/puppetlabs/puppetlabs-apt.git",
120 121
      :ref => 'feature/master/dans_refactor'

Tim Sharpe's avatar
Tim Sharpe committed
122 123
Our puppet infrastructure repository depends on the `apt` module from the
Puppet Labs GitHub repos and checks out the `dans_refactor` branch.
124

Tim Sharpe's avatar
Tim Sharpe committed
125 126
When using a Git source, we do not have to use a `:ref =>`.
If we do not, then librarian-puppet will assume we meant the `master` branch.
127

Tim Sharpe's avatar
Tim Sharpe committed
128 129
If we use a `:ref =>`, we can use anything that Git will recognize as a ref.
This includes any branch name, tag name, SHA, or SHA unique prefix. If we use a
Joshua Johnson's avatar
Joshua Johnson committed
130
branch, we can later ask Librarian-puppet to update the module by fetching the
Tim Sharpe's avatar
Tim Sharpe committed
131
most recent version of the module from that same branch.
132

133 134 135 136 137
Note that Librarian-puppet recognizes the [r10k Puppetfile's](https://github.com/puppetlabs/r10k/blob/master/doc/puppetfile.mkd) additional
options, `:tag`, `:commit`, and `:branch`, but only as aliases for `:ref`.
That is, there is no implementation of r10k's optimizations around fetching
these different types of git objects.

Tim Sharpe's avatar
Tim Sharpe committed
138 139 140 141 142 143
The Git source also supports a `:path =>` option. If we use the path option,
Librarian-puppet will navigate down into the Git repository and only use the
specified subdirectory. Some people have the habit of having a single repository
with many modules in it. If we need a module from such a repository, we can
use the `:path =>` option here to help Librarian-puppet drill down and find the
module subdirectory.
144

145
    mod 'puppetlabs-apt',
146
      :git => "git://github.com/fake/puppet-modules.git",
147 148
      :path => "modules/apt"

Tim Sharpe's avatar
Tim Sharpe committed
149 150
Our puppet infrastructure repository depends on the `apt` module, which we have
stored as a directory under our `puppet-modules` git repos.
151

152 153 154 155 156 157 158 159 160 161 162
    mod 'puppetlabs-apache', '0.6.0',
      :github_tarball => 'puppetlabs/puppetlabs-apache'

Our puppet infrastructure repository depends on the `puppetlabs-apache` module,
to be downloaded from GitHub tarball.

    mod 'acme-mymodule', :path => './some_folder'

Our puppet infrastructure repository depends on the `acme-mymodule` module,
which is already in the filesystem.

163 164 165 166
    exclusion 'acme-bad_module'

Exclude the module `acme-bad_module` from resolution and installation.

167 168
## How to Use

Tim Sharpe's avatar
Tim Sharpe committed
169
Install librarian-puppet:
170

Tom De Vylder's avatar
Tom De Vylder committed
171
    $ gem install librarian-puppet
172 173 174 175 176

Prepare your puppet infrastructure repository:

    $ cd ~/path/to/puppet-inf-repos
    $ (git) rm -rf modules
Tim Sharpe's avatar
Tim Sharpe committed
177
    $ librarian-puppet init
178

Tim Sharpe's avatar
Tim Sharpe committed
179 180 181 182
Librarian-puppet takes over your `modules/` directory, and will always
reinstall (if missing) the modules listed the `Puppetfile.lock` into your
`modules/` directory, therefore you do not need your `modules/` directory to be
tracked in Git.
183

Tim Sharpe's avatar
Tim Sharpe committed
184 185
Librarian-puppet uses a `.tmp/` directory for tempfiles and caches. You should
not track this directory in Git.
186

Tim Sharpe's avatar
Tim Sharpe committed
187 188
Running `librarian-puppet init` will create a skeleton Puppetfile for you as
well as adding `tmp/` and `modules/` to your `.gitignore`.
189 190 191

    $ librarian-puppet install [--clean] [--verbose]

Tim Sharpe's avatar
Tim Sharpe committed
192 193 194 195
This command looks at each `mod` declaration and fetches the module from the
source specified.  This command writes the complete resolution into
`Puppetfile.lock` and then copies all of the fetched modules into your
`modules/` directory, overwriting whatever was there before.
196

197 198 199
Librarian-puppet support both v1 and v3 of the Puppet Forge API.
Specify a specific API version when installing modules:

200 201
    $ librarian-puppet install --use-v1-api # this is default; ignored for official Puppet Forge
    $ librarian-puppet install --no-use-v1-api # use the v3 API; default for official Puppet Forge
202

203
Please note that this does not apply for the official Puppet Forge, where v3 is used by default.
204

Tim Sharpe's avatar
Tim Sharpe committed
205
Get an overview of your `Puppetfile.lock` with:
206 207 208

    $ librarian-puppet show

Tim Sharpe's avatar
Tim Sharpe committed
209
Inspect the details of specific resolved dependencies with:
210 211 212

    $ librarian-puppet show NAME1 [NAME2, ...]

Tim Sharpe's avatar
Tim Sharpe committed
213
Find out which dependencies are outdated and may be updated:
214 215 216

    $ librarian-puppet outdated [--verbose]

Tim Sharpe's avatar
Tim Sharpe committed
217
Update the version of a dependency:
218 219 220 221 222 223

    $ librarian-puppet update apt [--verbose]
    $ git diff Puppetfile.lock
    $ git add Puppetfile.lock
    $ git commit -m "bumped the version of apt up to 0.0.4."

224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271
## Configuration

Configuration comes from three sources with the following highest-to-lowest
precedence:

* The local config (`./.librarian/puppet/config`)
* The environment
* The global config (`~/.librarian/puppet/config`)

You can inspect the final configuration with:

    $ librarian-puppet config

You can find out where a particular key is set with:

    $ librarian-puppet config KEY

You can set a key at the global level with:

    $ librarian-puppet config KEY VALUE --global

And remove it with:

    $ librarian-puppet config KEY --global --delete

You can set a key at the local level with:

    $ librarian-puppet config KEY VALUE --local

And remove it with:

    $ librarian-puppet config KEY --local --delete

You cannot set or delete environment-level config keys with the CLI.

Configuration set at either the global or local level will affect subsequent
invocations of `librarian-puppet`. Configurations set at the environment level are
not saved and will not affect subsequent invocations of `librarian-puppet`.

You can pass a config at the environment level by taking the original config key
and transforming it: replace hyphens (`-`) with underscores (`_`) and periods
(`.`) with doubled underscores (`__`), uppercase, and finally prefix with
`LIBRARIAN_PUPPET_`. For example, to pass a config in the environment for the key
`part-one.part-two`, set the environment variable
`LIBRARIAN_PUPPET_PART_ONE__PART_TWO`.

Configuration affects how various commands operate.

272
* The `path` config sets the directory to install to. If a relative
273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288
  path, it is relative to the directory containing the `Puppetfile`. The
  equivalent environment variable is `LIBRARIAN_PUPPET_PATH`.

* The `tmp` config sets the cache directory for librarian. If a relative
  path, it is relative to the directory containing the `Puppetfile`. The
  equivalent environment variable is `LIBRARIAN_PUPPET_TMP`.

Configuration can be set by passing specific options to other commands.

* The `path` config can be set at the local level by passing the `--path` option
  to the `install` command. It can be unset at the local level by passing the
  `--no-path` option to the `install` command. Note that if this is set at the
  environment or global level then, even if `--no-path` is given as an option,
  the environment or global config will be used.


289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310
## Rsync Option

The default convergence strategy between the cache and the module directory is
to execute an `rm -r` on the module directory and just `cp -r` from the cache.
This causes the module to be removed from the module path every time librarian
puppet updates, regardless of whether the content has changed. This can cause
some problems in environments with lots of change. The problem arises when the
module directory gets removed while Puppet is trying to read files inside it.
The `puppet master` process will lose its CWD and the catalog will fail to
compile. To avoid this, you can use `rsync` to implement a more conservative
convergence strategy. This will use `rsync` with the `-avz` and `--delete`
flags instead of a `rm -r` and `cp -r`. To use this feature, just set the
`rsync` configuration setting to `true`.

    $ librarian-puppet config rsync true --global

Alternatively, using an environment variable:

    LIBRARIAN_PUPPET_RSYNC='true'

Note that the directories will still be purged if you run librarian-puppet with
the --clean or --destructive flags.
311

312 313
## How to Contribute

Tim Sharpe's avatar
Tim Sharpe committed
314 315
 * Pull requests please.
 * Bonus points for feature branches.
316 317 318

## Reporting Issues

Tim Sharpe's avatar
Tim Sharpe committed
319 320
Bug reports to the github issue tracker please.
Please include:
321

Joshua Johnson's avatar
Joshua Johnson committed
322
 * Relevant `Puppetfile` and `Puppetfile.lock` files
323
 * Version of ruby, librarian-puppet, and puppet
Tim Sharpe's avatar
Tim Sharpe committed
324 325 326
 * What distro
 * Please run the `librarian-puppet` commands in verbose mode by using the
  `--verbose` flag, and include the verbose output in the bug report as well.
327

Tim Sharpe's avatar
Tim Sharpe committed
328

329
## License
330
Please see the [LICENSE](https://github.com/voxpupuli/librarian-puppet/blob/master/LICENSE)
Tim Sharpe's avatar
Tim Sharpe committed
331
file.