Commit 7fa65ed6 authored by Stig Mathisen's avatar Stig Mathisen

Imported Upstream version 0.9.7

parents
pkg/
Gemfile.lock
tmp/
Copyright (c) 2012 Tim Sharpe
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# Librarian-puppet
## Introduction
Librarian-puppet is a bundler for your puppet infrastructure. You can use
librarian-puppet to manage the puppet modules your infrastructure depends on.
It is based on [Librarian](https://github.com/applicationsonline/librarian), a
framework for writing bundlers, which are tools that resolve, fetch, install,
and isolate a project's dependencies.
Librarian-puppet manages your `modules/` directory for you based on your
`Puppetfile`. Your `Puppetfile` becomes the authoritative source for what
modules you require and at what version, tag or branch.
Once using Librarian-puppet you should not modify the contents of your `modules`
directory. The individual modules' repos should be updated, tagged with a new
release and the version bumped in your Puppetfile.
## The Puppetfile
Every Puppet repository that uses Librarian-puppet will have a file named
`Puppetfile` in the root directory of that repository. The full specification
for which modules your puppet infrastructure repository depends goes in here.
### Example Puppetfile
forge "http://forge.puppetlabs.com"
mod "puppetlabs/razor"
mod "puppetlabs/ntp", "0.0.3"
mod "apt",
:git => "git://github.com/puppetlabs/puppetlabs-apt.git"
mod "stdlib",
:git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"
*See [jenkins-appliance](https://github.com/aussielunix/jenkins-appliance) for
a puppet repo already setup to use librarian-puppet.*
### Puppetfile Breakdown
forge "http://forge.puppetlabs.com"
This declares that we want to use the official Puppet Labs Forge as our default
source when pulling down modules. If you run your own local forge, you may
want to change this.
mod "puppetlabs/razor"
Pull in the latest version of the Puppet Labs Razor module from the default
source.
mod "puppetlabs/ntp", "0.0.3"
Pull in version 0.0.3 of the Puppet Labs NTP module from the default source.
mod "apt",
:git => "git://github.com/puppetlabs/puppetlabs-apt.git"
Our puppet infrastructure repository depends on the `apt` module from the
Puppet Labs GitHub repos and checks out the `master` branch.
mod "apt",
:git => "git://github.com/puppetlabs/puppetlabs-apt.git",
:ref => '0.0.3'
Our puppet infrastructure repository depends on the `apt` module from the
Puppet Labs GitHub repos and checks out a tag of `0.0.3`.
mod "apt",
:git => "git://github.com/puppetlabs/puppetlabs-apt.git",
:ref => 'feature/master/dans_refactor'
Our puppet infrastructure repository depends on the `apt` module from the
Puppet Labs GitHub repos and checks out the `dans_refactor` branch.
When using a Git source, we do not have to use a `:ref =>`.
If we do not, then librarian-puppet will assume we meant the `master` branch.
If we use a `:ref =>`, we can use anything that Git will recognize as a ref.
This includes any branch name, tag name, SHA, or SHA unique prefix. If we use a
branch, we can later ask Librarian-pupet to update the modulek by fetching the
most recent version of the module from that same branch.
The Git source also supports a `:path =>` option. If we use the path option,
Librarian-puppet will navigate down into the Git repository and only use the
specified subdirectory. Some people have the habit of having a single repository
with many modules in it. If we need a module from such a repository, we can
use the `:path =>` option here to help Librarian-puppet drill down and find the
module subdirectory.
mod "apt",
:git => "git://github.com/fake/puppet-modules.git",
:path => "modules/apt"
Our puppet infrastructure repository depends on the `apt` module, which we have
stored as a directory under our `puppet-modules` git repos.
## How to Use
Install librarian-puppet:
$ gem install librarian-puppet
Prepare your puppet infrastructure repository:
$ cd ~/path/to/puppet-inf-repos
$ (git) rm -rf modules
$ librarian-puppet init
Librarian-puppet takes over your `modules/` directory, and will always
reinstall (if missing) the modules listed the `Puppetfile.lock` into your
`modules/` directory, therefore you do not need your `modules/` directory to be
tracked in Git.
Librarian-puppet uses a `.tmp/` directory for tempfiles and caches. You should
not track this directory in Git.
Running `librarian-puppet init` will create a skeleton Puppetfile for you as
well as adding `tmp/` and `modules/` to your `.gitignore`.
$ librarian-puppet install [--clean] [--verbose]
This command looks at each `mod` declaration and fetches the module from the
source specified. This command writes the complete resolution into
`Puppetfile.lock` and then copies all of the fetched modules into your
`modules/` directory, overwriting whatever was there before.
Get an overview of your `Puppetfile.lock` with:
$ librarian-puppet show
Inspect the details of specific resolved dependencies with:
$ librarian-puppet show NAME1 [NAME2, ...]
Find out which dependencies are outdated and may be updated:
$ librarian-puppet outdated [--verbose]
Update the version of a dependency:
$ librarian-puppet update apt [--verbose]
$ git diff Puppetfile.lock
$ git add Puppetfile.lock
$ git commit -m "bumped the version of apt up to 0.0.4."
## How to Contribute
* Pull requests please.
* Bonus points for feature branches.
## Reporting Issues
Bug reports to the github issue tracker please.
Please include:
* relevant `Puppetfile` and `Puppetfile.lock` files.
* version of ruby, librarian-puppet
* What distro
* Please run the `librarian-puppet` commands in verbose mode by using the
`--verbose` flag, and include the verbose output in the bug report as well.
## Changelog
### 0.9.0
* Initial release
### 0.9.1
* Proper error message when a module that is sourced from the forge does not
exist.
* Added support for annotated tags as git references.
* `librarian-puppet init` adds `.tmp/` to gitignore instead of `tmp/`.
* Fixed syntax error in the template Puppetfile created by `librarian-puppet
init`.
* Checks for `lib/puppet` as well as `manifests/` when checking if the git
repository is a valid module.
* When a user specifies `<foo>/<bar>` as the name of a module sources from a
git repository, assume the module name is actually `<bar>`.
* Fixed gem description and summary in gemspec.
## License
Please see the [LICENSE](https://github.com/rodjek/librarian-puppet/blob/master/LICENSE)
file.
#!/usr/bin/env ruby
lib = File.expand_path('../../lib', __FILE__)
vendor = File.expand_path('../../vendor/librarian/lib', __FILE__)
$:.unshift(lib) unless $:.include?(lib)
$:.unshift(vendor) unless $:.include?(vendor)
require 'librarian/puppet/cli'
Librarian::Puppet::Cli.bin!
require 'librarian'
require 'puppet'
require 'fileutils'
require 'librarian/puppet/extension'
require 'librarian/puppet/version'
require 'librarian/action/install'
module Librarian
module Puppet
end
end
require 'librarian/helpers'
require 'librarian/cli'
require 'librarian/puppet'
module Librarian
module Puppet
class Cli < Librarian::Cli
module Particularity
def root_module
Puppet
end
end
include Particularity
extend Particularity
source_root Pathname.new(__FILE__).dirname.join("templates")
def init
copy_file environment.specfile_name
if File.exists? ".gitignore"
gitignore = File.read('.gitignore').split("\n")
else
gitignore = []
end
gitignore << ".tmp/" unless gitignore.include? ".tmp/"
gitignore << "modules/" unless gitignore.include? "modules/"
File.open(".gitignore", 'w') do |f|
f.puts gitignore.join("\n")
end
end
desc "install", "Resolves and installs all of the dependencies you specify."
option "quiet", :type => :boolean, :default => false
option "verbose", :type => :boolean, :default => false
option "line-numbers", :type => :boolean, :default => false
option "clean", :type => :boolean, :default => false
option "strip-dot-git", :type => :boolean
option "path", :type => :string
option "destructive", :type => :boolean, :default => false
option "local", :type => :boolean, :default => false
def install
ensure!
clean! if options["clean"]
unless options["destructive"].nil?
environment.config_db.local['destructive'] = options['destructive'].to_s
end
if options.include?("strip-dot-git")
strip_dot_git_val = options["strip-dot-git"] ? "1" : nil
environment.config_db.local["install.strip-dot-git"] = strip_dot_git_val
end
if options.include?("path")
environment.config_db.local["path"] = options["path"]
end
environment.config_db.local['mode'] = options['local'] ? 'local' : nil
resolve!
install!
end
desc "package", "Cache the puppet modules in vendor/puppet/cache."
option "quiet", :type => :boolean, :default => false
option "verbose", :type => :boolean, :default => false
option "line-numbers", :type => :boolean, :default => false
option "clean", :type => :boolean, :default => false
option "strip-dot-git", :type => :boolean
option "path", :type => :string
option "destructive", :type => :boolean, :default => false
def package
environment.vendor!
install
end
def version
say "librarian-puppet v#{Librarian::Puppet::VERSION}"
end
end
end
end
require 'librarian/dsl'
require 'librarian/puppet/source'
module Librarian
module Puppet
class Dsl < Librarian::Dsl
dependency :mod
source :forge => Source::Forge
source :git => Source::Git
source :path => Source::Path
source :github_tarball => Source::GitHubTarball
end
end
end
require "librarian/environment"
require "librarian/puppet/dsl"
require "librarian/puppet/source"
require "librarian/puppet/lockfile/parser"
module Librarian
module Puppet
class Environment < Librarian::Environment
def adapter_name
"puppet"
end
def install_path
part = config_db["path"] || "modules"
project_path.join(part)
end
def vendor_path
project_path.join('vendor/puppet')
end
def vendor_cache
vendor_path.join('cache')
end
def vendor_source
vendor_path.join('source')
end
def cache_path
project_path.join(".tmp/librarian/cache")
end
def scratch_path
project_path.join(".tmp/librarian/scratch")
end
def vendor!
vendor_cache.mkpath unless vendor_cache.exist?
vendor_source.mkpath unless vendor_source.exist?
end
def vendor?
vendor_path.exist?
end
def local?
config_db['mode'] == 'local'
end
end
end
end
require 'librarian/puppet/environment'
module Librarian
module Puppet
extend self
extend Librarian
end
end
require 'librarian/manifest'
require 'librarian/dependency'
require 'librarian/manifest_set'
module Librarian
class Lockfile
class Parser
def parse(string)
string = string.dup
source_type_names_map = Hash[dsl_class.source_types.map{|t| [t[1].lock_name, t[1]]}]
source_type_names = dsl_class.source_types.map{|t| t[1].lock_name}
lines = string.split(/(\r|\n|\r\n)+/).select{|l| l =~ /\S/}
sources = []
while source_type_names.include?(lines.first)
source = {}
source_type_name = lines.shift
source[:type] = source_type_names_map[source_type_name]
options = {}
while lines.first =~ /^ {2}([\w\-\/]+):\s+(.+)$/
lines.shift
options[$1.to_sym] = $2
end
source[:options] = options
lines.shift # specs
manifests = {}
while lines.first =~ /^ {4}([\w\-\/]+) \((.*)\)$/ # This change allows forward slash
lines.shift
name = $1
manifests[name] = {:version => $2, :dependencies => {}}
while lines.first =~ /^ {6}([\w\-\/]+) \((.*)\)$/
lines.shift
manifests[name][:dependencies][$1] = $2.split(/,\s*/)
end
end
source[:manifests] = manifests
sources << source
end
manifests = compile(sources)
manifests_index = Hash[manifests.map{|m| [m.name, m]}]
raise StandardError, "Expected DEPENDENCIES topic!" unless lines.shift == "DEPENDENCIES"
dependencies = []
while lines.first =~ /^ {2}([\w\-\/]+)(?: \((.*)\))?$/ # This change allows forward slash
lines.shift
name, requirement = $1, $2.split(/,\s*/)
dependencies << Dependency.new(name, requirement, manifests_index[name].source)
end
Resolution.new(dependencies, manifests)
end
end
end
end
require 'librarian/puppet/source/path'
require 'librarian/puppet/source/git'
require 'librarian/puppet/source/forge'
require 'librarian/puppet/source/githubtarball'
require 'uri'
require 'net/http'
require 'json'
module Librarian
module Puppet
module Source
class Forge
class Repo
attr_accessor :source, :name
private :source=, :name=
def initialize(source, name)
self.source = source
self.name = name
end
def versions
data = api_call("#{name}.json")
if data.nil?
raise Error, "Unable to find module '#{name}' on #{source}"
end
data['releases'].map { |r| r['version'] }.sort.reverse
end
def dependencies(version)
data = api_call("api/v1/releases.json?module=#{name}&version=#{version}")
data[name].first['dependencies']
end
def manifests
versions.map do |version|
Manifest.new(source, name, version)
end
end
def install_version!(version, install_path)
if environment.local? && !vendored?(name, version)
raise Error, "Could not find a local copy of #{name} at #{version}."
end
if environment.vendor?
vendor_cache(name, version) unless vendored?(name, version)
end
cache_version_unpacked! version
if install_path.exist?
install_path.rmtree
end
unpacked_path = version_unpacked_cache_path(version).join(name.split('/').last)
unless unpacked_path.exist?
raise Error, "#{unpacked_path} does not exist, something went wrong. Try removing it manually"
else
FileUtils.cp_r(unpacked_path, install_path)
end
end
def environment
source.environment
end
def cache_path
@cache_path ||= source.cache_path.join(name)
end
def version_unpacked_cache_path(version)
cache_path.join('version').join(hexdigest(version.to_s))
end
def hexdigest(value)
Digest::MD5.hexdigest(value)
end
def cache_version_unpacked!(version)
path = version_unpacked_cache_path(version)
return if path.directory?
# The puppet module command is only available from puppet versions >= 2.7.13
#
# Specifying the version in the gemspec would force people to upgrade puppet while it's still usable for git
# So we do some more clever checking
#
# Executing older versions or via puppet-module tool gives an exit status = 0 .
#
check_puppet_module_options
path.mkpath
target = vendored?(name, version) ? vendored_path(name, version) : name
command = "puppet module install --target-dir '#{path}' --modulepath '#{path}' --ignore-dependencies '#{target}'"
output = `#{command}`
# Check for bad exit code
unless $? == 0
# Rollback the directory if the puppet module had an error
path.unlink
raise Error, "Error executing puppet module install:\n#{command}\nError:\n#{output}"
end
end
def check_puppet_module_options
min_version = Gem::Version.create('2.7.13')
puppet_version = Gem::Version.create(`puppet --version`.strip)
if puppet_version < min_version
raise Error, "To get modules from the forge, we use the puppet faces module command. Your current version does not support the options #{options.join(',')} . For this you need at least puppet version 2.7.13 and you have #{puppet_version}"
end
end
def vendored?(name, version)
vendored_path(name, version).exist?
end
def vendored_path(name, version)
environment.vendor_cache.join("#{name.sub("/", "-")}-#{version}.tar.gz")
end
def vendor_cache(name, version)
File.open(vendored_path(name, version).to_s, 'w') do |f|
download(name, version) do |data|
f << data
end
end
end
def download(name, version, &block)
data = api_call("api/v1/releases.json?module=#{name}&version=#{version}")
info = data[name].detect {|h| h['version'] == version.to_s }
stream(info['file'], &block)
end
def stream(file, &block)
Net::HTTP.get_response(URI.parse("#{source}#{file}")) do |res|
res.code
res.read_body(&block)
end
end
private
def api_call(path)
base_url = source.to_s
resp = Net::HTTP.get_response(URI.parse("#{base_url}/#{path}"))
if resp.code.to_i != 200
nil
else
data = resp.body
JSON.parse(data)
end
end
end
class << self
LOCK_NAME = 'FORGE'
def lock_name
LOCK_NAME
end
def from_lock_options(environment, options)
new(environment, options[:remote], options.reject { |k, v| k == :remote })
end
def from_spec_args(environment, uri, options)
recognised_options = []
unrecognised_options = options.keys - recognised_options
unless unrecognised_options.empty?
raise Error, "unrecognised options: #{unrecognised_options.join(", ")}"
end
new(environment, uri, options)
end
end
attr_accessor :environment
private :environment=
attr_reader :uri
def initialize(environment, uri, options = {})
self.environment = environment
@uri = uri
@cache_path = nil
end
def to_s
uri
end
def ==(other)
other &&
self.class == other.class &&
self.uri == other.uri
end
def to_spec_args
[uri, {}]
end
def to_lock_options
{:remote => uri}
end
def pinned?
false
end
def unpin!
end
def install!(manifest)
manifest.source == self or raise ArgumentError
name = manifest.name
version = manifest.version
install_path = install_path(name)
repo = repo(name)
repo.install_version! version, install_path