Commit 759dbf05 authored by Dmitry Smirnov's avatar Dmitry Smirnov

Initial commit

parent 08249c8b
# Configures appc tests at Travis CI (https://travis-ci.org).
language: go
go:
- 1.3
- 1.4
install:
- go get golang.org/x/tools/cmd/cover
- go get golang.org/x/tools/cmd/vet
script:
- ./test
### v0.6.1
Minor release of the spec; the most important change is adjusting the type for
`annotations` to ACIdentifier (#442). This restores the ability for annotation
fields to be namespaced with DNS names.
Other changes:
- Added new maintainer (Ken Robertson)
- Created CHANGELOG.md to track changes instead of using git tags
- Fixed build scripts for FreeBSD (#433)
- Fixed acirenderer to work properly with empty images with just a rootfs
directory (#428)
- Added `arm6vl` as valid arch for Linux (#440)
### v0.6.0
This is an important milestone release of the spec. Critically, there are two
backwards-incompatible schema changes from previous releases:
- Dependency references' `app` field has been renamed to the more accurate and
unambiguous `imageName`: #397
- `ACName` has been redefined (with a stricter specification) to be suitable
for relative names within a pod, and a new type, `ACIdentifier`, has been
introduced for image, label and isolator names: #398
This release also sees the sections of the specification - image format,
discovery process, pods and executor definitions - being split into distinct
files. This is a first step towards clarifying the componentised nature of the
spec and the value in implementing individual sections.
Other changes of note in this release:
- Dependency references gained an optional `size` field. If this field is
specified, executors should validate that the size of resolved dependency
images is correct before attempting to retrieve them: #422
- The RFC3339 timestamp type definition was tweaked to clarify that it must
include a T rather than a space: #410
- The spec now prescribes that ACEs must set the `container` environment
variable to some value to indicate to applications that they are being run
inside a container: #302
- Added support for 64-bit big-endian ARM architectures: #414
- Clarifications to the ports definition in the schema: #405
- Fixed a bug in the discovery code where it was mutating supplied objects:
#412
### v0.5.2
This release features a considerable number of changes over the previous
(0.5.1) release. However, the vast majority are syntactical improvements to
clarity and wording in the text of the specification and do not change the
semantic behaviour in any significant way; hence, this should remain a
backwards-compatible release. As well as the changes to the spec itself, there
are various improvements to the schema/tooling code, including new
functionality in `actool`.
Some of the more notable changes since v0.5.1:
- `linux/aarch64`, `linux/armv7l` and `linux/armv7b` added as recognised
os/arch combinations for images
- added contribution/governance policy and new maintainers
- added `cat-manifest` and `patch-manifest` subcommands to actool to manipulate
existing ACIs
- added guidance around using authorization token (supplied in AC_METADATA_URL)
for identifying pods to the metadata service
- reduced the set of required environment variables that executors must provide
- fixed consistency between schema code and spec for capabilities
- all TODOs removed from spec text and moved to GitHub issues
- several optimizations and fixes in acirenderer package
### v0.5.1
This is primarily a bugfix release to catch 2a342dac which resolves an issue
preventing PodManifests from being successfully serialized.
Other changes:
- Update validator to latest Pod spec changes
- Added /dev/pts and /dev/ptmx to Linux requirements
- Added a script to bump release versions
- Moved to using types.ACName in discovery code instead of strings
### v0.5.0
The major change in this release is the introduction of _pods_,
via #207 and #248. Pods are a refinement (and replacement) of the
previous ContainerRuntimeManifest concept that define the minimum
deployable, executable unit in the spec, as a grouping of one or
more applications. The move to pods implicitly resolves various
issues with the CRM in previous versions of the spec (e.g #83, #84)
Other fixes and changes in this release:
- fix static builds of the tooling on go1.4
- add ability to use proxy from environment for discovery
- fix inheritance of readOnly flag
- properly validate layouts with relative paths
- properly tar named pipes and ignore sockets
- add /dev/shm to required Linux environment
### v0.4.1
This is a minor bugfix release to fix marshalling of isolators.
### v0.4.0
Major changes and additions since v0.3.0:
- Reworked isolators to objects instead of strings and clarify limits vs
reservations for resource isolators
- Introduced OS-specific requirements (e.g. device files on Linux)
- Moved much of the wording in the spec towards RFC2119 wording ("MUST", "MAY",
"SHOULD", etc) to be more explicit about which parts are
required/optional/recommended
- Greater explicitness around signing and encryption requirements
- Moved towards `.asc` filename extension for signatures
- Added MAINTAINERS
- Added an implementation guide
- Tighter restrictions on layout of ACI tars
- Numerous enhancements to discovery code
- Improved test coverage for various schema types
### v0.3.0
### v0.2.0
### v0.1.1
This marks the first versioned release of the app container spec.
# How to Contribute
appc is [Apache 2.0 licensed](LICENSE) and accepts contributions via GitHub
pull requests. This document outlines some of the conventions on development
workflow, commit message formatting, contact points and other resources to make
it easier to get your contribution accepted.
For more information on the policy for accepting contributions, see [POLICY](POLICY.md)
# Certificate of Origin
By contributing to this project you agree to the Developer Certificate of
Origin (DCO). This document was created by the Linux Kernel community and is a
simple statement that you, as a contributor, have the legal right to make the
contribution. See the [DCO](DCO) file for details.
# Email and Chat
The project uses the the appc-dev email list and #appc on Freenode for chat:
- Email: [appc-dev](https://groups.google.com/forum/#!forum/appc-dev)
- IRC: #[appc](irc://irc.freenode.org:6667/#appc) IRC channel on freenode.org
Please avoid emailing maintainers found in the MAINTAINERS file directly. They
are very busy and read the mailing lists.
## Getting Started
- Fork the repository on GitHub
- Read the [README](README.md) for build and test instructions
- Play with the project, submit bugs, submit patches!
## Contribution Flow
This is a rough outline of what a contributor's workflow looks like:
- Create a topic branch from where you want to base your work (usually master).
- Make commits of logical units.
- Make sure your commit messages are in the proper format (see below).
- Push your changes to a topic branch in your fork of the repository.
- Make sure the tests pass, and add any new tests as appropriate.
- Submit a pull request to the original repository.
Thanks for your contributions!
### Format of the Commit Message
We follow a rough convention for commit messages that is designed to answer two
questions: what changed and why. The subject line should feature the what and
the body of the commit should describe the why.
```
scripts: add the test-cluster command
this uses tmux to setup a test cluster that you can easily kill and
start for debugging.
Fixes #38
```
The format can be described more formally as follows:
```
<subsystem>: <what changed>
<BLANK LINE>
<why this change was made>
<BLANK LINE>
<footer>
```
The first line is the subject and should be no longer than 70 characters, the
second line is always blank, and other lines should be wrapped at 80 characters.
This allows the message to be easier to read on GitHub as well as in various
git tools.
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
# Implementation Guide
This document contains guidelines, tips and suggested best practices for implementers of the App Container specification.
It should not be considered prescriptive, but is instead a general resource to guide usability and optimization of implementations.
This is a living document; those implementing the spec are encouraged to submit feedback and improvements.
## Storing and working with App Container Images (ACIs)
The canonical ACI format is a simple tarball, which conveniently encapsulates the filesystem and metadata of an image in a single file.
This also provides a simple, unique and canonical way to reference an image, using the cryptographic hash of this artefact.
Very simple implementations of the spec might use this tarball format as their primary format for transport and local storage, extracting the tarball as necessary every time an image needs to be accessed.
However, when looking to minimize transport and startup times for running applications, this is inefficient; a variety of optimizations are possible, both for servers hosting images ("image repositories") and for local stores (for example any caches internal to Application Container Executors)
**Note that since images retaining a cryptographic identity is a key part of the specification, it should always be possible to recreate the canonical format of an image (and hence its cryptographic Image ID), regardless of what optimizations are leveraged.**
### Indexing ImageManifests
A simple example would be indexing the ImageManifest outside of the ACI.
For example, an image repository might use an RDBMS and store the information contained in ImageManifests in a normalized fashion.
This would facilitate efficient lookups of images based on certain attributes - for example, querying based on a combination of name and labels.
### Storing exploded ACIs
A similar optimization relates to the storage on disk of the root filesystem.
Instead of hosting the tarball form of an image on disk, and expanding it every time it is referenced, stores could simply store the root filesystem as an expanded tree on disk.
When taking this approach, care must be taken to preserve all the appropriate metadata that the tarball format encapsulates.
Storing exploded ACIs not only prevents the need to regularly tar and untar images (and hence improves performance for launching containers), but it also allows for further space optimizations: for example, if files are stored in a content-addressable-store, then they can be de-duplicated between ACIs.
(Future versions of the spec will likely define a streaming image format which would facilitate similar behaviour.)
### De-duplicating images at runtime
The specification stipulates that each execution of an application must start from a clean copy of its image.
The most basic implementation of this would involve a complete copy (e.g. `cp -arv`) of the image's root filesystem into a new directory for the application to use.
To make more efficient use of disk space, and improve performance, implementations should consider use of technologies like [device mapper copy-on-write block devices] [dm], [OverlayFS] [overlay], or [ZFS] [zfs].
[dm]: https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt
[overlay]: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/overlayfs.txt
[zfs]: http://en.wikipedia.org/wiki/ZFS
## Transporting ACIs
### Image Discovery
While the specification prescribes that the Meta Discovery process occurs over HTTP(S), it is intentionally agnostic with respect to the transport over which the discovered resource can be retrieved.
In the simplest cases (and as for example in Simple Discovery), the ACI payload can simply be retrieved over HTTPS itself.
However, in more advanced implementations - particulary in highly distributed environments - alternative protocols like [HDFS](hdfs) or [BitTorrent](bittorrent) could be used instead.
[hdfs]: http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
[bittorrent]: http://en.wikipedia.org/wiki/BitTorrent
This diff is collapsed.
Brandon Philips <brandon.philips@coreos.com> (@philips)
Charles Aylward <charles.aylward@gmail.com> (@cdaylward)
Jonathan Boulle <jonathan.boulle@coreos.com> (@jonboulle)
Ken Robertson <ken@apcera.com> (@krobertson)
Tim Hockin <thockin@google.com> (@thockin)
Vincent Batts <vbatts@redhat.com> (@vbatts)
# App Container Specification Governance and Contribution Policy
This document defines the current policies around governance and contributions for the App Container Specification (appc) project.
This is written from the perspective that there is a group of people who cooperatively support and manage the project (the _maintainers_, hereafter "we").
We will move towards an open governance model where multiple parties have commit access, roll-back rights, and can provide explicit support for features.
## Contributions
We use the following rules for accepting contributions to the specification repository, including the text of the specification itself and the associated schema and tooling.
In contrast to maintainers, _contributors_ are not actively managing the direction of the project, but contributing individual changes.
- We require all contributors to agree to the [DCO](https://github.com/appc/spec/blob/master/DCO)
- We accept well-written, clearly explained changes to the specification itself.
- We accept well-written, well-tested cleanup and refactoring changes of the schema code.
- We accept well-written, well-tested bug fixes to the schema code.
- We accept well-written, well-tested feature contributions to the schema tooling if a maintainer assumes support responsibilities, i.e., readily answers support questions and works on bugs. This includes feature contributions from external contributors. If there is no maintainer to support a feature, then we will deprecate and subsequently delete the feature - we will give three months' notice in such cases.
- We will not accept untested changes, except in very rare cases.
- We require a pre-commit code review from a maintainer for all changes. For changes submitted by maintainers, a review is required from at least one other maintainer.
## Major Schema Changes
Any changes that would require a new version of the schema/specification to be released must be approved by at least two of the current maintainers. A new version must be released any time a change is made that would break compatibility with the existing schema.
## Maintainers
Maintainers are the only contributors with merge privileges on the [specification repository](https://github.com/appc/spec). The group of maintainers is self-managing - new maintainers are added by a simple majority of votes from existing maintainers on the mailing list with zero no-votes within four business days. Maintainers may voluntarily step down with no voting required. Removal of existing maintainers is done through a supermajority of two-thirds of the votes of all maintainers. The voting period is 10 business days.
We expect that new contributors will submit a number of patches before they become maintainers.
The current set of maintainers is always recorded in the [MAINTAINERS](https://github.com/appc/spec/blob/master/MAINTAINERS) file.
The mailing list for the maintainers team is [appc-dev@googlegroups.com](https://groups.google.com/forum/#!forum/appc-dev)
## Governance
Changes to the rules in this document follow the same process as removing maintainers: a supermajority of two-thirds of the votes of all maintainers within a period of 10 business days.
## Credits
This document was inspired by and modelled on the [initial governance policy for Google's Bazel project](https://github.com/google/bazel/blob/efbcf00220a95c5ea1dfa7e3a5bff8311b52727d/site/governance.md)
This diff is collapsed.
# App Container Specification
_For version information, see [VERSION](VERSION)_
## Overview
"App Container" (appc) is a specification describing how applications can be packaged, distributed, and executed in a portable and self-contained way.
The specification defines an **image format**, an **image discovery mechanism**, a **deployable grouping**, and an **execution environment**.
The core goals of the specification include:
* Designing for fast downloads and starts of App Containers
* Ensuring images are cryptographically verifiable and highly cacheable
* Designing for composability and independent implementations
* Using common technologies for cryptography, archiving, compression and transport
* Using the DNS namespace to name and discover images
## Sections
The specification consists of several key sections; the goal is that each can be implemented independently, but are composable with one another.
1. The **[App Container Image](spec/aci.md)** defines: how files are assembled together into a single image, verified on download and placed onto disk to be run.
2. **[App Container Image Discovery](spec/discovery.md)** defines: how to take a name like `example.com/reduce-worker` and translate that into a downloadable image.
3. The **[App Container Pod](spec/pods.md)** (or "Pod") defines: how one or more App Container Images are grouped into a deployable, executable unit.
4. The **[App Container Executor](spec/ace.md)** defines: how pods are executed and the environment they are run inside (including, for example, filesystem layout, resource constraints, and networking).
* The [Metadata Service](spec/ace.md#app-container-metadata-service) defines how apps within pods can introspect and get a cryptographically verifiable identity from the execution environment.
## Example Use Case
Here's an example use case demonstrating how the different sections of the specification could be utilized together.
A user wants to launch an "App Container" running three processes.
The three processes the user wants to run are the apps named `example.com/reduce-worker-register`, `example.com/reduce-worker`, and `example.com/reduce-backup`.
First, an executor will make an HTTPS request to example.com and, on inspecting the `<meta>` tags in the returned page, determines that the images can be found at:
https://storage-mirror.example.com/reduce-worker.aci
https://storage-mirror.example.com/worker-backup.aci
https://storage-mirror.example.com/reduce-worker-register.aci
The executor downloads these three images and puts them into its local on-disk cache.
Then the executor extracts three fresh copies of the images to create instances of the "on-disk app format" and reads the three image manifests to figure out what binaries will need to be executed.
Based on user input, the executor now sets up the necessary cgroups, network interfaces, etc. and runs the `pre-start` event handlers for each app.
Next, it forks the `reduce-worker`, `worker-backup`, and `register` processes in their shared namespaces, chrooted into their respective root filesystems.
At some point, the App Container gets some notification that it needs to stop (for example, upon host shutdown).
The executor sends `SIGTERM` to the processes.
After they have exited, the executor runs the `post-stop` event handlers for each app.
#!/usr/bin/env bash
#
# Builds an ACI containing a go implementation of an ACE validator
#
set -eu
PREFIX="ace"
if ! [[ $0 =~ "${PREFIX}/build_aci" ]]; then
echo "invoke from repository root" 1>&2
exit 255
fi
if ! [[ -f "bin/ace-validator" ]]; then
./build
fi
for typ in main sidekick; do
layoutdir="bin/ace-${typ}-layout"
mkdir -p ${layoutdir}/rootfs/opt/acvalidator
cp bin/ace-validator ${layoutdir}/rootfs/
cp ${PREFIX}/image_manifest_${typ}.json ${layoutdir}/manifest
# now build the tarball, and sign it
pushd ${layoutdir} >/dev/null
# Set a consistent timestamp so we get a consistent hash
# TODO(jonboulle): make this cleaner..
for path in rootfs rootfs/ace-validator; do
touch -a -m -d 1970-01-01T00:00:00Z ${path}
done
../actool build --overwrite ./ ../ace-validator-${typ}.aci
# TODO(jonboulle): create uncompressed instead, then gzip?
HASH=sha512-$(gzip -d -f ../ace-validator-${typ}.aci -c | openssl dgst -sha512 -hex -r | awk '{print $1}')
if [ -z "$NO_SIGNATURE" ] ; then
gpg --cipher-algo AES256 --output ace-validator-${typ}.aci.asc --detach-sig ../ace-validator-${typ}.aci
mv ace-validator-${typ}.aci.asc ../
fi
popd >/dev/null
echo "Wrote ${typ} layout to ${layoutdir}"
echo "Wrote unsigned ${typ} ACI bin/ace-validator-${typ}.aci"
ln -s ${PWD}/bin/ace-validator-${typ}.aci bin/${HASH}
echo "Wrote ${typ} layout hash bin/${HASH}"
if [ -f "bin/ace-validator-${typ}.aci.asc" ]; then
echo "Wrote ${typ} ACI signature bin/ace-validator-${typ}.aci.asc"
fi
done
// Copyright 2015 The appc Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package ace contains a tool intended to be run within an _Application
// Container Executor_ to validate that the ACE has set up the container
// environment correctly. This tool can be built into an ACI image ready for
// running on an executor by using the `build_aci` script.
package main
{
"acVersion": "0.6.1",
"acKind": "ImageManifest",
"name": "coreos.com/ace-validator-main",
"labels": [
{ "name": "version", "value": "0.6.1" },
{ "name": "os", "value": "linux" },
{ "name": "arch", "value": "amd64" }
],
"app": {
"exec": [
"/ace-validator", "main"
],
"eventHandlers": [
{
"name": "pre-start",
"exec": [
"/ace-validator", "prestart"
]
},
{
"name": "post-stop",
"exec": [
"/ace-validator", "poststop"
]
}
],
"user": "0",
"group": "0",
"workingDirectory": "/opt/acvalidator",
"environment": [
{
"name": "IN_ACE_VALIDATOR",
"value": "correct"
}
],
"mountPoints": [
{
"name": "database",
"path": "/db",
"readOnly": false
}
],
"ports": [
{
"name": "www",
"protocol": "tcp",
"port": 80
}
],
"isolators": [
{
"name": "resource/memory",
"value": {"limit": "1G"}
}
]
},
"annotations": [
{
"name": "created",
"value": "2014-10-27T19:32:27.67021798Z"
},
{
"name": "authors",
"value": "Carly Container <carly@example.com>, Nat Network <nat@example.com>"
},
{
"name": "homepage",
"value": "https://github.com/appc/spec"
},
{
"name": "documentation",
"value": "https://github.com/appc/spec/blob/master/README.md"
},
{
"name": "lorem",
"value": "ipsum"
}
]
}
{
"acVersion": "0.6.1",
"acKind": "ImageManifest",
"name": "coreos.com/ace-validator-sidekick",
"labels": [
{ "name": "version", "value": "0.6.1" },
{ "name": "os", "value": "linux" },
{ "name": "arch", "value": "amd64" }
],
"app": {
"exec": [
"/ace-validator", "sidekick"
],
"user": "0",
"group": "0",
"mountPoints": [
{
"name": "database",
"path": "/db",
"readOnly": false
}
]
}
}
This diff is collapsed.
// Copyright 2015 The appc Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package aci
import (
"archive/tar"
"io"
"os"
"path/filepath"
"github.com/appc/spec/pkg/tarheader"
)
// BuildWalker creates a filepath.WalkFunc that walks over the given root
// (which should represent an ACI layout on disk) and adds the files in the
// rootfs/ subdirectory to the given ArchiveWriter
func BuildWalker(root string, aw ArchiveWriter) filepath.WalkFunc {
// cache of inode -> filepath, used to leverage hard links in the archive
inos := map[uint64]string{}
return func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
relpath, err := filepath.Rel(root, path)
if err != nil {
return err
}
if relpath == "." {
return nil
}
if relpath == ManifestFile {
// ignore; this will be written by the archive writer
// TODO(jonboulle): does this make sense? maybe just remove from archivewriter?
return nil
}
link := ""
var r io.Reader
switch info.Mode() & os.ModeType {
case os.ModeSocket:
return nil
case os.ModeNamedPipe:
case os.ModeCharDevice:
case os.ModeDevice:
case os.ModeDir:
case os.ModeSymlink:
target, err := os.Readlink(path)
if err != nil {
return err
}
link = target
default:
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
r = file
}
hdr, err := tar.FileInfoHeader(info, link)
if err != nil {
panic(err)
}
// Because os.FileInfo's Name method returns only the base
// name of the file it describes, it may be necessary to
// modify the Name field of the returned header to provide the
// full path name of the file.
hdr.Name = relpath
tarheader.Populate(hdr, info, inos)
// If the file is a hard link to a file we've already seen, we
// don't need the contents
if hdr.Typeflag == tar.TypeLink {
hdr.Size = 0
r = nil
}
if err := aw.AddFile(hdr, r); err != nil {
return err
}
return nil
}
}
// Copyright 2015 The appc Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package aci contains various functions for working with App Container Images.
package aci
// Copyright 2015 The appc Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package aci