Commit 9189840b authored by SVN-Git Migration's avatar SVN-Git Migration

Imported Upstream version 0.0.3

parents
dist
MANIFEST
test.xml
build
.testrepository
This diff is collapsed.
Copyright (c) Robert Collins and Testrepository contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the name of Robert Collins nor the names of Subunit contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY ROBERT COLLINS AND SUBUNIT CONTRIBUTORS ``AS IS''
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
Testrepository is licensed under two licenses, the Apache License, Version 2.0
or the 3-clause BSD License. You may use this project under either of these
licenses - choose the one that works best for you.
We require contributions to be licensed under both licenses. The primary
difference between them is that the Apache license takes care of potential
issues with Patents and other intellectual property concerns that some users
or contributors may find important.
Generally every source file in Testrepository needs a license grant under both
these licenses. As the code is shipped as a single unit, a brief form is used:
----
Copyright (c) [yyyy][,yyyy]* [name or 'Testrepository Contributors']
Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
license at the users choice. A copy of both licenses are available in the
project source as Apache-2.0 and BSD. You may not use this file except in
compliance with one of these two licences.
Unless required by applicable law or agreed to in writing, software
distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
license you chose for the specific language governing permissions and
limitations under that license.
----
A concordance of contributors is maintained here to provide an easy reference
for distributions such as Debian that wish to list all the copyright holders
in their metadata:
* Robert Collins <robertc@robertcollins.net>, 2009
Code that has been incorporated into Testrepository from other projects will
naturally be under its own license, and will retain that license.
A known list of such code is maintained here:
* No entries.
Installing Test Repository
++++++++++++++++++++++++++
Run time dependencies
~~~~~~~~~~~~~~~~~~~~~
* Python2.4 or newer.
* subunit
Test dependencies
~~~~~~~~~~~~~~~~~
* testtools (the python-testtools package, or
http://pypi.python.org/pypi/testtools/).
* testresources (https://launchpad.net/testresources, or
http://pypi.python.org/pypi/testresources/).
* testscenarios (https://launchpad.net/testscenarios).
Installing
~~~~~~~~~~
* ./setup.py install
include .bzrignore
include Apache-2.0
include BSD
include COPYING
include INSTALL.txt
include MANIFEST.in
include Makefile
include NEWS
include README.txt
include doc/*.txt
include testrepository/tests/*.py
recursive-include testrepository/tests *.py
#
# Copyright (c) 2009 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
all: README.txt check
.testrepository:
./testr init
check: .testrepository
./testr run
check-xml:
python -m subunit.run testrepository.tests.test_suite | subunit2junitxml -o test.xml -f | subunit2pyunit
release:
./setup.py sdist upload --sign
README.txt: testrepository/commands/quickstart.py
./testr quickstart > $@
.PHONY: check check-xml release all
############################
testrepository release notes
############################
NEXT (In development)
+++++++++++++++++++++
IMPROVEMENTS
~~~~~~~~~~~~
0.0.3
-----
* ``failing`` now correctly calls ``repository.get_failing`` and will this
track all seen failures rather than just the latest observed failures.
* New argument type ``StringArgument`` for use when a supplied argument is just
a string, rather than a typed argument.
* New subcommand 'failing' added.
* New subcommand ``run`` added which reads a .testr.conf file to figure out how
to run tests with subunit output. It then runs them and pipes into testr
load. This allows simpler integration and permits a programming interface so
that tools like Tribunal/Eclipe etc can refresh tests in a testrepository.
``run`` also passes arguments and options down to the child process. ``run``
can also supply test ids on the command, for test runners that want that.
* The command 'last' will no longer error on a new repository.
Metadata-Version: 1.0
Name: testrepository
Version: 0.0.3
Summary: A repository of test results.
Home-page: https://launchpad.net/testrepository
Author: Robert Collins
Author-email: robertc@robertcollins.net
License: UNKNOWN
Description: Test Repository
+++++++++++++++
Overview
~~~~~~~~
This project provides a database of test results which can be used as part of
developer workflow to ensure/check things like:
* No commits without having had a test failure, test fixed cycle.
* No commits without new tests being added.
* What tests have failed since the last commit (to run just a subset).
* What tests are currently failing and need work.
Test results are inserted using subunit (and thus anything that can output
subunit or be converted into a subunit stream can be accepted).
A mailing list for discussion, usage and development is at
https://launchpad.net/~testrepository-dev - all are welcome to join. Some folk
hang out on #testrepository on irc.freenode.net.
CI for the project is at http://build.robertcollins.net/job/testrepository-default/.
Licensing
~~~~~~~~~
Test Repository is under BSD / Apache 2.0 licences. See the file COPYING in the source for details.
Quick Start
~~~~~~~~~~~
Create a repository::
$ testr init
Load a test run into the repository::
$ testr load < testrun
Query the repository::
$ testr stats
$ testr last
$ testr failing
Delete a repository::
$ testr delete
Documentation
~~~~~~~~~~~~~
More detailed documentation including design and implementation details, a
user manual, and guidelines for development of Test Repository itself can be
found in the doc/ directory.
Platform: UNKNOWN
Test Repository
+++++++++++++++
Overview
~~~~~~~~
This project provides a database of test results which can be used as part of
developer workflow to ensure/check things like:
* No commits without having had a test failure, test fixed cycle.
* No commits without new tests being added.
* What tests have failed since the last commit (to run just a subset).
* What tests are currently failing and need work.
Test results are inserted using subunit (and thus anything that can output
subunit or be converted into a subunit stream can be accepted).
A mailing list for discussion, usage and development is at
https://launchpad.net/~testrepository-dev - all are welcome to join. Some folk
hang out on #testrepository on irc.freenode.net.
CI for the project is at http://build.robertcollins.net/job/testrepository-default/.
Licensing
~~~~~~~~~
Test Repository is under BSD / Apache 2.0 licences. See the file COPYING in the source for details.
Quick Start
~~~~~~~~~~~
Create a repository::
$ testr init
Load a test run into the repository::
$ testr load < testrun
Query the repository::
$ testr stats
$ testr last
$ testr failing
Delete a repository::
$ testr delete
Documentation
~~~~~~~~~~~~~
More detailed documentation including design and implementation details, a
user manual, and guidelines for development of Test Repository itself can be
found in the doc/ directory.
Design / Architecture of Test Repository
++++++++++++++++++++++++++++++++++++++++
Values
~~~~~~
Code reuse.
Focus on the project.
Do one thing well.
Goals
~~~~~
Achieve a Clean UI, responsive UI, small-tools approach. Simulataneously have
a small clean code base which is easily approachable.
Data model/storage
~~~~~~~~~~~~~~~~~~
testrepository stores subunit streams as subunit streams in .testrespository
with simple additional metadata - a format marker to allow this to be changed
in the future, and a next-stream counter counting the id to give the next
stream.
Code layout
~~~~~~~~~~~
One conceptual thing per module, packages for anything where multiple types
are expected (e.g. testrepository.commands, testrespository.ui).
Generic driver code should not trigger lots of imports: code dependencies
should be loaded when needed. For example, argument validation uses argument
types that each command can import, so the core code doesn't need to know about
all types.
The tests for the code in testrepository.foo.bar is in
testrepository.tests.foo.test_bar. Interface tests for testrepository.foo is
in testrepository.tests.test_foo.
Development guidelines for Test Repository
++++++++++++++++++++++++++++++++++++++++++
Coding style
~~~~~~~~~~~~
PEP-8 please. We don't enforce a particular style, but being reasonably
consistent aids readability.
Copyrights and licensing
~~~~~~~~~~~~~~~~~~~~~~~~
Code committed to Test Repository must be licensed under the BSD + Apache-2.0
licences that Test Repository offers its users. Copyright assignment is not
required. Please see COPYING for details about how to make a license grant in
a given source file. Lastly, all copyright holders need to add their name
to the master list in COPYING the first time they make a change in a given
calendar year.
Testing and QA
~~~~~~~~~~~~~~
For Test repository please add tests where possible. There is no requirement
for one test per change (because somethings are much harder to automatically
test than the benfit from such tests). Fast tests are preferred to slow tests,
and understandable tests to fast tests.
http://build.robertcollins.net/ has a job testing every commit made to trunk
of Test Repository, and there is no automated test-before-merge process at the
moment. The quid pro quo for committers is that they should check that the
automated job found their change acceptable after merging it, and either roll
it back or fix it immediately. A broken trunk is not acceptable!
See DESIGN.txt for information about code layout which will help you find
where to add tests (and indeed where to change things).
Running the tests
-----------------
Generally just ``make`` is all that is needed to run all the tests. However
if dropping into pdb, it is currently more convenient to use
``python -m testtools.run testrepository.tests.test_suite``.
Test Repository users manual
++++++++++++++++++++++++++++
Overview
~~~~~~~~
Test repository is a small application for tracking test results. Any test run
that can be represented as a subunit stream can be inserted into a repository.
Typical workflow is to have a repository into which test runs are inserted, and
then to query the repository to find out about issues that need addressing. For
instance, using the sample subunit stream included with Test repository::
# Create a store to manage test results in.
$ testr init
# add a test result (shows failures)
$ testr load < doc/example-failing-subunit-stream
# see the tracked failing tests again
$ testr failing
# fix things
$ testr load < doc/example-passing-subunit-stream
# Now there are no tracked failing tests
$ testr failing
Most commands in testr have comprehensive online help, and the commands::
$ testr help
$ testr commands
Will be useful to explore the system.
Test Repository can be taught how to run your tests by setting up a .testr.conf
file in your cwd. A file like::
[DEFAULT]
test_command=foo $IDOPTION
test_id_option=--bar $IDFILE
will cause 'testr run' to run 'foo | testr load', and 'testr run --failing' to
run 'foo --bar failing.list | testr load'. failing.list will be a newline
separated list of the test ids that your test runner outputs. Arguments passed
to run are passed through to your test runner command line. To pass options
through to your test running, use a ``--`` before your options.
For instance, ``testr run foo -- bar --no-plugins`` would run
``foo foo bar --no-plugins | testr load`` using the above config example. The
command help for ``testr run`` describes the availableoptions for .testr.conf.
Having setup a .testr.conf, a common workflow then becomes::
# Fix currently broken tests - repeat until there are no failures.
$ testr run --failing
# Do a full run to find anything thrown out during the reduction process.
$ testr run
# And either commit or loop around this again depending on whether errors
# were found.
#!/usr/bin/env python
#
# Copyright (c) 2009, 2010 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
from distutils.core import setup
import os
import testrepository
version = '.'.join(str(component) for component in testrepository.__version__[0:3])
phase = testrepository.__version__[3]
if phase != 'final':
import bzrlib.workingtree
t = bzrlib.workingtree.WorkingTree.open_containing(__file__)[0]
if phase == 'alpha':
# No idea what the next version will be
version = 'next-%s' % t.branch.revno()
else:
# Preserve the version number but give it a revno prefix
version = version + '~%s' % t.branch.revno()
description = file(os.path.join(os.path.dirname(__file__), 'README.txt'), 'rb').read()
setup(name='testrepository',
author='Robert Collins',
author_email='robertc@robertcollins.net',
url='https://launchpad.net/testrepository',
description='A repository of test results.',
long_description=description,
scripts=['testr'],
version=version,
packages=['testrepository',
'testrepository.arguments',
'testrepository.commands',
'testrepository.repository',
'testrepository.tests',
'testrepository.tests.arguments',
'testrepository.tests.commands',
'testrepository.tests.repository',
'testrepository.tests.ui',
'testrepository.ui',
])
#!/usr/bin/env python
#
# Copyright (c) 2009 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
"""The CLI entry point to testrepository.
No program logic is in this script - see testrepository.commands.run_argv.
"""
import sys
from testrepository.commands import run_argv
if __name__ == "__main__":
sys.exit(run_argv(sys.argv, sys.stdin, sys.stdout, sys.stderr))
#
# Copyright (c) 2009 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
"""The testrepository library.
This library is divided into some broad areas.
The commands package contains the main user entry points into the application.
The ui package contains various user interfaces.
The repository package contains the core storage code.
The tests package contains tests and test specific support code.
"""
# same format as sys.version_info: "A tuple containing the five components of
# the version number: major, minor, micro, releaselevel, and serial. All
# values except releaselevel are integers; the release level is 'alpha',
# 'beta', 'candidate', or 'final'. The version_info value corresponding to the
# Python version 2.0 is (2, 0, 0, 'final', 0)." Additionally we use a
# releaselevel of 'dev' for unreleased under-development code.
#
# If the releaselevel is 'alpha' then the major/minor/micro components are not
# established at this point, and setup.py will use a version of next-$(revno).
# If the releaselevel is 'final', then the tarball will be major.minor.micro.
# Otherwise it is major.minor.micro~$(revno).
__version__ = (0, 0, 3, 'final', 0)
#
# Copyright (c) 2010 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
"""'Arguments' for testr.
This is a small typed arguments concept - which is perhaps obsoleted by
argparse in Python 2.7, but for testrepository is an extension used with
optparse.
The code in this module contains the AbstractArgument base class. Individual
argument types are present in e.g. testrepository.arguments.string.
See testrepository.commands.Command for usage of Arguments.
Plugins and extensions wanting to add argument types should either define them
internally or install into testrepository.arguments as somename (perhaps by
extending the testrepository.arguments __path__ to include a directory
containing their argument types - no __init__ is needed in that directory.)
"""
class AbstractArgument(object):
"""A argument that a command may need.
Arguments can be converted into a summary for describing the UI to users,
and provide validator/parsers for the arguments.
:ivar: The name of the argument. This is used for retrieving the argument
from UI objects, and for generating the summary.
"""
def __init__(self, name, min=1, max=1):
"""Create an AbstractArgument.
While conceptually a separate SequenceArgument could be used, all
arguments support sequencing to avoid unnecessary boilerplate in user
code.
:param name: The name for the argument.
:param min: The minimum number of occurences permitted.
:param max: The maximum number of occurences permitted. None for
unlimited.
"""
self.name = name
self.minimum_count = min
self.maximum_count = max
def summary(self):
"""Get a regex-like summary of this argument."""
result = self.name
if (self.minimum_count == self.maximum_count and
self.minimum_count == 1):
return result
minmax = (self.minimum_count, self.maximum_count)
if minmax == (0, 1):
return result + '?'
if minmax == (1, None):
return result + '+'
if minmax == (0, None):
return result + '*'
if minmax[1] == None:
minmax = (minmax[0], '')
return result + '{%s,%s}' % minmax
def parse(self, argv):
"""Evaluate arguments in argv.
Used arguments are removed from argv.
:param argv: The arguments to parse.
:return: The parsed results as a list.
"""
count = 0
result = []
while len(argv) > count and (
count < self.maximum_count or self.maximum_count is None):
arg = argv[count]
count += 1
result.append(self._parse_one(arg))
if count < self.minimum_count:
raise ValueError('not enough arguments present in %s' % argv)
del argv[:count]
return result
def _parse_one(self, arg):
"""Parse a single argument.
:param arg: An arg from an argv.
:result: The parsed argument.
:raises ValueError: If the arg cannot be parsed/validated.
"""
raise NotImplementedError(self._parse_one)
#
# Copyright (c) 2010 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
"""An Argument that looks up a command object."""
from testrepository.arguments import AbstractArgument
from testrepository import commands
class CustomError(ValueError):
def __str__(self):
return self.args[0]
class CommandArgument(AbstractArgument):
"""An argument that looks up a command."""
def _parse_one(self, arg):
try:
return commands._find_command(arg)
except KeyError:
raise CustomError("Could not find command '%s'." % arg)
#
# Copyright (c) 2010 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
"""An Argument that simply stores a string."""
from testrepository.arguments import AbstractArgument
class StringArgument(AbstractArgument):
"""An argument that stores a string verbatim."""
def _parse_one(self, arg):
return arg
#
# Copyright (c) 2009, 2010 Testrepository Contributors
#
# Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
# license at the users choice. A copy of both licenses are available in the
# project source as Apache-2.0 and BSD. You may not use this file except in
# compliance with one of these two licences.
#
# Unless required by applicable law or agreed to in writing, software
# distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# license you chose for the specific language governing permissions and
# limitations under that license.
"""'Commands' for testr.
The code in this module contains the Command base class, the run_argv
entry point to run CLI commands.
Actual commands can be found in testrepository.commands.$commandname.
For example, testrepository.commands.init is the init command name, and
testrepository.command.show_stats would be the show-stats command (if one
existed). The Command discovery logic looks for a class in the module with
the same name - e.g. tesrepository.commands.init.init would be the class.
That class must obey the testrepository.commands.Command protocol, but does
not need to be a subclass.
Plugins and extensions wanting to add commands should install them into
testrepository.commands (perhaps by extending the testrepository.commands
__path__ to include a directory containing their commands - no __init__ is
needed in that directory.)
"""
import os
import sys
import subunit