Commit 090da3fd authored by John Westcott IV's avatar John Westcott IV

Send/Receive feature PR

Added dependencies and related to POSSIBLE_TYPES

They are needed for the import functionality

i before e except after c

Fixed typo in receive

Added receive option

Changed classname from Destroyer to Cleaner

Added empty command

Added required=False to scm-type field.

This field is not required by the API to create a project

Removed items from export if managed_by_tower = True

Added credential_type to dependencies for a credential

This allows resoluation of a credential_type when importing a credential

Added send command

Missing __init__.py file

Added type parameter to cleaner

Send and Receive now can do json or yaml

Removed possible types, adding its items into resource objects

Changed how assets are specified for export

Added exception if nothing was passed in to process

Added all option to cleaner method

Fixed issue where properties added in the source were not caught as needed to be updated

Added send/recieve/empty docs

Updated help text for empty

Move removeEncryptedValues to common

Added deepclone of asset when determining if asset needs update

This fixed an issue when removing items in the asset that were later referenced

Fixed some issues when importing credentials

Added workflow nodes import/export

Initial commit of inventory groups/hosts/sources and fix for vault_creds in job_template

Updated API to strip off /api/vX if passed in as url

This allows you send URLs retreived from the API back into TowerCLI

Caught additional exception

If already parsed Java is passed in you could get a TypeError exception which was not caught.

Added notification types

Added workflow name to error message when extracting nodes

When this condition occurs the message is displayed but in a receive no details are present as to what workflow had the error

Added all_pages when requesting workflow_nodes

Allowed for multiple tokens to be saved in ~/.tower_cli_token.json

Added debug log if asset does not have POST options

Removed unused import and debug message

If an object does not have POST options, just return

Changed how errors are logged

Added project updates if asset was created or updated

Removed local_path from SCM based projects

Changed removal of encrypted values from asset to exported_asset

Added workflow inventory and credential resolution

Added inventory group import/export

Added additional credential types for export/import

Added options for passwords for new items

Variablized workdlow node types

Fixed workflow comparison

Spell checking

All post PR fixes for send/receive

Modified print statements for python3

Wrapped TowerCLI class in try

This should give us a better error when performing exports

Don't get source project if its None

Fixed user passwords

Moved projects before inventory

An inventory with a source from a project requires projects to be created first

Added inventory source_script resolution

Another change for comparing workflow nodes

Revert "Merge remote-tracking branch 'upstream/master'"

This reverts commit d8b0d2dbff2713a86aee4c7ca901d789b6e1d470, reversing
changes made to 87ea5d2c6118f797dd908404974e790df8e4059a.

Fixed unintended injection of documentation

Added todo

Removed comments for pull request

Removed commants on dropping format

Fixed bad merge

Updated for flake8

Added all_pages to lists

Better handeling of extra_vars

Multiple changes

Made empty take params like receive

Output of send and empty are now ansible-ish

Changed function name to python standards

Extracted 'work' into method for calling programatically

Fixed missing :

Fixed newline at end of file causing flake8 failure

removed closing #'s

Changed from passing on TypeError to continuing if not a string

Added min_value and max_value to list of known types

Changed from .prefix to .get_prefix()

Moved out touchup_extra_vars so that it applies to all asset_types

Fixed issue where tox was sending back a json_token of 'invalid' causing issues with the dict

Added missing POST endpoint

Use six for string compare
parent 34b11e5f
......@@ -67,7 +67,25 @@ Some examples:
# Filter job list for currently running jobs
$ tower-cli job list --status=running
# Export all objects
$ tower-cli receive --all
# Export all credentials
$ tower-cli receive --credential all
# Export a credential named "My Credential"
$ tower-cli receive --credential "My Credential"
# Import from a JSON file named assets.json
$ tower-cli send assets.json
# Import anything except an organization defined in a JSON file named assets.json
$ tower-cli send --prevent organization assets.json
# Copy all assets from one instance to another
$ tower-cli receive --tower-host tower1.example.com --all | tower-cli send --tower-host tower2.example.com
When in doubt, help is available!
......@@ -94,6 +112,7 @@ In specific, ``tower-cli --help`` lists all available resources in the current v
config Read or write tower-cli configuration.
credential Manage credentials within Ansible Tower.
credential_type Manage credential types within Ansible Tower.
empty Empties assets from Tower.
group Manage groups belonging to an inventory.
host Manage hosts belonging to a group within an...
instance Check instances within Ansible Tower.
......@@ -108,8 +127,10 @@ In specific, ``tower-cli --help`` lists all available resources in the current v
notification_template Manage notification templates within Ansible...
organization Manage organizations within Ansible Tower.
project Manage projects within Ansible Tower.
receive Export assets from Tower.
role Add and remove users/teams from roles.
schedule Manage schedules within Ansible Tower.
send Import assets into Tower.
setting Manage settings within Ansible Tower.
team Manage teams within Ansible Tower.
user Manage users within Ansible Tower.
......
......@@ -225,6 +225,8 @@ class TowerAuthTokenTests(unittest.TestCase):
with mock.patch('tower_cli.api.json.load', return_value={'token': 'foobar', 'expires': expires}):
with client.test_mode as t:
t.register('/authtoken/', json.dumps({}), status_code=200, method='OPTIONS')
t.register('/authtoken/', json.dumps({'token': 'foobar', 'expires': expires}), status_code=200,
method='POST')
self.auth(self.req)
self.assertEqual(self.req.headers['Authorization'], 'Token foobar')
......
......@@ -54,18 +54,27 @@ class BasicTowerAuth(AuthBase):
def _get_auth_token(self):
filename = os.path.expanduser('~/.tower_cli_token.json')
token_json = None
try:
with open(filename) as f:
token_json = json.load(f)
if not isinstance(token_json, dict) or 'token' not in token_json or 'expires' not in token_json or \
dt.utcnow() > dt.strptime(token_json['expires'], TOWER_DATETIME_FMT):
if not isinstance(token_json, dict) or self.cli_client.get_prefix() not in token_json or \
'token' not in token_json[self.cli_client.get_prefix()] or \
'expires' not in token_json[self.cli_client.get_prefix()] or \
dt.utcnow() > dt.strptime(token_json[self.cli_client.get_prefix()]['expires'], TOWER_DATETIME_FMT):
raise Exception("Current token expires.")
return 'Token ' + token_json['token']
return 'Token ' + token_json[self.cli_client.get_prefix()]['token']
except Exception as e:
debug.log('Acquiring and caching auth token due to:\n%s' % str(e), fg='blue', bold=True)
token_json = self._acquire_token()
if not isinstance(token_json, dict) or 'token' not in token_json or 'expires' not in token_json:
raise exc.AuthError('Invalid Tower auth token format: %s' % json.dumps(token_json))
if not isinstance(token_json, dict):
token_json = {}
token_json[self.cli_client.get_prefix()] = self._acquire_token()
if not isinstance(token_json[self.cli_client.get_prefix()], dict) or \
'token' not in token_json[self.cli_client.get_prefix()] or \
'expires' not in token_json[self.cli_client.get_prefix()]:
raise exc.AuthError('Invalid Tower auth token format: %s' % json.dumps(
token_json[self.cli_client.get_prefix()]
))
with open(filename, 'w') as f:
json.dump(token_json, f)
try:
......@@ -75,7 +84,7 @@ class BasicTowerAuth(AuthBase):
'Unable to set permissions on {0} - {1} '.format(filename, e),
UserWarning
)
return 'Token ' + token_json['token']
return 'Token ' + token_json[self.cli_client.get_prefix()]['token']
def __call__(self, r):
if 'Authorization' in r.headers:
......@@ -188,6 +197,13 @@ class Client(Session):
"""Make a request to the Ansible Tower API, and return the
response.
"""
# If the URL has the api/vX at the front strip it off
# This is common to have if you are extracting a URL from an existing object.
# For example, any of the 'related' fields of an object will have this
import re
url = re.sub("^/?api/v[0-9]+/", "", url)
# Piece together the full URL.
use_version = not url.startswith('/o/')
url = '%s%s' % (self.get_prefix(use_version), url.lstrip('/'))
......
......@@ -29,8 +29,9 @@ from tower_cli.api import client
from tower_cli.conf import with_global_options, Parser, settings, _apply_runtime_setting
from tower_cli.utils import secho, supports_oauth
from tower_cli.constants import CUR_API_VERSION
from tower_cli.cli.transfer.common import SEND_ORDER
__all__ = ['version', 'config', 'login', 'logout']
__all__ = ['version', 'config', 'login', 'logout', 'receive', 'send', 'empty']
@click.command()
......@@ -209,6 +210,9 @@ def config(key=None, value=None, scope='user', global_=False, unset=False):
click.echo('Configuration updated successfully.')
# TODO:
# Someday it would be nice to create these for us
# Thus the import reference to transfer.common.SEND_ORDER
@click.command()
@click.argument('username', required=True)
@click.option('--password', required=True, prompt=True, hide_input=True)
......@@ -291,3 +295,106 @@ def logout():
'This version of Tower does not support OAuth2.0'
)
config.main(['oauth_token', '--unset', '--scope=user'])
@click.command()
@with_global_options
@click.option('--organization', required=False, multiple=True)
@click.option('--user', required=False, multiple=True)
@click.option('--team', required=False, multiple=True)
@click.option('--credential_type', required=False, multiple=True)
@click.option('--credential', required=False, multiple=True)
@click.option('--notification_template', required=False, multiple=True)
@click.option('--inventory_script', required=False, multiple=True)
@click.option('--inventory', required=False, multiple=True)
@click.option('--project', required=False, multiple=True)
@click.option('--job_template', required=False, multiple=True)
@click.option('--workflow', required=False, multiple=True)
@click.option('--all', is_flag=True)
def receive(organization=None, user=None, team=None, credential_type=None, credential=None,
notification_template=None, inventory_script=None, inventory=None, project=None, job_template=None,
workflow=None, all=None):
"""Export assets from Tower.
'tower receive' exports one or more assets from a Tower instance
For all of the possible assets types the TEXT can either be the assets name
(or username for the case of a user) or the keyword all. Specifying all
will export all of the assets of that type.
"""
from tower_cli.cli.transfer.receive import Receiver
receiver = Receiver()
assets_to_export = {}
for asset_type in SEND_ORDER:
assets_to_export[asset_type] = locals()[asset_type]
receiver.receive(all=all, asset_input=assets_to_export)
@click.command()
@with_global_options
@click.argument('source', required=False, nargs=-1)
@click.option('--prevent', multiple=True, required=False,
help='Prevents import of a specific asset type.\n'
'Multiple prevent options can be passed.')
@click.option('--secret_management', multiple=False, required=False, default='default',
type=click.Choice(['default', 'prompt', 'random']),
help='What to do with secrets for new items.\n'
'default - use "password", "token" or "secret" depending on the field'
'prompt - prompt for the secret to use'
'random - generate a random string for the secret'
)
@click.option('--no-color', is_flag=True,
help="Disable color output"
)
def send(source=None, prevent=None, secret_management='default', no_color=False):
"""Import assets into Tower.
'tower send' imports one or more assets into a Tower instance
The import can take either JSON or YAML.
Data can be sent on stdin (i.e. from tower-cli receive pipe) and/or from files
or directories passed as parameters.
If a directory is specified only files that end in .json, .yaml or .yml will be
imported. Other files will be ignored.
"""
from tower_cli.cli.transfer.send import Sender
sender = Sender(no_color)
sender.send(source, prevent, secret_management)
@click.command()
@with_global_options
@click.option('--organization', required=False, multiple=True)
@click.option('--user', required=False, multiple=True)
@click.option('--team', required=False, multiple=True)
@click.option('--credential_type', required=False, multiple=True)
@click.option('--credential', required=False, multiple=True)
@click.option('--notification_template', required=False, multiple=True)
@click.option('--inventory_script', required=False, multiple=True)
@click.option('--inventory', required=False, multiple=True)
@click.option('--project', required=False, multiple=True)
@click.option('--job_template', required=False, multiple=True)
@click.option('--workflow', required=False, multiple=True)
@click.option('--all', is_flag=True)
@click.option('--no-color', is_flag=True,
help="Disable color output"
)
def empty(organization=None, user=None, team=None, credential_type=None, credential=None, notification_template=None,
inventory_script=None, inventory=None, project=None, job_template=None, workflow=None,
all=None, no_color=False):
"""Empties assets from Tower.
'tower empty' removes all assets from Tower
"""
# Create an import/export object
from tower_cli.cli.transfer.cleaner import Cleaner
destroyer = Cleaner(no_color)
assets_to_export = {}
for asset_type in SEND_ORDER:
assets_to_export[asset_type] = locals()[asset_type]
destroyer.go_ham(all=all, asset_input=assets_to_export)
import click
from tower_cli.cli.transfer import common
from tower_cli.cli.transfer.logging_command import LoggingCommand
from tower_cli.exceptions import TowerCLIError
import tower_cli
class Cleaner(LoggingCommand):
def __init__(self, no_color):
self.no_color = no_color
def go_ham(self, all=False, asset_input=None):
stdout = click.get_text_stream('stdout')
stdin = click.get_text_stream('stdin')
assets_from_input = common.get_assets_from_input(all, asset_input)
stdout.write("Please confirm that you want to clean the Tower instance by typing 'YES': ")
response = stdin.readline()
if response.strip() != 'YES':
stdout.write("\nAborting request to empty the instance\n")
return
self.print_intro()
for asset_type in common.SEND_ORDER[::-1]:
if asset_type not in assets_from_input:
continue
identifier = common.get_identity(asset_type)
assets_to_remove = []
if assets_from_input[asset_type]['all']:
resources = tower_cli.get_resource(asset_type).list(all_pages=True)
if 'results' not in resources:
continue
assets_to_remove = assets_to_remove + resources['results']
else:
for name in assets_from_input[asset_type]['names']:
try:
resource = tower_cli.get_resource(asset_type).get(**{identifier: name})
assets_to_remove.append(resource)
except TowerCLIError:
self.print_header_row(asset_type, name)
self.log_ok("Asset does not exist")
for asset in assets_to_remove:
self.print_header_row(asset_type, asset[identifier])
if 'managed_by_tower' in asset and asset['managed_by_tower']:
self.log_warn("{} is managed by tower and can not be deleted".format(identifier))
continue
try:
tower_cli.get_resource(asset_type).delete(asset['id'])
self.log_change("Asset removed")
except Exception as e:
self.log_error("Unable to delete : {}".format(e))
self.print_recap()
This diff is collapsed.
from tower_cli.utils import secho
import click
class LoggingCommand:
ok_messages = 0
error_messages = 0
changed_messages = 0
warn_messages = 0
columns = None
no_color = False
def print_intro(self):
self.my_print("")
def get_rows(self):
self.columns = click.get_terminal_size()[0]
def print_header_row(self, asset_type, asset_name):
if self.columns is None:
self.get_rows()
else:
self.my_print('')
# The 4 is from 2 spaces and 2 brackets
stars = '*' * (int(self.columns) - (len(asset_type + asset_name) + 4))
self.my_print("{} [{}] {}".format(asset_type.replace("_", " ").upper(), asset_name, stars))
def print_recap(self):
if self.columns is None:
self.get_rows()
else:
self.my_print('')
# Print the recap message
recap_message = "PLAY RECAP"
stars = '*' * (int(self.columns) - (len(recap_message) + 1))
self.my_print("{} {}".format(recap_message, stars))
spaces_separating_messages = " "
# Print OK count
self.my_print(spaces_separating_messages, nl=False)
ok = "ok={}".format(self.ok_messages)
color = 'black'
if self.ok_messages > 0:
color = 'green'
self.my_print(ok, fg=color, nl=False)
# Print CHANGED count
self.my_print(spaces_separating_messages, nl=False)
changed = "changed={}".format(self.changed_messages)
color = 'black'
if self.changed_messages > 0:
color = 'yellow'
self.my_print(changed, fg=color, nl=False)
# Print WARNING count
self.my_print(spaces_separating_messages, nl=False)
warnings = "warnings={}".format(self.warn_messages)
color = 'black'
if self.warn_messages > 0:
color = 'magenta'
self.my_print(warnings, fg=color, nl=False)
# Print ERROR count
self.my_print(spaces_separating_messages, nl=False)
error = "failed={}".format(self.error_messages)
color = 'black'
if self.error_messages > 0:
color = 'red'
self.my_print(error, fg=color)
self.my_print("")
def log_warn(self, warn_message):
self.my_print(" [WARNING]: {}".format(warn_message), fg='magenta', bold=True)
self.warn_messages = self.warn_messages + 1
def log_ok(self, ok_message):
self.my_print("{}".format(ok_message), fg='green')
self.ok_messages = self.ok_messages + 1
def log_change(self, change_message):
self.my_print("{}".format(change_message), fg='yellow')
self.changed_messages = self.changed_messages + 1
def log_error(self, error_message):
self.my_print("{}".format(error_message), fg='red', bold=True)
self.error_messages = self.error_messages + 1
def my_print(self, message=None, fg='black', bold=False, nl=True):
if self.no_color:
secho(message, fg='black', bold=False, nl=nl)
else:
secho(message, fg=fg, bold=bold, nl=nl)
import tower_cli
from tower_cli.exceptions import TowerCLIError
from tower_cli.cli.transfer import common
from tower_cli.conf import settings
from tower_cli.utils import parser
import click
class Receiver:
def receive(self, all=False, asset_input=None):
exported_objects = self.export_assets(all, asset_input)
stdout = click.get_text_stream('stdout')
if settings.format == 'human' or settings.format == 'json':
import json
stdout.write(json.dumps(exported_objects, indent=2))
elif settings.format == 'yaml':
import yaml
stdout.write(parser.ordered_dump(exported_objects, Dumper=yaml.SafeDumper, default_flow_style=False))
else:
raise TowerCLIError("Format {} is unsupported".format(settings.format))
stdout.write("\n")
def export_assets(self, all, asset_input):
# Extract and consolidate all of the items we got on the command line
assets_to_export = common.get_assets_from_input(all, asset_input)
# These will be returned from this method
exported_objects = []
for asset_type in assets_to_export:
# Load the API options for this asset_type of asset
types_api_options = common.get_api_options(asset_type)
# Now we are going to extract the objects from Tower and put them into an array for processing
acquired_assets_to_export = []
identifier = common.get_identity(asset_type)
# Now we are either going to get everything or just one item and append that to the assets_to_export
if assets_to_export[asset_type]['all']:
resources = tower_cli.get_resource(asset_type).list(all_pages=True)
if 'results' not in resources:
continue
acquired_assets_to_export = acquired_assets_to_export + resources['results']
else:
for name in assets_to_export[asset_type]['names']:
try:
resource = tower_cli.get_resource(asset_type).get(**{identifier: name})
except TowerCLIError as e:
raise TowerCLIError("Unable to get {} named {} : {}".format(asset_type, name, e))
acquired_assets_to_export.append(resource)
# Next we are going to loop over the objects we got from Tower
for asset in acquired_assets_to_export:
# If this object is managed_by_tower then move on
if 'managed_by_tower' in asset and asset['managed_by_tower']:
continue
# Resolve the dependencies
common.resolve_asset_dependencies(asset, asset_type)
# Create a new object with the ASSET_TYPE_KEY and merge the options in from the object we got
exported_asset = {common.ASSET_TYPE_KEY: asset_type}
common.map_node_to_post_options(types_api_options, asset, exported_asset)
# Clean up any $encrypted$ values
common.remove_encrypted_values(exported_asset)
# Special cases for touch up
if asset_type == 'project':
# Exported projects that are not manual don't need a local path
common.remove_local_path_from_scm_project(exported_asset)
# Next we are going to go after any of
for relation in tower_cli.get_resource(asset_type).related:
if common.ASSET_RELATION_KEY not in exported_asset:
exported_asset[common.ASSET_RELATION_KEY] = {}
if relation == 'workflow_nodes':
exported_asset[common.ASSET_RELATION_KEY][relation] = common.extract_workflow_nodes(asset)
elif relation == 'survey_spec':
survey_spec = tower_cli.get_resource(asset_type).survey(asset['id'])
exported_asset[common.ASSET_RELATION_KEY][relation] = survey_spec
elif relation == 'host' or relation == 'inventory_source':
exported_asset[common.ASSET_RELATION_KEY][relation] = \
common.extract_inventory_relations(asset, relation)['items']
elif relation == 'group':
exported_asset[common.ASSET_RELATION_KEY][relation] = \
common.extract_inventory_groups(asset)['items']
elif relation == 'notification_templates':
for notification_type in common.NOTIFICATION_TYPES:
exported_asset[common.ASSET_RELATION_KEY][notification_type] = \
common.extract_notifications(asset, notification_type)
# Finally add the object to the list of objects that are being exported
exported_objects.append(exported_asset)
return exported_objects
This diff is collapsed.
......@@ -164,6 +164,8 @@ class BaseResource(six.with_metaclass(ResourceMeta)):
cli_help = ''
endpoint = None
identity = ('name',)
dependencies = []
related = []
# The basic methods for interacting with a resource are `read`, `write`,
# and `delete`; these cover basic CRUD situations and have options
......
......@@ -22,6 +22,7 @@ class Resource(models.Resource):
cli_help = 'Manage credentials within Ansible Tower.'
endpoint = '/credentials/'
identity = ('organization', 'user', 'team', 'name')
dependencies = ['organization', 'credential_type']
name = models.Field(unique=True)
description = models.Field(required=False, display=False)
......
......@@ -25,6 +25,8 @@ class Resource(models.Resource):
cli_help = 'Manage inventory within Ansible Tower.'
endpoint = '/inventories/'
identity = ('organization', 'name')
dependencies = ['organization']
related = ['host', 'group', 'inventory_source']
name = models.Field(unique=True)
description = models.Field(required=False, display=False)
......
......@@ -22,6 +22,7 @@ class Resource(models.Resource):
"""A resource for inventory scripts."""
cli_help = 'Manage inventory scripts within Ansible Tower.'
endpoint = '/inventory_scripts/'
dependencies = ['organization']
name = models.Field(unique=True)
description = models.Field(required=False, display=False)
......
......@@ -27,6 +27,8 @@ class Resource(models.SurveyResource):
"""A resource for job templates."""
cli_help = 'Manage job templates.'
endpoint = '/job_templates/'
dependencies = ['inventory', 'credential', 'project', 'vault_credential']
related = ['survey_spec', 'notification_templates']
name = models.Field(unique=True)
description = models.Field(required=False, display=False)
......
......@@ -16,6 +16,7 @@
import click
import json
import copy
import six
from tower_cli import get_resource, models, resources, exceptions as exc
from tower_cli.utils import debug
......@@ -26,6 +27,7 @@ class Resource(models.Resource):
"""A resource for notification templates."""
cli_help = 'Manage notification templates within Ansible Tower.'
endpoint = '/notification_templates/'
dependencies = ['organization']
# Actual fields
name = models.Field(unique=True)
......@@ -157,6 +159,11 @@ class Resource(models.Resource):
if field in kwargs:
result[field] = kwargs.pop(field)
if field in Resource.json_fields:
# If result[field] is not a string we can continue on
if not isinstance(result[field], six.string_types):
continue
try:
data = json.loads(result[field])
result[field] = data
......
......@@ -26,6 +26,8 @@ class Resource(models.Resource, models.MonitorableResource):
cli_help = 'Manage projects within Ansible Tower.'
endpoint = '/projects/'
unified_job_type = '/project_updates/'
dependencies = ['organization']
related = ['notification_templates']
name = models.Field(unique=True)
description = models.Field(required=False, display=False)
......@@ -38,7 +40,7 @@ class Resource(models.Resource, models.MonitorableResource):
('hg', 'hg'),
('svn', 'svn'),
('insights', 'insights'),
]),
]), required=False
)
scm_url = models.Field(required=False)
local_path = models.Field(
......
......@@ -22,6 +22,7 @@ class Resource(models.Resource):
cli_help = 'Manage teams within Ansible Tower.'
endpoint = '/teams/'
identity = ('organization', 'name')
dependencies = ['organization']
name = models.Field(unique=True)
organization = models.Field(type=types.Related('organization'))
......
......@@ -21,6 +21,7 @@ class Resource(models.Resource):
cli_help = 'Manage users within Ansible Tower.'
endpoint = '/users/'
identity = ('username',)
dependencies = ['organization']
username = models.Field(unique=True)
password = models.Field(required=False, display=False)
......
......@@ -155,6 +155,9 @@ class Resource(models.SurveyResource):
cli_help = 'Manage workflow job templates.'
endpoint = '/workflow_job_templates/'
unified_job_type = '/workflow_jobs/'
dependencies = ['organization']
related = ['survey_spec', 'workflow_nodes']
workflow_node_types = ['success_nodes', 'failure_nodes', 'always_nodes']
name = models.Field(unique=True)
description = models.Field(required=False, display=False)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment