New upstream version 1.13.2

parent 7c0c319d
Anthony Newnam <> Anthony <>
He Ping <> heping <>
Hu Xiuyun <> Hu xiuyun <>
Hu Xiuyun <> Hu Xiuyun <>
Jelly Chen <> chenguodong <>
......@@ -5,6 +5,6 @@
<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.6</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.7</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
......@@ -6,9 +6,11 @@ development workflow. Repo is not meant to replace Git, only to make it
easier to work with Git. The repo command is an executable Python script
that you can put anywhere in your path.
* Homepage:
* Bug reports:
* Source:
* Homepage:
* Bug reports:
* Source:
* Overview:
* Docs:
* [repo Manifest Format](./docs/
* [repo Hooks](./docs/
* [Submitting patches](./
......@@ -19,6 +19,7 @@ import platform
import re
import sys
from event_log import EventLog
from error import NoSuchProjectError
from error import InvalidProjectGroupsError
......@@ -28,6 +29,7 @@ class Command(object):
common = False
event_log = EventLog()
manifest = None
_optparse = None
......@@ -216,11 +218,6 @@ class Command(object):
return result
# pylint: disable=W0223
# Pylint warns that the `InteractiveCommand` and `PagedCommand` classes do not
# override method `Execute` which is abstract in `Command`. Since that method
# is always implemented in classes derived from `InteractiveCommand` and
# `PagedCommand`, this warning can be suppressed.
class InteractiveCommand(Command):
"""Command which requires user interaction on the tty and
must not run within a pager, even if the user asks to.
......@@ -236,8 +233,6 @@ class PagedCommand(Command):
def WantPager(self, _opt):
return True
# pylint: enable=W0223
class MirrorSafeCommand(object):
"""Command permits itself to run within a mirror,
# repo hooks
Repo provides a mechanism to hook specific stages of the runtime with custom
python modules. All the hooks live in one git project which is checked out by
the manifest (specified during `repo init`), and the manifest itself defines
which hooks are registered.
These are useful to run linters, check formatting, and run quick unittests
before allowing a step to proceed (e.g. before uploading a commit to Gerrit).
A complete example can be found in the Android project. It can be easily
re-used by any repo based project and is not specific to Android.<br>
## Approvals
When a hook is processed the first time, the user is prompted for approval.
We don't want to execute arbitrary code without explicit consent. For manifests
fetched via secure protocols (e.g. https://), the user is prompted once. For
insecure protocols (e.g. http://), the user is prompted whenever the registered
repohooks project is updated and a hook is triggered.
## Manifest Settings
For the full syntax, see the [repo manifest format](./
Here's a short example from
The `<project>` line checks out the repohooks git repo to the local
`tools/repohooks/` path. The `<repo-hooks>` line says to look in the project
with the name `platform/tools/repohooks` for hooks to run during the
`pre-upload` phase.
<project path="tools/repohooks" name="platform/tools/repohooks" />
<repo-hooks in-project="platform/tools/repohooks" enabled-list="pre-upload" />
## Source Layout
The repohooks git repo should have a python file with the same name as the hook.
So if you want to support the `pre-upload` hook, you'll need to create a file
named ``. Repo will dynamically load that module when processing
the hook and then call the `main` function in it.
Hooks should have their `main` accept `**kwargs` for future compatibility.
## Runtime
Hook return values are ignored.
Any uncaught exceptions from the hook will cause the step to fail. This is
intended as a fallback safety check though rather than the normal flow. If
you want your hook to trigger a failure, it should call `sys.exit()` (after
displaying relevant diagnostics).
Output (stdout & stderr) are not filtered in any way. Hooks should generally
not be too verbose. A short summary is nice, and some status information when
long running operations occur, but long/verbose output should be used only if
the hook ultimately fails.
The hook runs from the top level of the repo client where the operation is
For example, if the repo client is under `~/tree/`, then that is where the hook
runs, even if you ran repo in a git repository at `~/tree/src/foo/`, or in a
subdirectory of that git repository in `~/tree/src/foo/bar/`.
Hooks frequently start off by doing a `os.chdir` to the specific project they're
called on (see below) and then changing back to the original dir when they're
Python's `sys.path` is modified so that the top of repohooks directory comes
first. This should help simplify the hook logic to easily allow importing of
local modules.
Repo does not modify the state of the git checkout. This means that the hooks
might be running in a dirty git repo with many commits and checked out to the
latest one. If the hook wants to operate on specific git commits, it needs to
manually discover the list of pending commits, extract the diff/commit, and
then check it directly. Hooks should not normally modify the active git repo
(such as checking out a specific commit to run checks) without first prompting
the user. Although user interaction is discouraged in the common case, it can
be useful when deploying automatic fixes.
## Hooks
Here are all the points available for hooking.
### pre-upload
This hook runs when people run `repo upload`.
The `` file should be defined like:
def main(project_list, worktree_list=None, **kwargs):
"""Main function invoked directly by repo.
We must use the name "main" as that is what repo requires.
project_list: List of projects to run on.
worktree_list: A list of directories. It should be the same length as
project_list, so that each entry in project_list matches with a
directory in worktree_list. If None, we will attempt to calculate
the directories automatically.
kwargs: Leave this here for forward-compatibility.
......@@ -21,6 +21,7 @@ import subprocess
import tempfile
from error import EditorError
import platform_utils
class Editor(object):
"""Manages the user's preferred text editor."""
......@@ -82,7 +83,12 @@ least one of these before using this command.""", file=sys.stderr)
fd = None
if re.compile("^.*[$ \t'].*$").match(editor):
if platform_utils.isWindows():
# Split on spaces, respecting quoted strings
import shlex
args = shlex.split(editor)
shell = False
elif re.compile("^.*[$ \t'].*$").match(editor):
args = [editor + ' "$@"', 'sh']
shell = True
......@@ -107,4 +113,4 @@ least one of these before using this command.""", file=sys.stderr)
if fd:
# Copyright (C) 2017 The Android Open Source Project
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import print_function
import json
import multiprocessing
TASK_COMMAND = 'command'
TASK_SYNC_NETWORK = 'sync-network'
TASK_SYNC_LOCAL = 'sync-local'
class EventLog(object):
"""Event log that records events that occurred during a repo invocation.
Events are written to the log as a consecutive JSON entries, one per line.
Each entry contains the following keys:
- id: A ('RepoOp', ID) tuple, suitable for storing in a datastore.
The ID is only unique for the invocation of the repo command.
- name: Name of the object being operated upon.
- task_name: The task that was performed.
- start: Timestamp of when the operation started.
- finish: Timestamp of when the operation finished.
- success: Boolean indicating if the operation was successful.
- try_count: A counter indicating the try count of this task.
- parent: A ('RepoOp', ID) tuple indicating the parent event for nested
Valid task_names include:
- command: The invocation of a subcommand.
- sync-network: The network component of a sync command.
- sync-local: The local component of a sync command.
Specific tasks may include additional informational properties.
def __init__(self):
"""Initializes the event log."""
self._log = []
self._parent = None
def Add(self, name, task_name, start, finish=None, success=None,
try_count=1, kind='RepoOp'):
"""Add an event to the log.
name: Name of the object being operated upon.
task_name: A sub-task that was performed for name.
start: Timestamp of when the operation started.
finish: Timestamp of when the operation finished.
success: Boolean indicating if the operation was successful.
try_count: A counter indicating the try count of this task.
kind: The kind of the object for the unique identifier.
A dictionary of the event added to the log.
event = {
'id': (kind, _NextEventId()),
'name': name,
'task_name': task_name,
'start_time': start,
'try': try_count,
if self._parent:
event['parent'] = self._parent['id']
if success is not None or finish is not None:
self.FinishEvent(event, finish, success)
return event
def AddSync(self, project, task_name, start, finish, success):
"""Add a event to the log for a sync command.
project: Project being synced.
task_name: A sub-task that was performed for name.
start: Timestamp of when the operation started.
finish: Timestamp of when the operation finished.
success: Boolean indicating if the operation was successful.
A dictionary of the event added to the log.
event = self.Add(project.relpath, task_name, start, finish, success)
if event is not None:
event['project'] =
if project.revisionExpr:
event['revision'] = project.revisionExpr
if project.remote.url:
event['project_url'] = project.remote.url
if project.remote.fetchUrl:
event['remote_url'] = project.remote.fetchUrl
event['git_hash'] = project.GetCommitRevisionId()
except Exception:
return event
def GetStatusString(self, success):
"""Converst a boolean success to a status string.
success: Boolean indicating if the operation was successful.
status string.
return 'pass' if success else 'fail'
def FinishEvent(self, event, finish, success):
"""Finishes an incomplete event.
event: An event that has been added to the log.
finish: Timestamp of when the operation finished.
success: Boolean indicating if the operation was successful.
A dictionary of the event added to the log.
event['status'] = self.GetStatusString(success)
event['finish_time'] = finish
return event
def SetParent(self, event):
"""Set a parent event for all new entities.
event: The event to use as a parent.
self._parent = event
def Write(self, filename):
"""Writes the log out to a file.
filename: The file to write the log to.
with open(filename, 'w+') as f:
for e in self._log:
json.dump(e, f, sort_keys=True)
# An integer id that is unique across this invocation of the program.
_EVENT_ID = multiprocessing.Value('i', 1)
def _NextEventId():
"""Helper function for grabbing the next unique id.
A unique, to this invocation of the program, integer id.
with _EVENT_ID.get_lock():
val = _EVENT_ID.value
_EVENT_ID.value += 1
return val
......@@ -14,14 +14,14 @@
# limitations under the License.
from __future__ import print_function
import fcntl
import os
import select
import sys
import subprocess
import tempfile
from signal import SIGTERM
from error import GitError
import platform_utils
from trace import REPO_TRACE, IsTrace, Trace
from wrapper import Wrapper
......@@ -78,16 +78,6 @@ def terminate_ssh_clients():
_git_version = None
class _sfd(object):
"""select file descriptor class"""
def __init__(self, fd, dest, std_name):
assert std_name in ('stdout', 'stderr')
self.fd = fd
self.dest = dest
self.std_name = std_name
def fileno(self):
return self.fd.fileno()
class _GitCall(object):
def version(self):
p = GitCommand(None, ['--version'], capture_stdout=True)
......@@ -162,6 +152,7 @@ class GitCommand(object):
if ssh_proxy:
_setenv(env, 'REPO_SSH_SOCK', ssh_sock())
_setenv(env, 'GIT_SSH', _ssh_proxy())
_setenv(env, 'GIT_SSH_VARIANT', 'ssh')
if 'http_proxy' in env and 'darwin' == sys.platform:
s = "'http.proxy=%s'" % (env['http_proxy'],)
......@@ -253,19 +244,16 @@ class GitCommand(object):
def _CaptureOutput(self):
p = self.process
s_in = [_sfd(p.stdout, sys.stdout, 'stdout'),
_sfd(p.stderr, sys.stderr, 'stderr')]
s_in = platform_utils.FileDescriptorStreams.create()
s_in.add(p.stdout, sys.stdout, 'stdout')
s_in.add(p.stderr, sys.stderr, 'stderr')
self.stdout = ''
self.stderr = ''
for s in s_in:
flags = fcntl.fcntl(s.fd, fcntl.F_GETFL)
fcntl.fcntl(s.fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
while s_in:
in_ready, _, _ =, [], [])
while not s_in.is_done:
in_ready =
for s in in_ready:
buf =
buf =
if not buf:
......@@ -20,6 +20,7 @@ import errno
import json
import os
import re
import ssl
import subprocess
import sys
......@@ -41,6 +42,7 @@ else:
from signal import SIGTERM
from error import GitError, UploadError
import platform_utils
from trace import Trace
if is_python3():
from http.client import HTTPException
......@@ -50,16 +52,24 @@ else:
from git_command import GitCommand
from git_command import ssh_sock
from git_command import terminate_ssh_clients
from git_refs import R_CHANGES, R_HEADS, R_TAGS
R_HEADS = 'refs/heads/'
R_TAGS = 'refs/tags/'
ID_RE = re.compile(r'^[0-9a-f]{40}$')
def IsChange(rev):
return rev.startswith(R_CHANGES)
def IsId(rev):
return ID_RE.match(rev)
def IsTag(rev):
return rev.startswith(R_TAGS)
def IsImmutable(rev):
return IsChange(rev) or IsId(rev) or IsTag(rev)
def _key(name):
parts = name.split('.')
if len(parts) < 2:
......@@ -259,7 +269,7 @@ class GitConfig(object):
if os.path.getmtime(self._json) \
<= os.path.getmtime(self.file):
return None
except OSError:
return None
......@@ -271,7 +281,7 @@ class GitConfig(object):
except (IOError, ValueError):
return None
def _SaveJson(self, cache):
......@@ -283,7 +293,7 @@ class GitConfig(object):
except (IOError, TypeError):
if os.path.exists(self._json):
def _ReadGit(self):
......@@ -296,8 +306,9 @@ class GitConfig(object):
d = self._do('--null', '--list')
if d is None:
return c
for line in d.decode('utf-8').rstrip('\0').split('\0'): # pylint: disable=W1401
# Backslash is not anomalous
if not is_python3():
d = d.decode('utf-8')
for line in d.rstrip('\0').split('\0'):
if '\n' in line:
key, val = line.split('\n', 1)
......@@ -492,7 +503,7 @@ def close_ssh():
d = ssh_sock(create=False)
if d:
except OSError:
......@@ -524,7 +535,7 @@ def GetUrlCookieFile(url, quiet):
for line in p.stdout:
line = line.strip()
if line.startswith(cookieprefix):
cookiefile = line[len(cookieprefix):]
cookiefile = os.path.expanduser(line[len(cookieprefix):])
if line.startswith(proxyprefix):
proxy = line[len(proxyprefix):]
# Leave subprocess open, as cookie file may be transient.
......@@ -543,7 +554,10 @@ def GetUrlCookieFile(url, quiet):
if e.errno == errno.ENOENT:
pass # No persistent proxy.
yield GitConfig.ForUser().GetString('http.cookiefile'), None
cookiefile = GitConfig.ForUser().GetString('http.cookiefile')
if cookiefile:
cookiefile = os.path.expanduser(cookiefile)
yield cookiefile, None
def _preconnect(url):
m = URI_ALL.match(url)
......@@ -604,7 +618,7 @@ class Remote(object):
connectionUrl = self._InsteadOf()
return _preconnect(connectionUrl)
def ReviewUrl(self, userEmail):
def ReviewUrl(self, userEmail, validate_certs):
if self._review_url is None:
if is None:
return None
......@@ -612,7 +626,7 @@ class Remote(object):
u =
if u.startswith('persistent-'):
u = u[len('persistent-'):]
if u.split(':')[0] not in ('http', 'https', 'sso'):
if u.split(':')[0] not in ('http', 'https', 'sso', 'ssh'):
u = 'http://%s' % u
if u.endswith('/Gerrit'):
u = u[:len(u) - len('/Gerrit')]
......@@ -628,13 +642,20 @@ class Remote(object):
host, port = os.environ['REPO_HOST_PORT_INFO'].split()
self._review_url = self._SshReviewUrl(userEmail, host, port)
REVIEW_CACHE[u] = self._review_url
elif u.startswith('sso:'):
elif u.startswith('sso:') or u.startswith('ssh:'):
self._review_url = u # Assume it's right
REVIEW_CACHE[u] = self._review_url
elif 'REPO_IGNORE_SSH_INFO' in os.environ:
self._review_url = http_url
REVIEW_CACHE[u] = self._review_url
info_url = u + 'ssh_info'
info = urllib.request.urlopen(info_url).read()
if not validate_certs:
context = ssl._create_unverified_context()
info = urllib.request.urlopen(info_url, context=context).read()
info = urllib.request.urlopen(info_url).read()
if info == 'NOT_AVAILABLE' or '<' in info:
# If `info` contains '<', we assume the server gave us some sort
# of HTML response back, like maybe a login page.
......@@ -15,12 +15,14 @@
import os
from trace import Trace
import platform_utils
R_HEADS = 'refs/heads/'
R_TAGS = 'refs/tags/'
R_PUB = 'refs/published/'
R_M = 'refs/remotes/m/'
R_CHANGES = 'refs/changes/'
R_HEADS = 'refs/heads/'
R_TAGS = 'refs/tags/'
R_PUB = 'refs/published/'
R_M = 'refs/remotes/m/'
class GitRefs(object):
......@@ -126,9 +128,9 @@ class GitRefs(object):
def _ReadLoose(self, prefix):
base = os.path.join(self._gitdir, prefix)
for name in os.listdir(base):
for name in platform_utils.listdir(base):
p = os.path.join(base, name)
if os.path.isdir(p):
if platform_utils.isdir(p):
self._mtime[prefix] = os.path.getmtime(base)
self._ReadLoose(prefix + name + '/')
elif name.endswith('.lock'):
......@@ -138,7 +140,7 @@ class GitRefs(object):
def _ReadLoose1(self, path, name):
fd = open(path, 'rb')
fd = open(path)
except IOError:
# From Gerrit Code Review 2.12.1
# From Gerrit Code Review 2.14.6
# Part of Gerrit Code Review (
......@@ -20,7 +20,7 @@
# Check for, and add if missing, a unique Change-Id
# An example hook script to verify if you are on battery, in case you
# are running Linux or OS X. Called by git-gc --auto with no arguments.
# The hook should exit with non-zero status after issuing an appropriate
# message if it wants to stop the auto repacking.
# are running Windows, Linux or OS X. Called by git-gc --auto with no
# arguments. The hook should exit with non-zero status after issuing an
# appropriate message if it wants to stop the auto repacking.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
......@@ -19,7 +19,17 @@
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
if test -x /sbin/on_ac_power && /sbin/on_ac_power
if uname -s | grep -q "_NT-"
if test -x $SYSTEMROOT/System32/Wbem/wmic
STATUS=$(wmic path win32_battery get batterystatus /format:list | tr -d '\r\n')
[ "$STATUS" = "BatteryStatus=2" ] && exit 0 || exit 1
exit 0
if test -x /sbin/on_ac_power && (/sbin/on_ac_power;test $? -ne 1)
exit 0
elif test "$(cat /sys/class/power_supply/AC/online 2>/dev/null)" = 1
......@@ -37,6 +37,7 @@ except ImportError:
kerberos = None
from color import SetDefaultColoring
import event_log
from trace import SetTrace
from git_command import git, GitCommand
from git_config import init_ssh, close_ssh
......@@ -54,15 +55,13 @@ from error import NoSuchProjectError
from error import RepoChangedException
import gitc_utils
from manifest_xml import GitcManifest, XmlManifest
from pager import RunPager
from pager import RunPager, TerminatePager
from wrapper import WrapperPath, Wrapper
from subcmds import all_commands
if not is_python3():
# pylint:disable=W0622
input = raw_input
# pylint:enable=W0622
global_options = optparse.OptionParser(
usage="repo [-p|--paginate|--no-pager] COMMAND [ARGS]"
......@@ -85,6 +84,9 @@ global_options.add_option('--time',
dest='show_version', action='store_true',
help='display this version of repo')
dest='event_log', action='store',
help='filename of event log to append timeline to')
class _Repo(object):
def __init__(self, repodir):
......@@ -176,6 +178,8 @@ class _Repo(object):
start = time.time()
cmd_event = cmd.event_log.Add(name, event_log.TASK_COMMAND, start)
result = cmd.Execute(copts, cargs)
except (DownloadError, ManifestInvalidRevisionError,
......@@ -198,8 +202,13 @@ class _Repo(object):
print('error: project group must be enabled for the project in the current directory', file=sys.stderr)
result = 1
except SystemExit as e:
if e.code:
result = e.code
elapsed = time.time() - start
finish = time.time()
elapsed = finish - start
hours, remainder = divmod(elapsed, 3600)
minutes, seconds = divmod(remainder, 60)
if gopts.time:
......@@ -209,6 +218,12 @@ class _Repo(object):
print('real\t%dh%dm%.3fs' % (hours, minutes, seconds),