Commit e25685db authored by Erik Johnston's avatar Erik Johnston

Imported Upstream version 0.18.5

parent 4217a384
sudo: false
language: python
python: 2.7
# tell travis to cache ~/.cache/pip
cache: pip
- TOX_ENV=packaging
- TOX_ENV=pep8
- TOX_ENV=py27
- pip install tox
- tox -e $TOX_ENV
Changes in synapse v0.18.5 (2016-12-16)
Bug fixes:
* Fix federation /backfill returning events it shouldn't (PR #1700)
* Fix crash in url preview (PR #1701)
Changes in synapse v0.18.5-rc3 (2016-12-13)
* Add support for E2E for guests (PR #1653)
* Add new API appservice specific public room list (PR #1676)
* Add new room membership APIs (PR #1680)
* Enable guest access for private rooms by default (PR #653)
* Limit the number of events that can be created on a given room concurrently
(PR #1620)
* Log the args that we have on UI auth completion (PR #1649)
* Stop generating refresh_tokens (PR #1654)
* Stop putting a time caveat on access tokens (PR #1656)
* Remove unspecced GET endpoints for e2e keys (PR #1694)
Bug fixes:
* Fix handling of 500 and 429's over federation (PR #1650)
* Fix Content-Type header parsing (PR #1660)
* Fix error when previewing sites that include unicode, thanks to kyrias (PR
* Fix some cases where we drop read receipts (PR #1678)
* Fix bug where calls to ``/sync`` didn't correctly timeout (PR #1683)
* Fix bug where E2E key query would fail if a single remote host failed (PR
Changes in synapse v0.18.5-rc2 (2016-11-24)
Bug fixes:
* Don't send old events over federation, fixes bug in -rc1.
Changes in synapse v0.18.5-rc1 (2016-11-24)
* Implement "event_fields" in filters (PR #1638)
* Use external ldap auth pacakge (PR #1628)
* Split out federation transaction sending to a worker (PR #1635)
* Fail with a coherent error message if `/sync?filter=` is invalid (PR #1636)
* More efficient notif count queries (PR #1644)
Changes in synapse v0.18.4 (2016-11-22)
This diff is collapsed.
......@@ -10,13 +10,13 @@
Setting ReCaptcha Keys
The keys are a config option on the home server config. If they are not
visible, you can generate them via --generate-config. Set the following value:
The keys are a config option on the home server config. If they are not
visible, you can generate them via --generate-config. Set the following value::
recaptcha_public_key: YOUR_PUBLIC_KEY
recaptcha_private_key: YOUR_PRIVATE_KEY
In addition, you MUST enable captchas via:
In addition, you MUST enable captchas via::
enable_registration_captcha: true
......@@ -25,7 +25,6 @@ Configuring IP used for auth
The ReCaptcha API requires that the IP address of the user who solved the
captcha is sent. If the client is connecting through a proxy or load balancer,
it may be required to use the X-Forwarded-For (XFF) header instead of the origin
IP address. This can be configured as an option on the home server like so:
IP address. This can be configured as an option on the home server like so::
captcha_ip_origin_is_x_forwarded: true
......@@ -22,3 +22,4 @@ export SYNAPSE_CACHE_FACTOR=1
--federation-reader \
--client-reader \
--appservice \
--federation-sender \
......@@ -15,6 +15,6 @@ tox -e py27 --notest -v
$TOX_BIN/pip install setuptools
python synapse/ | xargs -n1 $TOX_BIN/pip install
$TOX_BIN/pip install lxml
$TOX_BIN/pip install psycopg2
{ python synapse/
echo lxml psycopg2
} | xargs $TOX_BIN/pip install
......@@ -23,6 +23,45 @@ import sys
here = os.path.abspath(os.path.dirname(__file__))
# Some notes on ` test`:
# Once upon a time we used to try to make ` test` run `tox` to run the
# tests. That's a bad idea for three reasons:
# 1: ` test` is supposed to find out whether the tests work in the
# *current* environmentt, not whatever tox sets up.
# 2: Empirically, trying to install tox during the test run wasn't working ("No
# module named virtualenv").
# 3: The tox documentation advises against it[1].
# Even further back in time, we used to use setuptools_trial [2]. That has its
# own set of issues: for instance, it requires installation of Twisted to build
# an sdist (because the recommended mode of usage is to add it to
# `setup_requires`). That in turn means that in order to successfully run tox
# you have to have the python header files installed for whichever version of
# python tox uses (which is python3 on recent ubuntus, for example).
# So, for now at least, we stick with what appears to be the convention among
# Twisted projects, and don't attempt to do anything when someone runs
# ` test`; instead we direct people to run `trial` directly if they
# care.
# [1]:
# [2]:
class TestCommand(Command):
user_options = []
def initialize_options(self):
def finalize_options(self):
def run(self):
print ("""Synapse's tests cannot be run via To run them, try:
PYTHONPATH="." trial tests
def read_file(path_segments):
"""Read a file from the package. Takes a list of strings to join to
make the path"""
......@@ -39,38 +78,6 @@ def exec_file(path_segments):
return result
class Tox(Command):
user_options = [('tox-args=', 'a', "Arguments to pass to tox")]
def initialize_options(self):
self.tox_args = None
def finalize_options(self):
self.test_args = []
self.test_suite = True
def run(self):
#import here, cause outside the eggs aren't loaded
import tox
except ImportError:
import tox
raise RuntimeError(
"The tests need 'tox' to run. Please install 'tox'."
import shlex
args = self.tox_args
if args:
args = shlex.split(self.tox_args)
args = []
errno = tox.cmdline(args=args)
version = exec_file(("synapse", ""))["__version__"]
dependencies = exec_file(("synapse", ""))
long_description = read_file(("README.rst",))
......@@ -86,5 +93,5 @@ setup(
scripts=["synctl"] + glob.glob("scripts/*"),
cmdclass={'test': Tox},
cmdclass={'test': TestCommand},
......@@ -16,4 +16,4 @@
""" This is a reference implementation of a Matrix home server.
__version__ = "0.18.4"
__version__ = "0.18.5"
......@@ -39,6 +39,9 @@ AuthEventTypes = (
# guests always get this device id.
GUEST_DEVICE_ID = "guest_device"
class Auth(object):
......@@ -51,17 +54,6 @@ class Auth(object): = hs.get_datastore()
self.state = hs.get_state_handler()
# Docs for these currently lives at
# In addition, we have type == delete_pusher which grants access only to
# delete pushers.
"gen = ",
"guest = ",
"type = ",
"time < ",
"user_id = ",
def check_from_context(self, event, context, do_sig_check=True):
......@@ -685,31 +677,28 @@ class Auth(object):
def get_user_by_access_token(self, token, rights="access"):
""" Get a registered user's ID.
""" Validate access token and get user_id from it
token (str): The access token to get the user by.
rights (str): The operation being performed; the access token must
allow this.
dict : dict that includes the user and the ID of their access token.
AuthError if no user by that token exists or the token is invalid.
ret = yield self.get_user_from_macaroon(token, rights)
except AuthError:
# TODO(daniel): Remove this fallback when all existing access tokens
# have been re-issued as macaroons.
if self.hs.config.expire_access_token:
ret = yield self._look_up_user_by_access_token(token)
macaroon = pymacaroons.Macaroon.deserialize(token)
except Exception: # deserialize can throw more-or-less anything
# doesn't look like a macaroon: treat it as an opaque token which
# must be in the database.
# TODO: it would be nice to get rid of this, but apparently some
# people use access tokens which aren't macaroons
r = yield self._look_up_user_by_access_token(token)
def get_user_from_macaroon(self, macaroon_str, rights="access"):
macaroon = pymacaroons.Macaroon.deserialize(macaroon_str)
user_id = self.get_user_id_from_macaroon(macaroon)
user = UserID.from_string(user_id)
......@@ -724,11 +713,36 @@ class Auth(object):
guest = True
if guest:
# Guest access tokens are not stored in the database (there can
# only be one access token per guest, anyway).
# In order to prevent guest access tokens being used as regular
# user access tokens (and hence getting around the invalidation
# process), we look up the user id and check that it is indeed
# a guest user.
# It would of course be much easier to store guest access
# tokens in the database as well, but that would break existing
# guest tokens.
stored_user = yield
if not stored_user:
raise AuthError(
"Unknown user_id %s" % user_id,
if not stored_user["is_guest"]:
raise AuthError(
"Guest access token used for regular user",
ret = {
"user": user,
"is_guest": True,
"token_id": None,
"device_id": None,
# all guests get the same device id
"device_id": GUEST_DEVICE_ID,
elif rights == "delete_pusher":
# We don't store these tokens in the database
......@@ -750,7 +764,7 @@ class Auth(object):
# macaroon. They probably should be.
# TODO: build the dictionary from the macaroon once the
# above are fixed
ret = yield self._look_up_user_by_access_token(macaroon_str)
ret = yield self._look_up_user_by_access_token(token)
if ret["user"] != user:
"Macaroon user (%s) != DB user (%s)",
......@@ -798,27 +812,38 @@ class Auth(object):
macaroon(pymacaroons.Macaroon): The macaroon to validate
type_string(str): The kind of token required (e.g. "access", "refresh",
type_string(str): The kind of token required (e.g. "access",
verify_expiry(bool): Whether to verify whether the macaroon has expired.
This should really always be True, but no clients currently implement
token refresh, so we can't enforce expiry yet.
user_id (str): The user_id required
v = pymacaroons.Verifier()
# the verifier runs a test for every caveat on the macaroon, to check
# that it is met for the current request. Each caveat must match at
# least one of the predicates specified by satisfy_exact or
# specify_general.
v.satisfy_exact("gen = 1")
v.satisfy_exact("type = " + type_string)
v.satisfy_exact("user_id = %s" % user_id)
v.satisfy_exact("guest = true")
# verify_expiry should really always be True, but there exist access
# tokens in the wild which expire when they should not, so we can't
# enforce expiry yet (so we have to allow any caveat starting with
# 'time < ' in access tokens).
# On the other hand, short-term login tokens (as used by CAS login, for
# example) have an expiry time which we do want to enforce.
if verify_expiry:
v.satisfy_general(lambda c: c.startswith("time < "))
v.verify(macaroon, self.hs.config.macaroon_secret_key)
# access_tokens include a nonce for uniqueness: any value is acceptable
v.satisfy_general(lambda c: c.startswith("nonce = "))
v = pymacaroons.Verifier()
v.verify(macaroon, self.hs.config.macaroon_secret_key)
def _verify_expiry(self, caveat):
......@@ -829,15 +854,6 @@ class Auth(object):
now = self.hs.get_clock().time_msec()
return now < expiry
def _verify_recognizes_caveats(self, caveat):
first_space = caveat.find(" ")
if first_space < 0:
return False
second_space = caveat.find(" ", first_space + 1)
if second_space < 0:
return False
return caveat[:second_space + 1] in self._KNOWN_CAVEAT_PREFIXES
def _look_up_user_by_access_token(self, token):
ret = yield
......@@ -39,6 +39,7 @@ class Codes(object):
......@@ -71,6 +71,21 @@ class Filtering(object):
if key in user_filter_json["room"]:
if "event_fields" in user_filter_json:
if type(user_filter_json["event_fields"]) != list:
raise SynapseError(400, "event_fields must be a list of strings")
for field in user_filter_json["event_fields"]:
if not isinstance(field, basestring):
raise SynapseError(400, "Event field must be a string")
# Don't allow '\\' in event field filters. This makes matching
# events a lot easier as we can then use a negative lookbehind
# assertion to split '\.' If we allowed \\ then it would
# incorrectly split '\\.' See
if r'\\' in field:
raise SynapseError(
400, r'The escape character \ cannot itself be escaped'
def _check_definition_room_lists(self, definition):
"""Check that "rooms" and "not_rooms" are lists of room ids if they
are present
......@@ -152,6 +167,7 @@ class FilterCollection(object):
self.include_leave = filter_json.get("room", {}).get(
"include_leave", False
self.event_fields = filter_json.get("event_fields", [])
def __repr__(self):
return "<FilterCollection %s>" % (json.dumps(self._filter_json),)
......@@ -186,6 +202,26 @@ class FilterCollection(object):
def filter_room_account_data(self, events):
return self._room_account_data.filter(self._room_filter.filter(events))
def blocks_all_presence(self):
return (
self._presence_filter.filters_all_types() or
def blocks_all_room_ephemeral(self):
return (
self._room_ephemeral_filter.filters_all_types() or
self._room_ephemeral_filter.filters_all_senders() or
def blocks_all_room_timeline(self):
return (
self._room_timeline_filter.filters_all_types() or
self._room_timeline_filter.filters_all_senders() or
class Filter(object):
def __init__(self, filter_json):
......@@ -202,6 +238,15 @@ class Filter(object):
self.contains_url = self.filter_json.get("contains_url", None)
def filters_all_types(self):
return "*" in self.not_types
def filters_all_senders(self):
return "*" in self.not_senders
def filters_all_rooms(self):
return "*" in self.not_rooms
def check(self, event):
"""Checks whether the filter matches the given event.
This diff is collapsed.
......@@ -89,6 +89,9 @@ class ApplicationService(object):
self.namespaces = self._check_namespaces(namespaces) = id
if "|" in
raise Exception("application service ID cannot contain '|' character")
# .protocols is a publicly visible field
if protocols:
self.protocols = set(protocols)
......@@ -19,6 +19,7 @@ from synapse.api.errors import CodeMessageException
from synapse.http.client import SimpleHttpClient
from import serialize_event
from synapse.util.caches.response_cache import ResponseCache
from synapse.types import ThirdPartyInstanceID
import logging
import urllib
......@@ -177,6 +178,13 @@ class ApplicationServiceApi(SimpleHttpClient):
" valid result", uri)
for instance in info.get("instances", []):
network_id = instance.get("network_id", None)
if network_id is not None:
instance["instance_id"] = ThirdPartyInstanceID(, network_id,
except Exception as ex:
logger.warning("query_3pe_protocol to %s threw exception %s",
......@@ -50,6 +50,7 @@ handlers:
class: logging.StreamHandler
formatter: precise
filters: [context]
......@@ -27,17 +27,23 @@ class PasswordAuthProviderConfig(Config):
ldap_config = config.get("ldap_config", {})
self.ldap_enabled = ldap_config.get("enabled", False)
if self.ldap_enabled:
from synapse.util.ldap_auth_provider import LdapAuthProvider
from ldap_auth_provider import LdapAuthProvider
parsed_config = LdapAuthProvider.parse_config(ldap_config)
self.password_providers.append((LdapAuthProvider, parsed_config))
providers = config.get("password_providers", [])
for provider in providers:
# We need to import the module, and then pick the class out of
# that, so we split based on the last dot.
module, clz = provider['module'].rsplit(".", 1)
module = importlib.import_module(module)
provider_class = getattr(module, clz)
# This is for backwards compat when the ldap auth provider resided
# in this package.
if provider['module'] == "synapse.util.ldap_auth_provider.LdapAuthProvider":
from ldap_auth_provider import LdapAuthProvider
provider_class = LdapAuthProvider
# We need to import the module, and then pick the class out of
# that, so we split based on the last dot.
module, clz = provider['module'].rsplit(".", 1)
module = importlib.import_module(module)
provider_class = getattr(module, clz)
provider_config = provider_class.parse_config(provider["config"])
......@@ -50,7 +56,7 @@ class PasswordAuthProviderConfig(Config):
def default_config(self, **kwargs):
return """\
# password_providers:
# - module: "synapse.util.ldap_auth_provider.LdapAuthProvider"
# - module: "ldap_auth_provider.LdapAuthProvider"
# config:
# enabled: true
# uri: "ldap://"
......@@ -32,7 +32,6 @@ class RegistrationConfig(Config):
self.registration_shared_secret = config.get("registration_shared_secret")
self.user_creation_max_duration = int(config["user_creation_max_duration"])
self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
self.trusted_third_party_id_servers = config["trusted_third_party_id_servers"]
......@@ -55,11 +54,6 @@ class RegistrationConfig(Config):
# secret, even if registration is otherwise disabled.
registration_shared_secret: "%(registration_shared_secret)s"
# Sets the expiry for the short term user creation in
# milliseconds. For instance the bellow duration is two weeks
# in milliseconds.
user_creation_max_duration: 1209600000
# Set the number of bcrypt rounds used to generate password hash.
# Larger numbers increase the work factor needed to generate the hash.
# The default number of rounds is 12.
......@@ -30,6 +30,11 @@ class ServerConfig(Config):
self.use_frozen_dicts = config.get("use_frozen_dicts", False)
self.public_baseurl = config.get("public_baseurl")
# Whether to send federation traffic out in this process. This only
# applies to some federation traffic, and so shouldn't be used to
# "disable" federation
self.send_federation = config.get("send_federation", True)
if self.public_baseurl is not None:
if self.public_baseurl[-1] != '/':
self.public_baseurl += '/'
......@@ -16,6 +16,17 @@
from synapse.api.constants import EventTypes
from . import EventBase
from frozendict import frozendict
import re
# Split strings on "." but not "\." This uses a negative lookbehind assertion for '\'
# (?<!stuff) matches if the current position in the string is not preceded
# by a match for 'stuff'.
# TODO: This is fast, but fails to handle "foo\\.bar" which should be treated as
# the literal fields "foo\" and "bar" but will instead be treated as "foo\\.bar"
SPLIT_FIELD_REGEX = re.compile(r'(?<!\\)\.')
def prune_event(event):
""" Returns a pruned version of the given event, which removes all keys we
......@@ -97,6 +108,83 @@ def prune_event(event):
def _copy_field(src, dst, field):
"""Copy the field in 'src' to 'dst'.
For example, if src={"foo":{"bar":5}} and dst={}, and field=["foo","bar"]
then dst={"foo":{"bar":5}}.
src(dict): The dict to read from.
dst(dict): The dict to modify.
field(list<str>): List of keys to drill down to in 'src'.
if len(field) == 0: # this should be impossible
if len(field) == 1: # common case e.g. 'origin_server_ts'
if field[0] in src:
dst[field[0]] = src[field[0]]
# Else is a nested field e.g. 'content.body'
# Pop the last field as that's the key to move across and we need the
# parent dict in order to access the data. Drill down to the right dict.
key_to_move = field.pop(-1)
sub_dict = src
for sub_field in field: # e.g. sub_field => "content"
if sub_field in sub_dict and type(sub_dict[sub_field]) in [dict, frozendict]:
sub_dict = sub_dict[sub_field]
if key_to_move not in sub_dict:
# Insert the key into the output dictionary, creating nested objects
# as required. We couldn't do this any earlier or else we'd need to delete
# the empty objects if the key didn't exist.
sub_out_dict = dst
for sub_field in field:
sub_out_dict = sub_out_dict.setdefault(sub_field, {})
sub_out_dict[key_to_move] = sub_dict[key_to_move]
def only_fields(dictionary, fields):
"""Return a new dict with only the fields in 'dictionary' which are present
in 'fields'.
If there are no event fields specified then all fields are included.
The entries may include '.' charaters to indicate sub-fields.
So ['content.body'] will include the 'body' field of the 'content' object.
A literal '.' character in a field name may be escaped using a '\'.
dictionary(dict): The dictionary to read from.
fields(list<str>): A list of fields to copy over. Only shallow refs are
dict: A new dictionary with only the given fields. If fields was empty,
the same dictionary is returned.
if len(fields) == 0:
return dictionary
# for each field, convert it:
# ["content.body.thing\.with\.dots"] => [["content", "body", "thing\.with\.dots"]]
split_fields = [SPLIT_FIELD_REGEX.split(f) for f in fields]
# for each element of the output array of arrays:
# remove escaping so we can use the right key names.
split_fields[:] = [
[f.replace(r'\.', r'.') for f in field_array] for field_array in split_fields
output = {}
for field_array in split_fields:
_copy_field(dictionary, output, field_array)
return output
def format_event_raw(d):
return d
......@@ -137,7 +225,7 @@ def format_event_for_client_v2_without_room_id(d):
def serialize_event(e, time_now_ms, as_client_event=True,