Commit caaff1ed authored by Nicolas Dandrimont's avatar Nicolas Dandrimont

New upstream version 2.0.0+hg20180501.1

parent ab407fad
repo: 26cf45c5599ff39ae649d05d52e820039256157e
node: 8af008648277cd9bf2a6699d6db550ad12d88453
node: 9f47c3efad9a83b16ebbb8e1e20cb3170dfe6683
branch: default
tag: 2.0.0
latesttag: 2.0.0
latesttagdistance: 59
changessincelatesttag: 82
......@@ -3,3 +3,4 @@
da62959c106a3f06ffc4a1fcdb5e6ae97e77152d master
6d69f09be1d4ac9365432b2a8a4bc130b521ccda github/master
a85679977ede1586585ae0b92cd121e1b8835481 github/master
8af008648277cd9bf2a6699d6db550ad12d88453 2.0.0
language: python
- "3.6"
- gnupg
- zip
- pip install -r requirements.txt
- make build
- NOSE=nosetests make test
......@@ -4,6 +4,161 @@
Note: This file contains only changes in the 'default' branch.
- Added polymorphic POSSIGN(_, account) function to the SQL shell in order to
support correcting for per-account sign.
- Implemented polymorphic NEG() function in the SQL shell, which works on
instances of Decimal, Amount, Position, Inventory.
- Changed the SQL shell's operators to dispatch to Python's default operator
selection instead of coercing to Decimal instances.
- Added QUARTER() function to the SQL shell.
- Updated version of flex on Ubuntu (2.6.1) generates warnings. Fixed a few.
But Beancount now fails on account names with a dash inthyem; added a test
for those. (The lexer is not the issue; I suspect a bison update from 4/17.)
- Fixed #157: Building a price map with zero prices should not fail,
especially when done accidentally.
- Fixed #264: Converted zip compression in bake from using an external zip
tool to using the 'zipfile' module packaged with Python. I'm doing this to
fix the CI build as well.
- Fixed #265: Refined the timezone specification of the price sources. In
doing this, I documented it in the beancount.prices.source module and now
pass in a timezone-aware datetime instance instead of a date, and the source
is now required to also require a timezone-aware datetime instance. I fixed
all the tests, and added a utility to simulate running them from different
- Implemented multi-file support for bean-doctor context command. Use it like
bean-doctor context <toplevel.beancount> <file.beancount>:<lineno>
The previous syntax is still supported; like this:
bean-doctor context <toplevel.beancount> <lineno>
- Since I added a dependency on 'requests', forgot to add to checkdeps. Done.
- Handle error on an invalid implementation of a price source (e.g., in this
case an obsoleted price source string of 'google' imports something else and
referenceing .Source fails).
- Bubble up error from Yahoo price source.
- Fixed #267: Implemented a price source from the IEX exchange.
- Implemented Quandl importer on its REST API. Another implementation, based
on the Quandl Python client API was proposed by Hugo Ideler (see PR34), but
I didn't want to add a dependency on the client library (nor on Pandas), and
after reading the description on Quandl API's website, I figured the library
was simple enough the output could be parsed directly. I wrote a new
implementation from scratch, parsing just the time-series datasets.
- Merged PR68 from Martin Michlmayr, added Travis Yaml support file.
- Fixed Emacs functionality to uncomment regions (change submitted by Stefan
Monnier on the list).
- Fixed #193: Removed the broken Google Finance price source importer. Google
has removed this service in the favor of a special Search page and this has
been broken for the recent little while.
- Fixed #203: Updated the Yahoo price source implementation to use the
undocumented v7 and v8 APIs; moved away from the "ichart" and "download"
APIs. The Yahoo importer is back in function.
This introduced a new dependency on the awesome 'requests' library, which
really, was quite inevitable).
- Merged PR65: Use parse_version to compare versions; then replicated
parse_version() locally to avoid the dependency on setuptools.
- Merged PR15 by Johannes Harms:
Added the following features to the forecasting plugin:
- Yearly repetition of recurring transactions (instead of just monthly)
- Users may optionally specify an end date
- Users may optionally specify the number of repetitions
"I used this code to calculate scenarios for a possible real estate purchase
and I contribute a cleaned-up version as an example file."
- Merged PR54 & Fixed #232: Align quoted commodities correctly in ledger price
statements (from Martin Michlmayr).
- Merged PR57: query_execute: Allow ordering by columns which may be None
(from Jakob Schnitzer). Made some adjustments to generalize on all
- Merged PR62: Also allow 0/0 in SAFEDIV() (from Jakob Schnitzer), with small
adjustments to catch only the specific invalid operations on Decimal.
- Merged PR64 from yagebu@ (Jakob Schnitzer), which makes the Inventory class
rely on the fact that Position is immutable in order to avoid unnecessary
copies. This yields an 18% speedup with no difference.
- Removed unused and invalid mutating method Position.set_units(). Amazing
obsolete stuff like this can still be found.
- Fixed #10: "print from close_date" causes problems on account open. This was
stemming from a bug in the implementation of the FROM clause execution. I
changed the "entry" context functions to also accept a RowContext object, as
the simple functions may be called in either entry or posting context.
(The SQL shell remains the part of Beancount which is severely undertested
and wading into that code once again has convinced me I need to rewrite the
shell from scratch, it'll be simpler and better. The distinction between
entry and posting context will go away in the next version.)
- Fixed #120: Documents directive with a trailing slash fails.
- Fixed failing test in amount_test probably due to some recent merges.
- Completed PR #59 which adds support for globbing patterns in includes,
inserting a chdir() around glob.glob() to handle expansion of relative
filenames. (Credits to Martin Michlmayr for submitting the patch in the
first place.)
- Released 2.0.0. Today we're starting versioning Beancount. Many people have
......@@ -129,7 +129,7 @@ debug:
# Push to github.
hg bookmark -r default master
hg push github
hg push git+ssh://
# Bake a release.
......@@ -137,7 +137,8 @@ release:
# Run the unittests.
NOSE = nosetests3
NOSE ?= nosetests3
vtest vtests verbose-test verbose-tests:
$(NOSE) -v -s beancount
......@@ -222,7 +223,7 @@ LINT_SRCS = \
# Note: Keeping to 3.5 because 3.6 pylint raises an exception (as of 2017-01-15).
#PYLINT = pylint
PYLINT = python3.5 $(shell which pylint)
PYLINT = python3 $(shell which pylint)
pylint lint:
$(PYLINT) --rcfile=$(PWD)/etc/pylintrc $(LINT_SRCS)
......@@ -247,3 +248,11 @@ sphinx sphinx_odt2rst:
./tools/ --cache=/tmp/convert_test.cache '1WjARst_cSxNE-Lq6JnJ5CC41T3WndEsiMw4d46r2694' /tmp/
# This does not work well; import errors just won't go away, it's slow, and it
# seems you have to pregenerate all .pyi to do anything useful.
find $(PWD)/beancount -name '*.py' | parallel -j16 pytype --pythonpath=$(PWD) -o {}i {}
pytype --pythonpath=$(PWD) beancount/utils/
......@@ -1153,6 +1153,9 @@
* The new rendering code should use the DisplayContext. This would close the
"display_context" branch.
* Type checking should be implemented using type annotations, but it should
also be implemented on the basic operators (see #6, for instance).
*** SQLite3 Integration
......@@ -4669,6 +4672,26 @@
it down. Every couple of years I clean this mess up and put it in the sections
- It might be interesting to support some sort of transfer syntax that would
allow the movement of Position's across accounts, without converting to the
cost, something like this:
2000-01-01 * "Transfer shares from BrokerOne to BrokerTwo to sell at BrokerTwo"
Assets:BrokerOne -10 HOOL {}
While this example is simple enough, a fully general version of this might
not be, but it's worth prototyping. This would require some sort of new
syntax for these "transfers."
- Rationalize the strategy for reporting errors.
- Rendering idea: If a posting moved the balance in the unusual direction
(positive for Income, negative for Expenses), it could be rendered with a
highlight or a different color. These are rare and unusualy. One could also
do the same for Assets (when the amount moves up) or Liabilities (when it
moves down).
- Create a new "AssertQuery" directive to assert that the result of a
particular query produces a particular Inventory. For example, you could
check your total contributions to an account over a period of time like
......@@ -45,6 +45,10 @@ class TestAmount(unittest.TestCase):
amount2 = Amount.from_string('100 USD')
self.assertEqual(amount1, amount2)
amount3 = Amount(D('0.00000001'), 'BTC')
amount4 = Amount.from_string('0.00000001 BTC')
self.assertEqual(amount3, amount4)
Amount.from_string(' 100.00 USD ')
with self.assertRaises(ValueError):
......@@ -57,13 +61,15 @@ class TestAmount(unittest.TestCase):
Amount.from_string('100.00 U')
def test_tostring(self):
amount = Amount(D('100034.023'), 'USD')
amount1 = Amount(D('100034.023'), 'USD')
self.assertEqual('100034.023 USD', str(amount1))
self.assertEqual('100034.023 USD', str(amount))
amount2 = Amount(D('0.00000001'), 'BTC')
self.assertEqual('0.00000001 BTC', str(amount2))
dcontext = display_context.DisplayContext()
dformat =
self.assertEqual('100,034.023 USD', amount.to_string(dformat))
self.assertEqual('100,034.023 USD', amount1.to_string(dformat))
def test_comparisons(self):
amount1 = Amount(D('100'), 'USD')
......@@ -194,6 +194,23 @@ def get_all_payees(entries):
return sorted(all_payees)
def get_all_links(entries):
"""Return a list of all the links seen in the given entries.
entries: A list of directive instances.
A set of links strings.
all_links = set()
for entry in entries:
if not isinstance(entry, Transaction):
if entry.links:
return sorted(all_links)
def get_leveln_parent_accounts(account_names, level, nrepeats=0):
"""Return a list of all the unique leaf names are level N in an account hierarchy.
......@@ -96,6 +96,11 @@ class TestGetters(unittest.TestCase):
payees = getters.get_all_payees(entries)
self.assertEqual(['La Colombe', 'Whole Foods Market'], payees)
def test_get_all_links(self):
entries = loader.load_string(TEST_INPUT)[0]
links = getters.get_all_links(entries)
self.assertEqual(['ee89ada94a39'], links)
def test_get_leveln_parent_accounts(self):
account_names = ['Assets:US:Cash',
......@@ -64,13 +64,17 @@ class Inventory(list):
"""Create a new inventory using a list of existing positions.
positions: A list of Position instances.
positions: A list of Position instances or an existing Inventory
if positions:
assert isinstance(positions, Iterable)
for position in positions:
if isinstance(positions, Inventory):
list.__init__(self, positions)
if positions:
assert isinstance(positions, Iterable)
for position in positions:
def to_string(self, dformat=DEFAULT_FORMATTER, parens=True):
"""Convert an Inventory instance to a printable string.
......@@ -114,7 +118,7 @@ class Inventory(list):
An instance of Inventory, equal to this one.
return Inventory(list(map(copy.copy, self)))
return Inventory(self)
def __eq__(self, other):
"""Equality predicate.
......@@ -257,16 +257,6 @@ class Position(_Position):
# Note: We use Decimal() for efficiency.
return Position(copy.copy(self.units), copy.copy(self.cost))
def set_units(self, units):
"""Set the units. This is required to abstract over the old and the new position
units: An instance of Amount.
assert isinstance(units, Amount)
self.units = units # pylint: disable=assigning-non-slot
def currency_pair(self):
"""Return the currency pair associated with this position.
......@@ -110,7 +110,8 @@ def build_price_map(entries):
del price_map[remove]
inverted_list = [(date, ONE/rate)
for (date, rate) in remove_list]
for (date, rate) in remove_list
if rate != ZERO]
# Unzip and sort each of the entries and eliminate duplicates on the date.
......@@ -85,6 +85,17 @@ class TestPriceMap(unittest.TestCase):
self.assertEqual(5, len(price_map[('CAD', 'USD')]))
def test_build_price_map_zero_prices(self, entries, _, __):
1999-12-27 commodity EFA
2010-10-01 price EFA 57.53 EFA
2010-11-01 price EFA 0 EFA
2011-03-01 price EFA 60.69 EFA
price_map = prices.build_price_map(entries)
def test_lookup_price_and_inverse(self, entries, _, __):
......@@ -3,8 +3,10 @@
__copyright__ = "Copyright (C) 2013-2016 Martin Blais"
__license__ = "GNU GPLv2"
from os import path
import collections
import functools
import glob
import hashlib
import importlib
import io
......@@ -16,7 +18,6 @@ import struct
import textwrap
import time
import warnings
from os import path
from beancount.utils import misc_utils
from beancount.core import data
......@@ -26,6 +27,7 @@ from beancount.parser import options
from beancount.parser import printer
from beancount.ops import validation
from beancount.utils import encryption
from beancount.utils import file_utils
LoadError = collections.namedtuple('LoadError', 'source message entry')
......@@ -371,8 +373,20 @@ def _parse_recursive(sources, log_timings, encoding=None):
aggregate_options_map(options_map, src_options_map)
# Add includes to the list of sources to process.
for include_filename in src_options_map['include']:
# Add includes to the list of sources to process. chdir() for glob,
# which uses it indirectly.
include_expanded = []
with file_utils.chdir(cwd):
for include_filename in src_options_map['include']:
matched_filenames = glob.glob(include_filename, recursive=True)
if matched_filenames:
LoadError(data.new_metadata("<load>", 0),
'File glob "{}" does not match any files'.format(
include_filename), None))
for include_filename in include_expanded:
if not path.isabs(include_filename):
include_filename = path.join(cwd, include_filename)
include_filename = path.normpath(include_filename)
......@@ -153,7 +153,7 @@ class TestLoadIncludes(unittest.TestCase):
entries, errors, options_map = loader.load_file(
path.join(tmp, 'root.beancount'))
self.assertEqual(1, len(errors))
self.assertRegex(errors[0].message, 'does not exist')
self.assertRegex(errors[0].message, 'does not (match any|exist)')
list(map(path.basename, options_map['include'])))
......@@ -42,7 +42,7 @@ def process_documents(entries, options_map):
accounts = getters.get_accounts(entries)
# Accumulate all the entries.
for directory in document_dirs:
for directory in map(path.normpath, document_dirs):
new_entries, new_errors = find_documents(directory, filename, accounts)
......@@ -64,6 +64,22 @@ class TestDocuments(account_test.TmpFilesTestBase, cmptest.TestCase):
self.assertEqual(0, len(errors))
def test_process_documents_trailing_slash(self):
input_filename = path.join(self.root, 'input.beancount')
open(input_filename, 'w').write(textwrap.dedent("""
option "plugin_processing_mode" "raw"
option "documents" "ROOT/"
2014-01-01 open Assets:US:Bank:Checking
2014-01-01 open Liabilities:US:Bank
""").replace('ROOT', self.root))
entries, _, options_map = loader.load_file(input_filename)
entries, errors = documents.process_documents(entries, options_map)
doc_entries = [entry for entry in entries if isinstance(entry, data.Document)]
self.assertEqual(1, len(doc_entries))
def test_verify_document_files_exist(self):
entries, _, options_map = loader.load_string(textwrap.dedent("""
option "plugin_processing_mode" "raw"
This diff is collapsed.
......@@ -3,7 +3,6 @@
#define yyIN_HEADER 1
#line 6 "beancount/parser/lexer.h"
#line 23 "beancount/parser/lexer.l"
/* Includes. */
#include <math.h>
......@@ -12,7 +11,6 @@
#include "parser.h"
#include "grammar.h"
/* Build and accumulate an error on the builder object. */
void build_lexer_error(const char* string, size_t length);
......@@ -20,8 +18,6 @@ void build_lexer_error(const char* string, size_t length);
* exception state. */
void build_lexer_error_from_exception(void);
/* Callback call site with error handling. */
#define BUILD_LEX(method_name, format, ...) \
yylval->pyobj = PyObject_CallMethod(builder, method_name, format, __VA_ARGS__); \
......@@ -37,13 +33,11 @@ void build_lexer_error_from_exception(void);
return LEX_ERROR; \
/* Initialization/finalization methods. These are separate from the yylex_init()
* and yylex_destroy() and they call them. */
void yylex_initialize(const char* filename, const char* encoding);
void yylex_finalize(void);
/* Global declarations; defined below. */
extern int yy_eof_times;
extern const char* yy_filename;
......@@ -57,8 +51,6 @@ extern char* strbuf_end; /* Current buffer sentinel (points to the final nul).
extern char* strbuf_ptr; /* Current insertion point in buffer. */
void strbuf_realloc(size_t num_new_chars);
/* Handle detecting the beginning of line. */
extern int yy_line_tokens; /* Number of tokens since the bol. */
......@@ -70,15 +62,12 @@ extern int yy_line_tokens; /* Number of tokens since the bol. */
yycolumn += yyleng; \
/* Skip the rest of the input line. */
int yy_skip_line(void);
/* Utility functions. */
int strtonl(const char* buf, size_t nchars);
/* Append characters to the static string buffer and verify. */
#define SAFE_COPY_CHAR(value) \
if (strbuf_ptr >= strbuf_end) { \
......@@ -86,10 +75,7 @@ int strtonl(const char* buf, size_t nchars);
} \
*strbuf_ptr++ = value;
#line 93 "beancount/parser/lexer.h"
#line 79 "beancount/parser/lexer.h"
#define YY_INT_ALIGNED short int
......@@ -98,7 +84,7 @@ int strtonl(const char* buf, size_t nchars);
#define FLEX_BETA
......@@ -177,25 +163,13 @@ typedef unsigned int flex_uint32_t;
#endif /* ! FLEXINT_H */
#ifdef __cplusplus
/* The "const" storage-class-modifier is valid. */
#define YY_USE_CONST
#else /* ! __cplusplus */
/* C99 requires __STDC__ to be defined as 1. */
#if defined (__STDC__)
#define YY_USE_CONST
#endif /* defined (__STDC__) */
#endif /* ! __cplusplus */
/* TODO: this is always defined, so inline it */
#define yyconst const
#if defined(__GNUC__) && __GNUC__ >= 3
#define yynoreturn __attribute__((__noreturn__))
#define yyconst
#define yynoreturn
/* Size of default input buffer. */
......@@ -221,7 +195,7 @@ typedef struct yy_buffer_state *YY_BUFFER_STATE;
typedef size_t yy_size_t;
extern yy_size_t yyleng;
extern int yyleng;
extern FILE *yyin, *yyout;
......@@ -237,12 +211,12 @@ struct yy_buffer_state
/* Size of input buffer in bytes, not including room for EOB
* characters.
yy_size_t yy_buf_size;
int yy_buf_size;
/* Number of characters read into yy_ch_buf, not including EOB
* characters.
yy_size_t yy_n_chars;
int yy_n_chars;
/* Whether we "own" the buffer - i.e., we know we created it,
* and can realloc() it to grow it, and should free() it to
......@@ -265,7 +239,7 @@ struct yy_buffer_state
int yy_bs_lineno; /**< The line count. */
int yy_bs_column; /**< The column count. */
/* Whether to try to fill the input buffer when we reach the
* end of it.
......@@ -286,7 +260,7 @@ void yypop_buffer_state (void );
YY_BUFFER_STATE yy_scan_buffer (char *base,yy_size_t size );
YY_BUFFER_STATE yy_scan_string (yyconst char *yy_str );
YY_BUFFER_STATE yy_scan_bytes (yyconst char *bytes,yy_size_t len );
YY_BUFFER_STATE yy_scan_bytes (yyconst char *bytes,int len );
void *yyalloc (yy_size_t );
void *yyrealloc (void *,yy_size_t );
......@@ -345,7 +319,7 @@ FILE *yyget_out (void );
void yyset_out (FILE * _out_str );
yy_size_t yyget_leng (void );
int yyget_leng (void );
char *yyget_text (void );
......@@ -430,6 +404,6 @@ extern int yylex \
#line 382 "beancount/parser/lexer.l"
#line 434 "beancount/parser/lexer.h"
#line 408 "beancount/parser/lexer.h"
#undef yyIN_HEADER
#endif /* yyHEADER_H */
......@@ -314,10 +314,10 @@ FALSE {
/* All other characters. */
[^\\\"]+ {
if ( yyleng > (yy_size_t)(strbuf_end - strbuf_ptr) ) {
if ( yyleng > (strbuf_end - strbuf_ptr) ) {
size_t i;
ssize_t i;
for (i = 0; i < yyleng; ++i) {
*strbuf_ptr++ = yytext[i];
......@@ -252,6 +252,18 @@ class TestLexer(unittest.TestCase):
], tokens)
def test_account_names_with_dash(self, tokens, errors):
('ACCOUNT', 1, 'Equity:Beginning-Balances', 'Equity:Beginning-Balances'),
('EOL', 2, '\n', None),
('EOL', 2, '\x00', None),
], tokens)
def test_invalid_directive(self, tokens, errors):
......@@ -16,7 +16,25 @@ A user can create a create a transaction like this:
and new transactions will be created monthly for the following year.
Note the use of the '#' flag and the word 'MONTHLY' which defines the
periodicity. This needs more expansion, but as an example, it works.
The number of recurrences can optionally be specified either by providing an
end date or by specifying the number of times that the transaction will be
repeated. For example:
2014-03-08 # "Electricity bill [MONTHLY UNTIL 2019-12-31]""
Expenses:Electricity 50.10 USD
Assets:Checking -50.10 USD
2014-03-08 # "Electricity bill [MONTHLY REPEAT 10 TIMES]""
Expenses:Electricity 50.10 USD
Assets:Checking -50.10 USD
Transactions can be also be repeated at yearly intervals, e.g.:
2014-03-08 # "Electricity bill [YEARLY REPEAT 10 TIMES]""
Expenses:Electricity 50.10 USD
Assets:Checking -50.10 USD