Skip to content
Commits on Source (9)
......@@ -153,11 +153,6 @@ matrix:
# OSX Clang Builds
- os: osx
osx_image: xcode8.3
compiler: xcode8-clang-dev
env: CXX='clang++' BUILD_TYPE='Dev'
- os: osx
osx_image: xcode9.4
compiler: xcode9-clang-dev
......
......@@ -13,6 +13,30 @@ This project adheres to [Semantic Versioning](https://semver.org/).
### Fixed
## [1.11.1] - 2019-11-20
### Added
* Introduce a generic facility for setting output format options. They can
be set on the command line (`--format-option`/`-x`) or in the
`format_options` section in the config file. Settings can be any
OPTION=VALUE type string. There are two new settings: For the geojsonseq
format, the option `print_record_separator=false` replaces the command
line option `--omit-rs`/`-r` which is now deprecated. The `tags_format`
option for the Pg output format allows using the `hstore` type for tags
instead of `json(b)`.
### Changed
* Open output file earlier in tags-filter command, so we see it immediately
in case this fails.
### Fixed
* When tags-filter is used with `--remove-tags`, matching ways got their
tags removed if they are also referenced from relations. This was clearly
wrong.
## [1.11.0] - 2019-09-16
### Added
......@@ -517,7 +541,8 @@ This project adheres to [Semantic Versioning](https://semver.org/).
- Minor updates to documentation and build system
[unreleased]: https://github.com/osmcode/osmium-tool/compare/v1.11.0...HEAD
[unreleased]: https://github.com/osmcode/osmium-tool/compare/v1.11.1...HEAD
[1.11.1]: https://github.com/osmcode/osmium-tool/compare/v1.11.0...v1.11.1
[1.11.0]: https://github.com/osmcode/osmium-tool/compare/v1.10.0...v1.11.0
[1.10.0]: https://github.com/osmcode/osmium-tool/compare/v1.9.1...v1.10.0
[1.9.1]: https://github.com/osmcode/osmium-tool/compare/v1.9.0...v1.9.1
......
......@@ -26,7 +26,7 @@ project(osmium)
set(OSMIUM_VERSION_MAJOR 1)
set(OSMIUM_VERSION_MINOR 11)
set(OSMIUM_VERSION_PATCH 0)
set(OSMIUM_VERSION_PATCH 1)
set(OSMIUM_VERSION ${OSMIUM_VERSION_MAJOR}.${OSMIUM_VERSION_MINOR}.${OSMIUM_VERSION_PATCH})
......
osmium-tool (1.11.1-1~bpo10+1) buster-backports; urgency=medium
* Rebuild for buster-backports.
-- Bas Couwenberg <sebastic@debian.org> Tue, 26 Nov 2019 05:48:03 +0100
osmium-tool (1.11.1-1) unstable; urgency=medium
* New upstream release.
* Bump Standards-Version to 4.4.1, no changes.
* Drop spelling-errors.patch, applied upstream.
-- Bas Couwenberg <sebastic@debian.org> Thu, 21 Nov 2019 05:24:55 +0100
osmium-tool (1.11.0-1~bpo10+1) buster-backports; urgency=medium
* Rebuild for buster-backports.
......
......@@ -12,7 +12,7 @@ Build-Depends: debhelper (>= 9),
libosmium2-dev (>= 2.15.2),
pandoc,
zlib1g-dev
Standards-Version: 4.4.0
Standards-Version: 4.4.1
Vcs-Browser: https://salsa.debian.org/debian-gis-team/osmium-tool/
Vcs-Git: https://salsa.debian.org/debian-gis-team/osmium-tool.git -b buster-backports
Homepage: https://osmcode.org/osmium-tool/
......
Description: Fix spelling errors.
* wih -> with
Author: Bas Couwenberg <sebastic@debian.org>
Forwarded: https://github.com/osmcode/osmium-tool/pull/175
--- a/man/osmium-fileinfo.md
+++ b/man/osmium-fileinfo.md
@@ -99,7 +99,7 @@ The following variables are available:
data.bbox - BOX
(in JSON as nested ARRAY with coordinates)
data.timestamp.first - STRING with TIMESTAMP
- data.timestamp.last - STRING wih TIMESTAMP
+ data.timestamp.last - STRING with TIMESTAMP
data.objects_ordered - BOOL (yes|no)
data.multiple_versions - STRING (yes|no|unknown)
(in JSON as BOOL and missing if "unknown")
......@@ -9,6 +9,8 @@
"user": false,
"way_nodes": false
},
"format_options": {
},
"linear_tags": true,
"area_tags": true,
"exclude_tags": [],
......
......@@ -93,7 +93,8 @@ files created with JOSM).
-r, \--omit-rs
: Do not print the RS (0x1e, record separator) character when using the
GeoJSON Text Sequence Format. Ignored for other formats.
GeoJSON Text Sequence Format. Ignored for other formats. THIS OPTION
IS DEPRECATED, PLEASE USE "-x print_record_separator=false" INSTEAD.
-u, \--add-unique-id=TYPE
: Add a unique ID to each feature. TYPE can be either *counter* in which
......@@ -107,6 +108,13 @@ files created with JOSM).
In spaten exports the ID is written into the @fid field. For *counter* the
value will be an integer, for *type_id* it will be a string.
-x, \--format-option=OPTION(=VALUE)
: Set an output format option. The options available depend on the output
format. See the **OUTPUT FORMAT OPTIONS** section for available options.
If the VALUE is not set, the OPTION will be set to "true". If needed
you can specify this option multiple times to set several options. Options
set on the command line overwrite options set in the config file.
@MAN_COMMON_OPTIONS@
@MAN_PROGRESS_OPTIONS@
@MAN_INPUT_OPTIONS@
......@@ -136,6 +144,10 @@ the following optional names:
* `attributes`: An object specifying which attributes of OSM objects to export.
See the ATTRIBUTES section.
* `format_options`: An object specifying output format options. The options
available depend on the output format. See the **OUTPUT FORMAT OPTIONS**
section for available options. These options can also be set using the
command line option **\--format-option/-x**.
* `linear_tags`: An expression specifying tags that should be treated
as linear tags. See below for details and also look at the AREA HANDLING
section.
......@@ -304,6 +316,15 @@ The following output formats are supported:
at the moment. THE FORMAT MIGHT CHANGE WITHOUT NOTICE!
# OUTPUT FORMAT OPTIONS
* `print_record_separator` (default: `true`). Set to `false` to not print the
RS (0x1e, record separator) character when using the GeoJSON Text Sequence
Format. Ignored for other formats.
* `tags_format` (default: `jsonb`). Set to `hstore` to use HSTORE format
instead of JSON/JSONB when using the Pg Format. Ignored in other formats.
# DIAGNOSTICS
**osmium export** exits with exit code
......
......@@ -99,7 +99,7 @@ The following variables are available:
data.bbox - BOX
(in JSON as nested ARRAY with coordinates)
data.timestamp.first - STRING with TIMESTAMP
data.timestamp.last - STRING wih TIMESTAMP
data.timestamp.last - STRING with TIMESTAMP
data.objects_ordered - BOOL (yes|no)
data.multiple_versions - STRING (yes|no|unknown)
(in JSON as BOOL and missing if "unknown")
......
......@@ -75,7 +75,7 @@ static std::string get_attr_string(const rapidjson::Value& object, const char* k
return "";
}
void CommandExport::parse_options(const rapidjson::Value& attributes) {
void CommandExport::parse_attributes(const rapidjson::Value& attributes) {
if (!attributes.IsObject()) {
throw config_error{"'attributes' member must be an object."};
}
......@@ -90,6 +90,38 @@ void CommandExport::parse_options(const rapidjson::Value& attributes) {
m_options.way_nodes = get_attr_string(attributes, "way_nodes");
}
void CommandExport::parse_format_options(const rapidjson::Value& options) {
if (!options.IsObject()) {
throw config_error{"'format_options' member must be an object."};
}
for (const auto& kv : options.GetObject()) {
const auto type = kv.value.GetType();
const char* key = kv.name.GetString();
switch (type) {
case rapidjson::kNullType:
m_options.format_options.set(key, false);
break;
case rapidjson::kTrueType:
m_options.format_options.set(key, true);
break;
case rapidjson::kFalseType:
m_options.format_options.set(key, false);
break;
case rapidjson::kObjectType:
throw config_error{"Option value for key '" + std::string(key) + "' can not be of type object."};
case rapidjson::kArrayType:
throw config_error{"Option value for key '" + std::string(key) + "' can not be an array."};
break;
case rapidjson::kStringType:
m_options.format_options.set(key, kv.value.GetString());
break;
case rapidjson::kNumberType:
m_options.format_options.set(key, std::to_string(kv.value.GetInt64()));
break;
}
}
}
static Ruleset parse_tags_ruleset(const rapidjson::Value& object, const char* key) {
Ruleset ruleset;
......@@ -179,7 +211,12 @@ void CommandExport::parse_config_file() {
const auto json_attr = doc.FindMember("attributes");
if (json_attr != doc.MemberEnd()) {
parse_options(json_attr->value);
parse_attributes(json_attr->value);
}
const auto json_opts = doc.FindMember("format_options");
if (json_opts != doc.MemberEnd()) {
parse_format_options(json_opts->value);
}
m_linear_ruleset = parse_tags_ruleset(doc, "linear_tags");
......@@ -224,6 +261,7 @@ bool CommandExport::setup(const std::vector<std::string>& arguments) {
opts_cmd.add_options()
("add-unique-id,u", po::value<std::string>(), "Add unique id to each feature ('counter' or 'type_id')")
("config,c", po::value<std::string>(), "Config file")
("format-option,x", po::value<std::vector<std::string>>(), "Output format options")
("fsync", "Call fsync after writing file")
("geometry-types", po::value<std::string>(), "Geometry types that should be written (default: 'point,linestring,polygon')")
("index-type,i", po::value<std::string>()->default_value(default_index_type), "Index type to use")
......@@ -271,6 +309,8 @@ bool CommandExport::setup(const std::vector<std::string>& arguments) {
"user": false,
"way_nodes": false
},
"format_options": {
},
"linear_tags": true,
"area_tags": true,
"exclude_tags": [],
......@@ -292,6 +332,39 @@ bool CommandExport::setup(const std::vector<std::string>& arguments) {
setup_progress(vm);
setup_input_file(vm);
if (vm.count("output")) {
m_output_filename = vm["output"].as<std::string>();
const auto pos = m_output_filename.rfind('.');
if (pos != std::string::npos) {
m_output_format = m_output_filename.substr(pos + 1);
}
} else {
m_output_filename = "-";
}
if (vm.count("output-format")) {
m_output_format = vm["output-format"].as<std::string>();
}
canonicalize_output_format();
if (m_output_format != "geojson" &&
m_output_format != "geojsonseq" &&
m_output_format != "pg" &&
m_output_format != "text" &&
m_output_format != "spaten") {
throw argument_error{"Set output format with --output-format or -f to 'geojson', 'geojsonseq', 'pg', 'spaten', or 'text'."};
}
// Set defaults for output format options depending on output format
if (m_output_format == "geojsonseq") {
m_options.format_options.set("print_record_separator", true);
}
if (m_output_format == "pg") {
m_options.format_options.set("tags_type", "json");
}
if (vm.count("config")) {
m_config_file_name = vm["config"].as<std::string>();
......@@ -350,33 +423,19 @@ bool CommandExport::setup(const std::vector<std::string>& arguments) {
m_options.keep_untagged = true;
}
if (vm.count("output")) {
m_output_filename = vm["output"].as<std::string>();
const auto pos = m_output_filename.rfind('.');
if (pos != std::string::npos) {
m_output_format = m_output_filename.substr(pos + 1);
}
} else {
m_output_filename = "-";
}
if (vm.count("output-format")) {
m_output_format = vm["output-format"].as<std::string>();
if (vm.count("overwrite")) {
m_output_overwrite = osmium::io::overwrite::allow;
}
canonicalize_output_format();
if (m_output_format != "geojson" && m_output_format != "geojsonseq" && m_output_format != "pg" && m_output_format != "text" && m_output_format != "spaten") {
throw argument_error{"Set output format with --output-format or -f to 'geojson', 'geojsonseq', 'pg', 'spaten', or 'text'."};
if (vm.count("format-option")) {
for (const auto& str : vm["format-option"].as<std::vector<std::string>>()) {
m_options.format_options.set(str);
}
if (vm.count("overwrite")) {
m_output_overwrite = osmium::io::overwrite::allow;
}
if (vm.count("omit-rs")) {
m_options.print_record_separator = false;
m_options.format_options.set("print_record_separator", false);
warning("The --omit-rs/-r option is deprecated. Please use '-x print_record_separator=false' instead.\n");
if (m_output_format != "geojsonseq") {
warning("The --omit-rs/-r option only works for GeoJSON Text Sequence (geojsonseq) format. Ignored.\n");
}
......@@ -450,12 +509,7 @@ void CommandExport::show_arguments() {
m_vout << " output options:\n";
m_vout << " file name: " << m_output_filename << '\n';
if (m_output_format == "geojsonseq") {
m_vout << " file format: geojsonseq (with" << (m_options.print_record_separator ? " RS)\n" : "out RS)\n");
} else {
m_vout << " file format: " << m_output_format << '\n';
}
m_vout << " overwrite: " << yes_no(m_output_overwrite == osmium::io::overwrite::allow);
m_vout << " fsync: " << yes_no(m_fsync == osmium::io::fsync::yes);
m_vout << " attributes:\n";
......@@ -468,6 +522,13 @@ void CommandExport::show_arguments() {
m_vout << " user: " << (m_options.user.empty() ? "(omitted)" : m_options.user) << '\n';
m_vout << " way_nodes: " << (m_options.way_nodes.empty() ? "(omitted)" : m_options.way_nodes) << '\n';
if (m_options.format_options.size() > 0) {
m_vout << " output format options:\n";
for (const auto& option : m_options.format_options) {
m_vout << " " << option.first << " = " << option.second << '\n';
}
}
m_vout << " linear tags: ";
print_ruleset(m_vout, m_linear_ruleset);
m_vout << " area tags: ";
......@@ -512,6 +573,11 @@ static std::unique_ptr<ExportFormat> create_handler(const std::string& output_fo
}
bool CommandExport::run() {
auto handler = create_handler(m_output_format, m_output_filename, m_output_overwrite, m_fsync, m_options);
if (m_vout.verbose()) {
handler->debug_output(m_vout, m_output_filename);
}
osmium::area::Assembler::config_type assembler_config;
osmium::area::MultipolygonManager<osmium::area::Assembler> mp_manager{assembler_config};
......@@ -520,15 +586,9 @@ bool CommandExport::run() {
m_vout << "First pass done.\n";
m_vout << "Second pass (of two) through input file...\n";
m_linear_ruleset.init_filter();
m_area_ruleset.init_filter();
auto handler = create_handler(m_output_format, m_output_filename, m_output_overwrite, m_fsync, m_options);
if (m_vout.verbose()) {
handler->debug_output(m_vout, m_output_filename);
}
ExportHandler export_handler{std::move(handler), m_linear_ruleset, m_area_ruleset, m_geometry_types, m_show_errors, m_stop_on_error};
osmium::handler::CheckOrder check_order_handler;
......
......@@ -63,7 +63,8 @@ class CommandExport : public Command, public with_single_osm_input {
bool m_stop_on_error = false;
void canonicalize_output_format();
void parse_options(const rapidjson::Value& attributes);
void parse_attributes(const rapidjson::Value& attributes);
void parse_format_options(const rapidjson::Value& options);
void parse_config_file();
public:
......
......@@ -306,11 +306,11 @@ void CommandTagsFilter::find_nodes_in_ways() {
osmium::io::Reader reader{m_input_file, osmium::osm_entity_bits::way};
while (osmium::memory::Buffer buffer = reader.read()) {
for (const auto& way : buffer.select<osmium::Way>()) {
if (m_referenced_ids(osmium::item_type::way).get(way.positive_id())) {
add_nodes(way);
} else if (matches_way(way) != m_invert_match) {
if (matches_way(way) != m_invert_match) {
m_matching_ids(osmium::item_type::way).set(way.positive_id());
add_nodes(way);
} else if (m_referenced_ids(osmium::item_type::way).get(way.positive_id())) {
add_nodes(way);
}
}
}
......@@ -335,21 +335,25 @@ void CommandTagsFilter::find_referenced_objects() {
}
bool CommandTagsFilter::run() {
m_vout << "Opening input file to get header...\n";
osmium::io::Reader reader_only_for_header{m_input_file, osmium::osm_entity_bits::nothing};
m_vout << "Opening output file...\n";
osmium::io::Header header{reader_only_for_header.header()};
setup_header(header);
reader_only_for_header.close();
osmium::io::Writer writer{m_output_file, header, m_output_overwrite, m_fsync};
if (m_add_referenced_objects) {
find_referenced_objects();
}
m_vout << "Opening input file...\n";
++m_count_passes;
osmium::io::Reader reader{m_input_file, get_needed_types()};
m_vout << "Opening output file...\n";
osmium::io::Header header{reader.header()};
setup_header(header);
osmium::io::Writer writer{m_output_file, header, m_output_overwrite, m_fsync};
m_vout << "Copying matching objects to output file...\n";
++m_count_passes;
osmium::ProgressBar progress_bar{reader.file_size(), display_progress()};
while (osmium::memory::Buffer buffer = reader.read()) {
progress_bar.update(reader.offset());
......
......@@ -20,6 +20,7 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include "../exception.hpp"
#include "../util.hpp"
#include "export_format_json.hpp"
......@@ -43,7 +44,7 @@ ExportFormatJSON::ExportFormatJSON(const std::string& output_format,
m_fd(osmium::io::detail::open_for_writing(output_filename, overwrite)),
m_fsync(fsync),
m_text_sequence_format(output_format == "geojsonseq"),
m_with_record_separator(m_text_sequence_format && options.print_record_separator),
m_with_record_separator(m_text_sequence_format && options.format_options.is_true("print_record_separator")),
m_writer(m_stream),
m_factory(m_writer) {
m_stream.Reserve(initial_buffer_size);
......@@ -51,6 +52,13 @@ ExportFormatJSON::ExportFormatJSON(const std::string& output_format,
add_to_stream(m_stream, "{\"type\":\"FeatureCollection\",\"features\":[\n");
}
m_committed_size = m_stream.GetSize();
if (output_format == "geojsonseq") {
const auto prs = options.format_options.get("print_record_separator");
if (prs != "true" && prs != "false") {
throw config_error{"Unknown value for print_record_separator option: '" + prs + "'."};
}
}
}
void ExportFormatJSON::flush_to_output() {
......
......@@ -20,6 +20,7 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include "../exception.hpp"
#include "../util.hpp"
#include "export_format_pg.hpp"
......@@ -52,6 +53,15 @@ ExportFormatPg::ExportFormatPg(const std::string& /*output_format*/,
m_fd(osmium::io::detail::open_for_writing(output_filename, overwrite)),
m_fsync(fsync) {
m_buffer.reserve(initial_buffer_size);
const auto tt = options.format_options.get("tags_type");
if (tt == "hstore") {
m_tags_type = tags_output_format::hstore;
} else if (tt == "json" || tt == "jsonb") {
m_tags_type = tags_output_format::json;
} else {
throw config_error{"Unknown value for tags_format option: '" + tt + "'."};
}
}
void ExportFormatPg::flush_to_output() {
......@@ -154,7 +164,7 @@ void ExportFormatPg::add_attributes(const osmium::OSMObject& object) {
}
}
bool ExportFormatPg::add_tags(const osmium::OSMObject& object) {
bool ExportFormatPg::add_tags_json(const osmium::OSMObject& object) {
bool has_tags = false;
rapidjson::StringBuffer stream;
......@@ -175,6 +185,55 @@ bool ExportFormatPg::add_tags(const osmium::OSMObject& object) {
return has_tags;
}
static void add_escape_hstore(std::string& out, const char* str) {
out += "\"";
while (*str) {
if (*str == '"') {
out += "\\\"";
} else if (*str == '\\') {
out += "\\\\";
} else {
out += *str;
}
++str;
}
out += "\"";
}
bool ExportFormatPg::add_tags_hstore(const osmium::OSMObject& object) {
if (object.tags().empty()) {
return false;
}
bool has_tags = false;
std::string data;
for (const auto& tag : object.tags()) {
if (options().tags_filter(tag)) {
has_tags = true;
add_escape_hstore(data, tag.key());
data += "=>";
add_escape_hstore(data, tag.value());
data += ',';
}
}
if (has_tags) {
data.resize(data.size() - 1);
append_pg_escaped(data.c_str());
}
return has_tags;
}
bool ExportFormatPg::add_tags(const osmium::OSMObject& object) {
return m_tags_type == tags_output_format::json ? add_tags_json(object)
: add_tags_hstore(object);
}
void ExportFormatPg::finish_feature(const osmium::OSMObject& object) {
m_buffer += '\t';
add_attributes(object);
......@@ -223,7 +282,11 @@ void ExportFormatPg::close() {
void ExportFormatPg::debug_output(osmium::VerboseOutput& out, const std::string& filename) {
out << '\n';
out << "Create table with something like this:\n";
if (m_tags_type == tags_output_format::hstore) {
out << "CREATE EXTENSION IF NOT EXISTS hstore;\n";
}
out << "CREATE TABLE osmdata (\n";
if (options().unique_id == unique_id_type::counter) {
......@@ -266,7 +329,14 @@ void ExportFormatPg::debug_output(osmium::VerboseOutput& out, const std::string&
out << " way_nodes BIGINT[],\n";
}
switch (m_tags_type) {
case tags_output_format::json:
out << " tags JSONB -- or JSON, or TEXT\n";
break;
case tags_output_format::hstore:
out << " tags hstore\n";
break;
}
out << ");\n";
out << "Then load data with something like this:\n";
out << "\\copy osmdata FROM '" << filename << "'\n";
......
......@@ -33,16 +33,25 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
class ExportFormatPg : public ExportFormat {
enum tags_output_format {
json,
hstore
};
osmium::geom::WKBFactory<> m_factory{osmium::geom::wkb_type::ewkb, osmium::geom::out_type::hex};
std::string m_buffer;
std::size_t m_commit_size = 0;
int m_fd;
osmium::io::fsync m_fsync;
tags_output_format m_tags_type = tags_output_format::json;
void flush_to_output();
void start_feature(char type, osmium::object_id_type id);
void add_attributes(const osmium::OSMObject& object);
bool add_tags_json(const osmium::OSMObject& object);
bool add_tags_hstore(const osmium::OSMObject& object);
bool add_tags(const osmium::OSMObject& object);
void finish_feature(const osmium::OSMObject& object);
void append_pg_escaped(const char* str, std::size_t size);
......
......@@ -24,6 +24,7 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
#include <osmium/tags/tags_filter.hpp>
#include <osmium/util/options.hpp>
#include <string>
......@@ -46,8 +47,9 @@ struct options_type {
unique_id_type unique_id = unique_id_type::none;
osmium::Options format_options;
bool keep_untagged = false;
bool print_record_separator = true;
};
struct geometry_types {
......
......@@ -11,7 +11,7 @@ function(check_export _name _options _input _output)
endfunction()
check_export(geojson "-f geojson" input.osm output.geojson)
check_export(geojsonseq "-f geojsonseq -r" input.osm output.geojsonseq)
check_export(geojsonseq "-f geojsonseq -x print_record_separator=false" input.osm output.geojsonseq)
check_export(spaten "-f spaten" input.osm output.spaten)
check_export(missing-node "-f geojson" input-missing-node.osm output-missing-node.geojson)
......