libosmium

Added

• New RelationsManager class superseeds the relations::Collector class. The new class is much more modular and easier to extend. If you are using the Collector class, you are encouraged to switch.
• New MultipolygonManager based on the RelationsManager class superseeds the MultipolygonCollector class. The examples have been changed to use the new class and all users are encouraged to switch. There is also a MultipolygonManagerLegacy class if you still need old-style multipolygon support (see below).
• New FlexMem index class that works with input files of any size and stores the index in memory. This should now be used as the default index for node location stores. Several example programs now use this index.
• New CallbackBuffer class, basically a convenient wrapper around the Buffer class with an additional callback function that is called whenever the buffer is full.
• Introduce new ItemStash class for storing OSM objects in memory.
• New osmium::geom::overlaps() function to check if two Box objects overlap.
• Add function IdSet::used_memory() to get estimate of memory used in the set.
• New is_defined() and is_undefined() methods on Location class.
• Tests for all provided example programs. (Some tests currently fail on Windows for the osmium_index_lookup program.)

Changed

• The area Assembler now doesn't work with old-style multipolygons (those are multipolygon relations with the tags on the outer ways(s) instead of on the relation) any more. Because old-style multipolygons are now (mostly) gone from the OSM database this is usually what you want. The new AssemblerLegacy class can be used if you actually need support for old-style multipolygons, for instance if you are working with historical data. (In that case you also need to use the MultipolygonManagerLegacy class instead of the MultipolygonManager class.)
• Changes for consistent ordering of OSM data: OSM data can come in any order, but usual OSM files are ordered by type, ID, and version. These changes extend this ordering to negative IDs which are sometimes used for objects that have not been uploaded to the OSM server yet. The negative IDs are ordered now before the positive ones, both in order of their absolute value. This is the same ordering as JOSM uses.
• Multipolygon assembler now checks for three or more overlapping segments which are always an error and can report them.
• Enable use of user-provided thread::Pool instances in Reader and Writer for special use cases.
• Growing a Buffer will now work with any capacity parameter, it is always rounded up for proper alignment. Buffer constructor with three arguments will now check that commmitted is not larger than capacity.
• Updated embedded protozero to 1.5.2.
• Update version of Catch unit test framework to 1.9.7.
• And, as always, lots of small code cleanups and more tests.

Fixed

• Buffers larger than 2^32 bytes do now work.
• Output coordinate with value of -2^31 correctly.
• Changeset comments with more than 2^16 characters are now allowed. The new maximum size is 2^32.
• ChangesetDiscussionBuilder::add_comment_text() could fail silently instead of throwing an exception.
• Changeset bounding boxes are now always output to OSM files (any format) if at least one of the corners is defined. This is needed to handle broken data from the main OSM database which contains such cases. The OPL reader has also been fixed to handle this case.
• In the example osmium_location_cache_create, the index file written is always truncated first.

Fixed

• Two bugs in area assembler affecting very complex multipolygons and multipolygons with overlapping or nearly overlapping lines.
• Invalid use of iterators leading to undefined behaviour in area assembler code.
• Read OPL file correctly even if trailing newline in file is missing.

Added

• Add two argument (key, value) overload of TagMatcher::operator().

Changed

• Detect, report, and remove duplicate ways in multipolygon relations.
• Change EOF behaviour of Reader: The Reader::read() function will now always return an invalid buffer exactly once to signal EOF.
• Update QGIS multipolygon project that is part of the test suite to show more problem types.
• Copy multipolygon QGIS file for tests to build dir in cmake step.
• Some code cleanups and improved debug output in multipolygon code.
• Refactor I/O code to simplify code.
• Disable some warnings on MSVC.
• Various small code and build script changes.

Fixed

• Two bugs in area assembler affecting very complex multipolygons and multipolygons with overlapping or nearly overlapping lines.
• Invalid use of iterators leading to undefined behaviour in area assembler code.
• Area assembler stats were not correctly counting inner rings that are areas in their own right.
• Fix a thread problem valgrind found that might or might not be real.
• Read OPL file correctly even if trailing newline in file is missing.
• Include order for osmium/index/map headers and osmium/index/node_locations_map.hpp (or osmium/handler/node_locations_for_ways.hpp) doesn't matter any more.

Fixed

• Use minimum size of 64 bytes for buffers. This fixes an infinite loop
  when buffer size is zero.

Added

• New TagsFilter::set_default_result() function.

Changed

• Use larger capacity for Buffer if necessary for alignment instead of
  throwing an exception. Minimum buffer size is now 64 bytes.
• Check order of input data in relations collector. The relations collector
  can not deal with history data or a changes file. This was documented as a
  requirement, but often lead to problems, because this was ignored by users.
  So it now checks that the input data it gets is ordered and throws an
  exception otherwise.
• When writing an OSM file, set generator to libosmium if not set by app.

Fixed

• Infinite loop in Buffer::reserve_space(). (Issue #202.)
• ObjectPointerCollection::unique() now removes elements at end.
• Tests comparing double using == operator.
• Build on Cygwin.

Fixed

• Terminate called on full non-auto-growing buffer. (Issue #189.)
• When file formats were used that were not compiled into the binary, it
  terminated instead of throwing. (Issue #197.)
• The Filter::count() method didn't compile at all.

Added

• TagMatcher and TagsFilter classes for more flexibly matching tags and
  selecting objects based on tags. This obsoletes the less flexible classes
  based on osmium::tags::Filter classes.
• Extended index::RelationsMap(Stash|Index) classes to also allow
  parent-to-member lookups.
• New nrw_array helper class.
• ObjectPointerCollection::unique() function.

Changed

• Area assembler can now detect invalid locations and report them in the
  stats and through the problem reporter. If the new config option
  ignore_invalid_locations is set, the Assembler will pretend they weren't
  even referenced in the ways. (Issue #195.)
• osmium::area::Assembler::operator() will now return a boolean reporting
  whether building of the area(s) was successful.
• Split up area Assembler class into three classes: The
  detail::BasicAssembler is now the parent class. Assembler is the child
  class for usual use. The new GeomAssembler also derives from
  BasicAssembler and builds areas without taking tags into account at all.
  This is to support osm2pgsql which does tag handling itself. (Issue #194.)
• The Projection class can do any projection supported by the Proj.4
  library. As a special case it now uses our own Mercator projection
  functions when the web mercator projection (EPSG 3857) is used. This is
  much faster than going through Proj.4.
• Better error messages for low-level file utility functions.
• Mark build_tag_list* functions in builder_helper.hpp as deprecated. You
  should use the functions from osmium/builder/attr.hpp instead.
• Improved performance of the osmium::tags::match_(any|all|none)_of
  functions.
• Improved performance of string comparison in tags::Filter.
• Update version of Catch unit test framework to 1.8.1. This meant some
  tests had to be updated.
• Use get_noexcept() in NodeLocationsForWays handler.
• And lots of code and test cleanups...

Fixed

• Terminate called on full non-auto-growing buffer. (Issue #189.)
• When file formats were used that were not compiled into the binary, it
  terminated instead of throwing. (Issue #197.)
• Windows build problem related to including two different winsock versions.
• Windows build problem related to forced build for old Windows versions.
  (Issue #196.)
• Clear stream contents in ProblemReporterException correctly.
• Add -pthread compiler and linker options on Linux/OSX. This should fix
  a problem where some linker versions will not link binaries correctly when
  the --as-needed option is used.
• The Filter::count() method didn't compile at all.
• XML reader doesn't fail on relation member ref=0 any more.

Added

• New index::RelationsMap(Stash|Index) classes implementing an index for
  looking up parent relation IDs given a member relation ID.
• Add get_noexcept() method to all index maps. For cases where ids are
  often not in the index using this can speed up a program considerably.
• New non-const WayNodeList::operator[].
• Default constructed "invalid" Coordinates.
• Tile constructor from web mercator coordinates and some helper
  functions for tile arithmetic.
• Tag matcher matching keys using a regex.
• New envelope() functions on NodeRefList, Way, and Area returning a
  Box object with the geometric envelope of the object.
• Add amenity_list example.

Changed

• Replaced the implementation for the web mercator projection using the usual
  tan-formula with a polynomial approximation which is much faster and good
  enough for OSM data which only has ~1cm resolution anyway. See
  https://github.com/osmcode/mercator-projection for all the details and
  benchmarks. You can disable this by defining the macro
  OSMIUM_USE_SLOW_MERCATOR_PROJECTION before including any of the Osmium
  headers.
• Removed the outdated Makefile. Always use CMake directly to build.
• Refactoring of osmium::apply() removing the resursive templates for faster
  compile times and allowing rvalue handlers.
• Lots of code and test cleanups and more documentation.

Fixed

• Handle endianess on FreeBSD properly.
• Fixed doxygen config for reproducible builds.

Changed

• Round out ObjectPointerCollection implementation and test it.
• Updated embedded protozero to 1.4.5.

Changed

• Updated embedded protozero to 1.4.4.

Fixed

• Buffer overflow in osmium::Buffer.

Changed

• Updated embedded protozero to 1.4.3.

Fixed

• Made IdSet work on 32bit systems.
• Fixed endianness check for WKB tests.

Added

• The Reader can take an additional optional read_meta flag. If this is
  set to false the PBF input will ignore metadata on OSM objects (like version,
  timestamp, uid, ...) which speeds up file reading by 10 to 20%.
• New IdSet virtual class with two implementations: IdSetDense and
  IdSetSmall. Used to efficiently store a set of Ids. This is often needed
  to track, for instance, which nodes are needed for ways, etc.
• Added more examples and better documented existing examples.
• Add a benchmark "mercator" converting all node locations in a file to
  WebMercator and creating geometries in WKB format.

Changed

• Better queue handling makes I/O faster in some circumstances.
• The FindOsmium.cmake CMake script can now check a current enough libosmium
  version is found.
• Builders can now be constructed with a reference to parent builder.
• Made builders more robust by adding asserts that will catch common usage
  problems.
• Calling OSMObjectBuilder::add_user() is now optional, and the method was
  renamed to set_user(). (add_user() is marked as deprecated.)
• Benchmarks now show compiler and compiler options used.
• Builder::add_item() now takes a reference instead of pointer (old version
  of the function marked as deprecated).
• GEOS support is deprecated. It does not work any more for GEOS 3.6 or newer.
  Reason is the changed interface in GEOS 3.6. If there is interest for the
  GEOS support, we can add support back in later (but probably using the
  GEOS C API which is more stable than the C++ API). Some tests using GEOS
  were rewritten to work without it.
• The BoolVector has been deprecated in favour of the new IdSet classes.
• Lots of code cleanups and improved API documentation in many places.
• The relations collector can now tell you whether a relation member was in
  the input data. See the new is_available() and
  get_availability_and_offset() methods.
• Updated embedded Catch unit test header to version 1.5.8.

Fixed

• Parsing of coordinates starting with decimal dot and coordinates in
  scientific notation.
• ~ operator for entity_bits doesn't set unused bits any more.
• Progress bar can now be (temporarily) removed, to allow other output.

Added

• Support for reading OPL files.
• For diff output OSM objects in buffers can be marked as only in one or the
  other file. The OPL and debug output formats support diff output based on
  this.
• Add documentation and range checks to Tile struct.
• More documentation.
• More examples and more extensive comments on examples.
• Support for a progress report in osmium::io::Reader() and a ProgressBar
  utility class to use it.
• New OSMObject::set_timestamp(const char*) function.

Changed

• Parse coordinates in scientific notations ourselves.
• Updated included protozero version to 1.4.2.
• Lots of one-argument constructors are now explicit.
• Timestamp parser now uses our own implementation instead of strptime.
  This is faster and independant of locale settings.
• More cases of invalid areas with duplicate segments are reported as
  errors.

Fixed

• Fixed a problem limiting cache file sizes on Windows to 32 bit.
• Fixed includes.
• Exception messages for invalid areas do not report "area contains no rings"
  any more, but "invalid area".

Added

• EWKT support.
• Track pop type calls and queue underruns when OSMIUM_DEBUG_QUEUE_SIZE
  environment variable is set.

Changed

• Switched to newest protozero v1.4.0. This should deliver some speedups
  when parsing PBF files. This also removes the DeltaEncodeIterator class,
  which isn't needed any more.
• Uses std::unordered_map instead of std::map in PBF string table code
  speeding up writing of PBF files considerably.
• Uses less memory when writing PBF files (smaller string table by default).
• Removes dependency on sparsehash and boost program options libraries for
  examples.
• Cleaned up threaded queue code.

Fixed

• A potentially very bad bug was fixed: When there are many and/or long strings
  in tag keys and values and/or user names and/or relation roles, the string
  table inside a PBF block would overflow. I have never seen this happen for
  normal OSM data, but that doesn't mean it can't happen. The result is that
  the strings will all be mixed up, keys for values, values for user names or
  whatever.
• Automatically set correct SRID when creating WKB and GEOS geometries.
  Note that this changes the behaviour of libosmium when creating GEOS
  geometries. Before we created them with -1 as SRID unless set otherwise.
  Manual setting of the SRID on the GEOSGeometryFactory is now deprecated.
• Allow coordinates of nodes in scientific notation when reading XML files.
  This shouldn't be used really, but sometimes you can find them.

Changed

• Much faster output of OSM files in XML, OPL, or debug formats.

Fixed

• Parsing and output of coordinates now faster and always uses decimal dot
  independant of locale setting.
• Do not output empty discussion elements in changeset XML output.
• Data corruption regression in mmap based indexes.

New release because I forgot to update the version number in version.hpp.

Added

• New functions for iterating over specific item types in buffers
  (osmium::memory::Buffer::select()), over specific subitems
  (osmium::OSMObject::subitems()), and for iterating over all rings of
  an area (osmium::Areas::outer_rings(), inner_rings()).
• Debug output optionally prints CRC32 when add_crc32 file option is set.

Changed

• XML parser will not allow any XML entities which are usually not used in OSM
  files anyway. This can help avoiding DOS attacks.
• Removed SortedQueue implementation which was never used.
• Also incorporate Locations in NodeRefs into CRC32 checksums. This means
  all checksums will be different compared to earlier versions of libosmium.
• The completely new algorithm for assembling multipolygons is much faster,
  has better error reporting, generates statistics and can build more complex
  multipolygons correctly. The ProblemReporter classes have changed to make
  this happen, if you have written your own, you have to fix it.
• Sparse node location stores are now only sorted if needed, ie. when nodes
  come in unordered.

Fixed

• Output operator for Location shows full precision.
• Undefined behaviour in WKB writer and types_from_string() function.
• Fix unsigned overflow in pool.hpp.
• OSM objects are now ordered by type (nodes, then ways, then relations),
  then ID, then version, then timestamp. Ordering by timestamp is normally
  not necessary, because there can't be two objects with same type, ID, and
  version but different timestamp. But this can happen when diffs are
  created from OSM extracts, so we check for this here. This change also
  makes sure IDs are always ordered by absolute IDs, positives first, so
  order is 0, 1, -1, 2, -2, ...
• Data corruption bug fixed in disk based indexes (used for the node
  location store for instance). This only affected you, if you created
  and index, closed it, and re-opened it (possibly in a different process)
  and if there were missing nodes. If you looked up those nodes, you got
  location (0,0) back instead of an error.
• Memory corruption bug showing up with GDAL 2.

Added

• Add WITH_PROFILING option to CMake config. When enabled, this sets the
  -fno-omit-frame-pointer compiler option.

Changed

• Massive speed improvements when building multipolygons.
• Uses (and includes) new version 1.3.0 of protozero library.
• Removed dependency on Boost Iterator for PBF writer.
• Example program osmium_area_test now uses cerr instead of cout for
  debug output.

Added

• The new handler osmium::handler::CheckOrder can be used to check that a
  file is properly ordered.
• Add new method to build OSM nodes, ways, relations, changesets, and areas
  in buffers that wraps the older Builder classes. The new code is much easier
  to use and very flexible. There is no documentation yet, but the tests in
  test/t/builder/test_attr.cpp can give you an idea how it works.
• Add util class to get memory usage of current process on Linux.

Changed

• New Buffer memory management speeds up Buffer use, because it doesn't clear
  the memory unnecessarily.

Fixed

• osmium::Box::extend() function now ignores invalid locations.
• Install of external library headers.
• Check way has at least one node before calling is_closed() in area
  assembler.
• Declaration/definition of some friend functions was in the wrong namespace.