3.8.0beta1

This is the first beta release for the 3.8.x series.

Changes compared to 3.7.x:
Note, that for beta release we generate the changes with
a tool (may contain false information). Final changelog will be more sophisticated (and will
include Credits section).

Implemented enhancements:

• Support an alternative build system: CMake #966
• SCL for Logmatic format and destination #799
• SCL for Loggly format and destination #798
• support multiple drivers with the same name in syslog-ng config #661
• HTTP destination driver in Java #539
• HTTP destination driver in Python #534
• F/unset value #1108 (bazsi)
• F/elasticsearch v2 mode http #1053 (lbudai)
• Add $(sum), $(min) and $(max) template functions #1037 (MrAnno)
• Add ability to use templates in both url and message format #1033 (avcbvamorec)
• F/libmongo client compatibility over mongo c driver #981 (bkil-syslogng)
• Improve "curl" module #978 (litterbear)
• Prepare OS X support #953 (MrAnno)
• Add Elasticsearch 2 destination with Shield support #912 (lbudai)
• Use official MongoDB C Driver instead of libmongo-client #891 (bkil-syslogng)
• Support native Elasticsearch configuration for transport mode #890 (lbudai)
• Set 0.11.0 as the minimal required version of hiredis to avoid possible deadlocks #887 (ihrwein)
• Add inherit-environment() option to program driver #861 (MrAnno)
• Remove fix relative path of syslog-ng in func test #858 (bazsi)
• Add support of Kafka 0.9.0.0 #856 (ihrwein)
• Log HTTP response error codes in HTTP destination #855 (MrAnno)
• Improve the performance of value-pairs #851 (bazsi)
• Format CEF extension #842 (bkil-syslogng)
• Implement serialization of log messages #834 (juhaszviktor)
• scl: add logmatic() destination #812 (bazsi)
• F/scl varargs refined #699 (ihrwein)
• F/unix socket source creates dir #632 (ihrwein)
• ... NEWS.md

Fixed bugs:

• The output of pdbtool is scrambled #1043
• 3.8 journal source problem #914
• Global option inheritance problem in afunix-source #894
• Deadlock in redis destination #792
• Deadlock with suppress option #781
• tests/unit/test_zone fails on Unix epoch #726
• Every second config reload kills marking #701
• Runs in a different $CWD when foregrounding via "-F" #700
• Segfault on TLS errors #695
• Compile error related to python module #674
• syslog-ng is stuck in an infinite loop of setsockopt() returning ENOTSOCK #670
• syslog-ng 3.6 may kill init process #586
• message formatting on remote destinations did not follow the switch to legacy from IETF syslog format #570
• Missing mark message on TCP destination in case of mark_mode(dst_idle) #547
• Cannot write filter plugins #427
• ... NEWS.md

Unofficial Debian packages:

• https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng

3.7.3

Changes compared to 3.7.2:

Improvements

• Updated Python package requirements.
• Can now compile without MongoDB.
• Added eventlog to the list of required pkg-config packages.
• Basic FreeBSD and HP-UX support of syslog debug bundle generator by
  improving POSIX shell compatibility.
• Keep the program destination open between configuration reloads.
• system-source now uses keep-timestamp(no) for Linux kernel log.
  The time source used by /dev/kmsg is not updated after system
  SUSPEND/RESUME.

Fixes

• Fix a SIGSEGV when a Redis command returns an error.
• Resolve deadlock in logwriter triggered by suppress()
• Mitigate possible deadlock in patterndb
• Fixed global inheritance of pass-unix-credentials() and create-dirs().
• Certain compilers complained about an undefined symbol when setting
  keep-alive(yes).
• For certain use cases, afsocket would not handle procfs read errors due
  to an integer underflow.
• Enhanced Java version check and the handling of SyslogNgInternalLogger
  (used by Kafka), the FATAL loglevel and getLocationInformation().
• When a big amount of kernel log was produced in a very short time,
  the syslog-ng process sometimes entered into a spin and stop processing
  messages.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

András Mitzki, Avleen Vig, Balázs Scheidler, Ben Kibbey, Christian Herzig,
David Schweikert, Douglas Carmichael, Dezso Endre Molnar, Fabien Wernli,
Gergely Czuczy, Gergely Nagy, Gergo Nagy, Hanno Böck, Herzig, Christian,
Laszlo Budai, László Várady, MÓZES Ádám István, PÁSZTOR György, Péter Czanik,
Robert Fekete, Saurabh Shukla, Tamás Nagy, Tibor Benke, Viktor Juhász,
Vincent Bernat, Wang Long, Zdenek Styblik, Zoltán FRIED, Zoltán Pallagi

3.7.2

This is the first maintenance release for the 3.7.x series.

Changes compared to 3.7.1:

Improvements

• Added mbox() source.
  This source can be used to fetch emails from local mbox files:
  source { mbox("/var/spool/mail/root"); };
  This will fetch root emails and parse them into a multiline $MSG.
  Original implementation by Fabien Wernli, I only converted it into
  an SCL.
• It is possible to append dynamically options into SCL blocks from now.
• concurrent_request option added to ElasticSearch with default value 1.
• In elasticsearch destinaton, message_template() argument renamed to
  template().
• SCL added to every Java module (ElasticSearch, Kafka, HDFS).
• Linux Audit Parser added for parsing key-value pairs produced by
  the Linux Audit subsystem.
• HTTP destination is now able to receive HTTP method as an option.
  All the supported methods are available
  (POST, PUT, HEAD, OPTIONS, DELETE, TRACE, GET).

Fixes

• In some circumstances syslog-ng mod-journal re-read every already
  processed messages.
• When syslog-ng got a reload and the reload process done within 1 second then
  mafter the reload, syslog-ng stop generating mark-messages.
• When initialization of a network destination in syslog-ng failed (eg. due to
  DNS resolution failure) we didn't create a queue which caused message loss.
• syslog-ng segfaulted on TLS errors when wrong certs was provided
  (eg.: CA cert with the cert-file directive instead of the server cert).
• Fixed a continuous spinning case in the file driver, when the
  destination file is a device (e.g. /dev/stdout).
• A memory leak in around template functions in grammar fixed.
• Fixed Python3 support.
• Fixed Python GIL issue in python destination.
• From now, instead of skipping doc/ alltogether when ENABLE_MANPAGES is
  not set, only skip the actual man pages, but handle the rest properly.
• Allow overriding the python setup.py options.

When installing the python modules, allow overriding the options. This
is useful for distributions that want to pass extra options. For
example, on Debian, we want --install-layout="deb" instead of the
--prefix and --root options.

With this change, the previous behaviour remains the default, but one
can supply PYSETUP_OPTIONS on the make command-line to override it.

• The systemd service file read /etc/default/syslog-ng and /etc/sysconfig/syslog-ng,
  but didn't do anything with their contents. $SYSLOGNG_OPTS added to ExecStart, so
  that the EnvironmentFiles have an effect (at least on Debian).
• Java support checking fixed (not only jdk is required but also gradle).
• Memory leak around ping() in Redis fixed.
• A crash in pdbtool fixed around r_parser_email().
• Removed cygwin fdlimit statement.
  Make the default for RLIMIT_NOFILE equal to the current system limits.
  --fd-limit can still override this, but the default will be configured
  based on existing system limits.
• Fixed BSD year inference.
  Fixed logic and made clearer the inference of year from bsd-style
  rfc3164 syslog-messages, which do not include a year.
• Handle correctly the epoch 0 timestamp.
  (Previously, syslog-ng cached the zero timestamp and treated 1970 as it was
  1900.)

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Adam Arsenault, Adam Istvan Mozes, Andras Mitzki, Avleen Vig,
Balazs Scheidler, Fabien Wernli, Gergely Czuczy, Gergely Nagy, Gergo Nagy,
Laszlo Budai, Peter Czanik, Robert Fekete, Saurabh Shukla, Tamas Nagy,
Tibor Benke, Viktor Juhasz, Vincent Bernat, Wang Long, Zdenek Styblik,
Zoltan Pallagi.

3.7.1

New dependencies

OpenSSL is now a required dependency for syslog-ng because the newly added
hostid and uniqid features requires a CPRNG provided by OpenSSL.
Therefore non-embedded crypto lib is not a real option, so the support of
having such a crypto lib discontinued and all SSL-dependent features enabled
by default.

Library updates

• Minimal libriemann-client version bumped from 1.0.0 to 1.6.0.
• Added support for the monolithic libsystemd library (systemd 209).
• RabbitMQ submodule upgraded.

Features

Language bindings

• Java-destination driver ported from syslog-ng-incubator.
  Purpose of having Java destination driver is to make it possible
  to implement destination drivers in the Java language (and using
  'official' Java client libraries).
• Python language support is ported from syslog-ng incubator and
  has been completely reworked. Now, it is possible to implement template
  functions in Python language and also destination drivers.
  Main purpose of supporting Python language is to implement a nice
  interactive syslog-ng config debugger for syslog-ng.

New drivers

New Java destination drivers

ElastiSearch, Kafka and HDFS destination drivers are implemented by using
the 'official' Java client libraries and syslog-ng provides a way to set
their own, native configuration file. Log messages generated by the client
Java libraries are redirected to syslog-ng via our own Log4JAppender which
means that those logs are available as internal syslog-ng messages.

• ElasticSearch
• Kafka
• Hadoop/HDFS
• HTTP

Parsers

• Added a geoip() parser, that can look up the country code and
  latitude/longitude information from an IPv4 address. For lat/long to
  work, one will need the City database.
• New parser, extract-solaris-msgid() added for automatically extracts
  (parses & removes) the msgid portion of Solaris messages.
• Extended the set of supported characters to every printable ASCII's except
  ., [ and ] in extract-prefix for json-parser().
• Added string-delimiters option to csvparser to support multi character
  delimiters in CSV parsing.
• A kv-parser() introduced for WELF (WebTrens Enhanced Log Format) that
  implements key=value parsing. The kv-parser() tries to extract
  key=value formatted name-value pairs from the input string.
• value-pairs: make it possible to pass --key as a positional argument
  From now it is possible to use value-pairs expressions like this:
  $(format-json MSG DATE)
  instead of
  $(format-json --key MSG --key DATE)

Filters

• Added IPv6 netmask filter for selecting only messages sent by a host whose
  IP address belongs to the specified IPv6 subnet.

Macros

• Added a new macro, called HOSTID which is a 32-bit number generated by
  a cryptographically secure PRNG. Its purpose is to identify the
  syslog-ng host, thus it is the same for every message generated on the same
  host.
• Added a new macro, called UNIQID which is a practically unique ID generated
  from the HOSTID and the RCPTID in the format of HOSTID@RCPTID.
  Uniqid is a derived value: it is built up from the always available hostid
  and the optional rcptid. In other words: uniqid is an extension over rcptid.
  For that reason use-rcptid has been deprecated and use-uniqid could be
  use instead.

Templates

• welf was renamed to kvformat
  As this reflects the purpose of this module much better, WELF is just
  one of the format it has support for.
• $(format-cim) template function added into an SCL module.
• It is possible to create templates without braces.

SMTP destination

• The afsmtp driver now supports templatable recipients field.
  Just like the subject() and body() fields, now the address containing
  parameters of to(), from(), cc() and bcc() can contain macros.

Unix Domain Sockets

• Added pass-unix-credentials() global option for enabling/disabling unix
  credentials passing on those platforms which has this feature. By default
  it is enabled.
• Added create-dirs() option to unix-*() sources for creating the
  containing directories for Unix domain sockets.

Riemann destination

• Added batched event sending support for riemann destination driver which
  makes the riemann destination respect flush-lines(), and send event
  in batches of configurable amount (defaults to 1). In case of an error,
  all messages within the batch will be dropped. Dropped messages, and
  messages that result in formatting errors do not count towards the batch
  size. There is no timeout, but messages will be flushed upon deinit.
• A timeout() option added to the Riemann destination.

PatternDB

• Earlier, in patterndb, the first applicable rule won, even if it was
  only a partial match. This means that when rules overlapped, the shorter
  match would have been found, if it was the first to be loaded.
  A strong preference introduced for rules that match the input string
  completely. The load order is still applicable though, it is possible to
  create two distinct rules that would match the same input, in those cases
  the first one to be loaded wins.

Miscellaneous features

• New builtin interactive syslog-ng.conf debugger implemented for syslog-ng.
  The debugger has a Python frontend which contains a full Completer
  (just press TABs and works like bash)
• Added a reset option to syslog-ng-ctl stats. With this option the non-stored
  stats counters can be zeroed.
• New parameter added to loggen: --permanent (-T) wich is for sending logs
  indefinitely.
• Loggen uses the proper timezone offset in generated message.
• The ssl_options inside tls() extended with the following set:
  no-sslv2, no-sslv3, no-tlsv1, no-tlsv11, no-tlsv12.
• Added syslog-debug bundle generator script to make it easier to reproduce bugs
  by collecting debug related information, like:
  • process information gathering
  • syscall tracing (strace/truss)
  • configuration gathering
  • selinux related information gathering
  • solaris information gathering (sysdef, kstat, showrev, release)
  • get information about syslog-ng svr4 solaris packages, if possible

Bugfixes

• New utf8 string sanitizers instead of old broken one.

• syslog-ng won't send SIGTERM when getpgid() fails in program destination
  (afprog).

• In some cases program destination respawned during syslog-ng stop/restart
  (afprog).

• syslog-ng generates mark messages when mark-mode is set
  to host-idle.

• Using msg_control only when credential passing is supported in socket
  destination (afsocket).

• Writer is replaced only when protocol changed during reload in socket
  destination (afsocket).

• Fix spinning on EOF for unix-stream() sockets. Root cause of the spinning
  was that a unix-dgram socket was created even in case of unix-stream.

• When the configured host was not available during the initialization of
  afsocket destination syslog-ng just didn't start. From now, syslog-ng
  starts in that case and will retry connecting to the host periodically.

• Fixed BSD year inference in syslogformat. When the difference between the
  current month and the month part of the timestamp of an incoming logmessage
  in BSD format (which has no year part) was greater than 1 then syslog-ng
  computed the year badly.

• In some cases, localtime related macros had a wrong value(eg.:$YEAR).

• TLS support added to Riemann destination

• Excluded "tags" from Riemann destination driver as an attribute which
  conflicts with reserved keyword

• When a not writeable/non-existent file becomes writeable/exists later,
  syslog-ng recognize it (with the help of reopen-timer) and delivers messages
  to the file without dropping those which were received while the file was
  not available (affile).

• Fixed a crash around affile at the first message delivery when templates
  were used (affile).

• Fixed a configure error around libsystemd-journal.

• Removed syslog.socket from service file on systems using systemd.
  Syslog-ng reads the messages directly from journal on systems with systemd.

• Fixed compilation where the monolitic libsystemd was not available.

• Fixed compilation failure on OpenBSD.

• AMQP connection process fixed.

• Added DOS/Windows line ending support in config.

• Retries fixed in SQL destination. In some circumstances when
  retry_sql_inserts was set to 1, after an insertion failure all incoming
  messages were dropped.

• Transaction handling fixed in SQL destination. In some circumstances when
  both select and insert commands were run within a single transaction and
  the select failed (eg.: in case of mssql), the log messages related to
  the insert commands, broken by the invalid transaction, were lost.

• Fixed a memleak in SQL destination driver.
  The memleak occured during one of the transaction failures.

• Memory leak around reload and internal queueing mechanism has been fixed.

• Fixed a potential abort when the localhost name cannot be detected.

• Security issue fixed around $HOST.
  Tech details:
  When the name of the host is too long, the buffer we use to format the
  chained hostname is truncated. However snprintf() returns the length the
  result would be if no truncation happened, thus we will read uninitialized
  bytes off the stack when we use that pointer to set $HOST
  with log_msg_set_value().

  There can be some security implications, like reading values from the stack
  that can help to craft further exploits, especially in the presense of
  address space randomization. It can also cause a DoS if the hostname length
  is soo large that we would read over the top-of-the-stack, which is probably
  not mmapped causing a SIGSEGV.

• Journal entries containing name-value pairs without '=' caused syslog-ng
  to crash. Instead of crashing, syslog-ng just drop these nv pairs.

• Fixed the encoding of characters below 32 if escaping is enabled in
  templates. Templated outputs never contained references to characters below
  32, essentially they were dropped from the output for two reasons:

  • the prefixing backslash was removed from the code
  • the format_uint32_padded() function produced no outputs in base 8

• Fixed afstomp destination port issue. It always tried to connect to the port 0.

• Fixed memleak in db-parser which could happen at every reload.

• Fixed a class of rule conflicts in db-parser:
  Because an error in the pdb load algorithms, some rules would conflict which
  shouldn't have done that. The problem was that several programs would use
  the same RADIX tree to store their patterns. Merging independent programs
  meant that if they the same pattern listed, it would clash, even though
  their $PROGRAM is different.

  There were multiple issues:

  • we looked up pattern string directly, even they might have contained
    @parser@ references. It was simply not designed that way and only
    worked as long as we didn't have the possibility to use parsers
    in program names
  • we could merge programs with the same prefix, e.g.
    su, supervise/syslog-ng and supervise/logindexd would clash, on "su",
    which is a common prefix for all three.

  The solution involved in using a separate hash table for loading, which
  at the end is turned into the radix tree.

• pdbtool match when used with the --debug-pattern option used a low-level
  lookup function, that didn't perform all the db-parser actions specified
  in the rule

• Max packet length for spoof source is set to 1024 (previously : 256).

• A certificate which is not contained by the list of fingerprints is
  rejected from now.

• Hostname check in tls certificate is case insensitive from now.

• There is a use-case where user wants to ignore an assignment to a name-value
  pair. (eg.: when using csv-parser(), sometimes we get a column we really
  want to drop instead of adding it to the message). In previous versions an
  error message was printed out:
  'Name-value pairs cannot have a zero-length name'.
  That error message has been removed.

• Fixed a docbook related compilation error: there was a hardcoded path that
  caused build to fail if docbook is not on that path. Debian based
  platforms did not affected by this problem.
  Now a new option was created for ./configure that is --enable-manpages
  that enables the generation of manpages using docbook from online source.
  '--with-docbook=PATH' gives you the opportunity to specify the path for
  your own installed docbook.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Adam Arsenault, Adam Istvan Mozes, Alex Badics, Andras Mitzki,
Balazs Scheidler, Bence Tamas Gedai, Ben Kibbey, Botond Borsits, Fabien Wernli,
Gergely Nagy, Gergo Nagy, Gyorgy Pasztor, Kristof Havasi, Laszlo Budai,
Manikandan-Selvaganesh, Michael Sterrett, Peter Czanik, Robert Fekete,
Sean Hussey, Tibor Benke, Toralf Förster, Viktor Juhasz, Viktor Tusa,
Vincent Bernat, Zdenek Styblik, Zoltan Fried, Zoltan Pallagi.

3.7.0beta2

This is the second beta release of the upcoming syslog-ng OSE 3.7
branch.

Changes compared to the previous alpha release:

Features

• Added a geoip parser.
• ssl_options inside tls() extended with the following set:
  no-sslv2, no-sslv3, no-tlsv1, no-tlsv11, no-tlsv12
• minimal libriemann-client version bumped from 1.0.0 to 1.6.0
• TLS support added to Riemann destination
• timeout() option added to Riemann destination

Fixes

• SyslogNg.jar removed from the release tarball.
• When the configured host was not available during the initialization of
  afsocket destination syslog-ng just didn't start. From now, syslog-ng
  starts in that case and will retry connecting to the host periodically.
• When a not writeable file becomes writeable later, syslog-ng recognize it
  (with the help of reopen-timer) and delivers messages to the file without
  dropping those which were received during the file was not available.
• Fixed a configure error around libsystemd-journal.
• --disable-python option and other Python related fixes addded to
  configure
• Retries fixed in SQL destination. In some circumstances when
  retry_sql_inserts was set to 1, after an insertion failure all incoming
  messages were dropped.
• Added DOS/Windows line ending support in config.
• Parallel build is supported for Python and Java destination drivers.
• Fixed compilation failure on OpenBSD
• Memory leak around reload and internal queueing mechanism has been fixed.
• AMQP connection process fixed.
• Fixed a potential abort when the localhost name cannot be detected.
• Security issue fixed around $HOST.
  Tech details:
  When the name of the host is too long, the buffer we use to format the
  chained hostname is truncated. However snprintf() returns the length the
  result would be if no truncation happened, thus we will read uninitialized
  bytes off the stack when we use that pointer to set $HOST
  with log_msg_set_value().
  There can be some security implications, like reading values from the stack
  that can help to craft further exploits, especially in the presense of
  address space randomization. It can also cause a DoS if the hostname length
  is soo large that we would read over the top-of-the-stack, which is probably
  not mmapped causing a SIGSEGV.
• Journal entries containing name-value pairs without '=' caused syslog-ng
  to crash. Instead of crashing, syslog-ng just drop these nv pairs.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Alex Badics, Andras Mitzki, Balazs Scheidler, Bence Tamas Gedai,
Fabien Wernli, Gergely Nagy, Gergo Nagy, Gyorgy Pasztor, Istvan Adam Mozes,
Laszlo Budai, Peter Czanik, Robert Fekete, Tibor Benke, Viktor Juhasz,
Zoltan Pallagi.

3.6.4

This is the fourth maintenance (extra) release for 3.6.x series
and fixes some critical issues.

Fixes

• systemd support fixed on those platforms which has systemd < 209
  (with modular libraries)
• on some platforms(eg.: RHEL6) there was a configure error around
  libsystemd-journal
• AMQP segfaulted right after starting on some platforms

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Balazs Scheidler, Laszlo Budai, Peter Czanik, Tibor Benke,
Viktor Juhasz .

3.6.3

This is the third maintanance release for 3.6.x series.

Changes compared to 3.6.2:

Core fixes

• Inaccurate timestamps fixed on Linux for messages read from /dev/kmsg.
  For those messages syslog-ng uses keep-timestamp(no).
• Added DOS/Windows line ending support in config.
• In some cases, not all the existing plugins were loaded by default.
• In some cases, syslog-ng crashed during stop phase when user wanted
  syslog-ng to stop immediately after start.
• Some memory leak around reload and internal queueing mechanism has been fixed.

Build related fixes

• Manpage build issue fixed by adding --enable-manpages and --with-docbook
  configure option. --with-docbook=PATH gives the user the opportunity to
  specify the path for the user's own installed docbook.
• Fixed parallel build by adding correct dependencies to
  syslog-ng-ctl/Makefile.am.

Module fixes

• When a not writeable file becomes writeable later, syslog-ng recognize it
  (with the help of reopen-timer) and delivers messages to the file without
  dropping those which were received during the file was not available.
• Fixed a crash at the first message delivery when templates are used in
  a filename.
• Fixed a memory leak around file destination driver.
• In some circumstances, during reload, syslog-ng crashed when
  high internal message rate occured.
• When the configured host was not available during the initialization of
  afsocket destination syslog-ng just didn't start. From now, syslog-ng
  starts in that case and will retry connecting to the host periodically.
• Retries fixed in SQL destination. In some circumstances when
  retry_sql_inserts was set to 1, after an insertion failure all incoming
  messages were dropped.
• Connection process fixed in amqp destination and RabbitMQ module is
  set to upstream.
• Monolithic libsystemd library support added.
  In systemd 209, the various small libsystemd-* libraries were merged
  into a single libsystemd. From now, syslog-ng detects and
  uses the merged library when present, while still supports the split
  ones too. If the merged library is found, that will be preferred.
• Destination port fixed in afstomp.
• A memory leak fixed around ping functionality in redis.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Adam Mozes, Andras Mitzki, Balazs Scheidler, Ben Kibbey, Fabien Wernli,
Gergely Nagy, Gergo Nagy, Henrik Grindal Bakken, Laszlo Budai, Peter Czanik,
Pradeep Sanders, Robert Fekete, Tibor Benke, Tomáš Novosad, Toralf Förster,
Viktor Juhasz, Viktor Tusa, Zoltan Pallagi .

3.7.0beta1

This is the first beta release of the upcoming syslog-ng OSE 3.7
branch.

Further releases will focus on fixes and small Getting started ...
documentations.

Changes compared to the previous alpha release:

Features

• Added batched event sending support for riemann destination driver which
  makes the riemann destination respect flush-lines(), and send event
  in batches of configurable amount (defaults to 1). In case of an error,
  all messages within the batch will be dropped. Dropped messages, and
  messages that result in formatting errors do not count towards the batch
  size. There is no timeout, but messages will be flushed upon deinit.
• Added IPv6 netmask filter for selecting only messages sent by a host whose
  IP address belongs to the specified IPv6 subnet.
• Added syslog-ng debug bundle generator script for collecting debug related
  information.
• Added a new macro, called HOSTID which is a 32-bit number generated by
  a cryptographically secure PRNG. Its purpose is to identify the
  syslog-ng host, thus it is the same for every message generated on the same
  host.
• Added a new macro, called UNIQID which is a practically unique ID generated
  from the HOSTID and the RCPTID in the format of HOSTID@RCPTID.
  Uniqid is a derived value: it is built up from the always available hostid
  and the optional rcptid. In other words: uniqid is an extension over rcptid.
  For that reason use-rcptid has been deprecated and use-uniqid could be
  use instead.
• Added a reset option to syslog-ng-ctl stats. With this option the non-stored
  stats counters can be zeroed.
• Java-destination driver ported from syslog-ng-incubator.
  Purpose of having Java destination driver is to provide the right way to
  support all player in the "Java related logging ecosystem"
  (Kafka, HDFS, ElasticSearch, ...). Java dest driver is a special driver,
  a bridge between the C and the Java world from syslog-ng point of view.
• Python language support is ported from syslog-ng incubator and
  has been completely reworked. Now, it is possible to implement template
  functions in Python language and also destination drivers.
  Main purpose of supporting Python language is to implement a nice
  interactive syslog-ng config debugger for syslog-ng.
• New builtin interactive syslog-ng.conf debugger implemented for syslog-ng.
  The debugger has a Python frontend which contains a full Completer
  (just press TABs and works like bash)

Enhancements

• Extended the set of supported characters to every printable ASCII's except
  ., [ and ] in extract-prefix for json-parser().

• OpenSSL set as a hard dependency for syslog-ng because the newly added
  hostid and uniqid features requires a CPRNG provided by OpenSSL.

  After OpenSSL is a hard dependency

  • non-embedded crypto lib is not a real option, so the support of having
    such a crypto lib discontinued
  • all SSL-dependent features enabled by default

• Added string-delimiters option to csvparser to support multi character
  delimiters in CSV parsing.

• Upgrade RabbitMQ submodule to the upstream.

• Extended rcpt-id to 64 bits (formerly it was 48 bits).

Fixes

• Fixed the encoding of characters below 32 if escaping is enabled in
  templates. Templated outputs never contained references to characters below
  32, essentially they were dropped from the output for two reasons:

  • the prefixing backslash was removed from the code
  • the format_uint32_padded() function produced no outputs in base 8

• Fixed afstomp destination port issue. It always tried to connect to the port 0.

• Fixed compilation where the monolitic libsystemd was not available.

• Fixed memleak in db-parser which could happen at every reload.

• Fixed a class of rule conflicts in db-parser:

  Because an error in the pdb load algorithms, some rules would conflict which
  shouldn't have done that. The problem was that several programs would use
  the same RADIX tree to store their patterns. Merging independent programs
  meant that if they the same pattern listed, it would clash, even though
  their $PROGRAM is different.

  There were multiple issues:

  • we looked up pattern string directly, even they might have contained
    @parser@ references. It was simply not designed that way and only
    worked as long as we didn't have the possibility to use parsers
    in program names
  • we could merge programs with the same prefix, e.g.
    su, supervise/syslog-ng and supervise/logindexd would clash, on "su",
    which is a common prefix for all three.

  The solution involved in using a separate hash table for loading, which
  at the end is turned into the radix tree.

  • Fixed a crash around affile at the first message delivery when templates
    were used.
  • Excluded "tags" from riemann destination driver as an attribute which
    conflicts with reserved keyword
  • Fixed a docbook related compilation error: there was a hardcoded path that
    caused build to fail if docbook is not on that path. Debian based
    platforms did not affected by this problem.
    Now a new option was created for ./configure that is --enable-manpages
    that enables the generation of manpages using docbook from online source.
    '--with-docbook=PATH' gives you the opportunity to specify the path for
    your own installed docbook.

Developer notes

• filter: fix external filter plugin lookup

  The filter_plugin rule expected an LL_IDENTIFIER and filter_comparison
  expected a string which in turn is an LL_IDENTIFIER or LL_STRING. It
  caused a conflict in the grammar which prevented to load external
  filter plugins.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Balazs Scheidler, Botond Borsits, Fabien Wernli, Gergely Nagy,
Gergo Nagy, Gyorgy Pasztor, Istvan Adam Mozes, Laszlo Budai,
Manikandan-Selvaganesh, Peter Czanik, Robert Fekete, Tibor Benke,
Viktor Juhasz, Vincent Bernat, Zoltan Fried, Zoltan Pallagi.

3.6.2

This is the first maintenance release for 3.6.x series.

Changes compared to 3.6.1:

Features

• New parameter added to loggen: --permanent (-T) wich is for sending logs
  indefinitely.

Fixes

• From now, syslog-ng won't crash when using a Riemann destination and
  no attributes are set.
• In some cases program destination respawned during syslog-ng stop/restart.
• Max packet length for spoof source is set to 1024 (previously : 256).
• Removed syslog.socket from service file on systems using systemd.
  Syslog-ng reads the messages directly from journal on systems with systemd.
• In some cases, localtime related macros had a wrong value(eg.:$YEAR).
• Transaction handling fixed in SQL destination. In some circumstances when
  both select and insert commands were run within a single transaction and
  the select failed (eg.: in case of mssql), the log messages related to
  the insert commands, broken by the invalid transaction, were lost.
• Fixed a memleak in SQL destination driver.
  The memleak occured during one of the transaction failures.
• A certificate which is not contained by the list of fingerprints is
  rejected from now.
• Hostname check in tls certificate is case insensitive from now.
• Fix spinning on EOF for unix-stream() sockets. Root cause of the spinning
  was that a unix-dgram socket was created even in case of unix-stream.

Compatibility notes

• Prefer SYSLOG_IDENTIFIER over _COMM in systemd-journal.
  In order to not break assumptions, prefer SYSLOG_IDENTIFIER over _COMM.
  For example, postfix uses postfix/qmgr as SYSLOG_IDENTIFIER, but _COMM
  is only "qmgr". The journal itself uses SYSLOG_IDENTIFIER when
  reconstructing the syslog message, so we should not deviate from that
  behaviour, either.

  Similarly, rsyslog also prefers SYSLOG_IDENTIFIER, so for the sake of
  compatibility, doing the same is preferable.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Alexander Görtz, Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy,
Jasper Lievisse Adriaanse, Laszlo Budai, Michael Sterrett, Peter Czanik,
Robert Fekete, Tibor Benke, Viktor Juhasz, Viktor Tusa, Zoltan Fried .

3.7.0alpha2

This is the second alpha release of the syslog-ng OSE 3.7
branch.

Changes compared to the previous alpha release:

Features

• Added support for the monolithic libsystemd library (systemd 209).
• New parameter added to loggen: --permanent (-T) wich is for sending logs
  indefinitely.
• Earlier, in patterndb, the first applicable rule won, even if it was
  only a partial match. This means that when rules overlapped, the shorter
  match would have been found, if it was the first to be loaded.
  A strong preference introduced for rules that match the input string
  completely. The load order is still applicable though, it is possible to
  create two distinct rules that would match the same input, in those cases
  the first one to be loaded wins.
• New parser, extract-solaris-msgid() added for automatically extracts
  (parses & removes) the msgid portion of Solaris messages.

Fixes

• In some cases program destination respawned during syslog-ng stop/restart.
• Max packet length for spoof source is set to 1024 (previously : 256).
• Removed syslog.socket from service file on systems using systemd.
  Syslog-ng reads the messages directly from journal on systems with systemd.
• In some cases, localtime related macros had a wrong value(eg.:$YEAR).
• Transaction handling fixed in SQL destination. In some circumstances when
  both select and insert commands were run within a single transaction and
  the select failed (eg.: in case of mssql), the log messages related to
  the insert commands, broken by the invalid transaction, were lost.
• Fixed a memleak in SQL destination driver.
  The memleak occured during one of the transaction failures.
• A certificate which is not contained by the list of fingerprints is
  rejected from now.
• Hostname check in tls certificate is case insensitive from now.
• Fix spinning on EOF for unix-stream() sockets. Root cause of the spinning
  was that a unix-dgram() socket was created even in case of unix-stream().
• There is a use-case where user wants to ignore an assignment to a name-value
  pair. (eg.: when using csv-parser(), sometimes we get a column we really
  want to drop instead of adding it to the message). In previous versions an
  error message was printed out:
  'Name-value pairs cannot have a zero-length name'.
  That error message has been removed.
• pdbtool match when used with the --debug-pattern option used a low-level
  lookup function, that didn't perform all the db-parser actions specified
  in the rule

Developer notes

• PatternDB lookup refactored (it is easier to understand the code).

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai,
Michael Sterrett, Peter Czanik, Robert Fekete, Tibor Benke, Sean Hussey,
Viktor Juhasz, Viktor Tusa, Zoltan Fried .

3.7.0alpha1

This is the first alpha release of the syslog-ng OSE 3.7
branch.

Changes compared to the latest stable release (3.6.1):

Features

• It is possible to create templates without braces.
• User defined template-function support added.
  User can define template functions in her/his configuration the same
  way she/he would define a template.
• $(format-cim) template function added into an SCL module.
• A new choice for inherit-properties implemented that will merge
  all name-value pairs into the new synthetic message, with the most recent
  being beferred over older values.

Developer notes

• Added implementation for user-defined template functions.
  A new API added, user_template_function_register() that allows
  registering a LogTemplate instance as a template function, dynamically.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai,
Peter Czanik, Viktor Juhasz, Viktor Tusa

3.6.1

This is the first production ready version of syslog-ng OSE 3.6.
More than 25000 lines fof code changed, with about 500 file modified.
The changes since the latest release are the following:

New dependencies

PCRE is now a required dependency of syslog-ng, and is not optional
anymore.

Changed defaults

• Threaded mode is now enabled by default. To turn it off, use
  threaded(no) in the global options section.

• The versioning of the libsyslog-ng internal library has changed:
  instead of always using the current release number, we will now try
  to maintain ABI compatibility during the lifetime of a stable
  branch. Therefore, we use only the first two components of our
  version as the base of the library version. Another number will be
  part of the SONAME too, but that will only change when we break
  compatibility.

  The SONAME is currently set to libsyslog-ng-3.6.so.0, and will
  remain the same during alpha and beta releases, even when the ABI
  changes. We will start bumping the version after the first stable
  release from this branch, if needed.

• The flush-lines() setting now defaults to 100, rather than 1,
  for increased speed.

Features

New options

• A new custom-domain() global setting was introduced, which allows
  the administrator to override the local domain name used by
  syslog-ng. It affects all locally generated log messages.
• Added a use-rcptid() global option, that tells syslog-ng to assign
  a reception ID to each message received and generated by syslog-ng.
  This ID is available as the $RCPTID macro, and is unique on a
  given host. The counter wraps around at 48 bits and is never zero.

New drivers

• The pseudofile() destination driver is a very simple driver, aimed
  at delivering messages to special files in /proc or /dev. It
  opens and closes the file on each message, instead of keeping it
  open. It does not support templates in the filename, and does not
  have a queue (and as such, is not adequate in high traffic
  situations).
• The new nodejs() source driver (implemented as an SCL macro) adds
  a source driver that allows syslog-ng to accept messages from
  node.js applications that use the winston logging API.
• The new systemd-syslog() source replaces the former implicit
  support for the same thing. Users who use systemd are advised to use
  either the system() source, or this new one when they want to
  receive logs from systemd via the /run/systemd/journal/syslog
  socket.
• The new source driver systemd-journal() reads from the Journal directly,
  not via the syslog forwarding socket. The system() source defaults
  to using this source when systemd is detected.
• Added groupset rewrite object.
  Groupset allows the user to modify multiple log message properties at once.
  It also allows referencing the old value of the property as the $_ macro.

Features from the Incubator

• The $(or) template function that returns the first non-empty
  argument is now included in syslog-ng itself.
• The $(padding) template function, to pad text with custom padding
  to a given length is also included.
• The $(graphite-output) template function, to be used for sending
  metrics to Graphite was ported over from the Incubator.
  The graphite() destination SCL block is also available now, to
  make it even easier to talk to Graphite.
• The riemann() destination, which allows sending metrics to the
  Riemann monitoring system was also ported over from the
  Incubator.

Threaded destinations

A number of features were implemented for all threaded destinations:
amqp(), mongodb(), redis(), riemann(), smtp() and stomp().

• The destinations gained support for SEQNUM persistence: the
  counter will be preserved across reloads and restarts.
• A new option called retries() was implemented for all of these,
  which controls how many times a message delivery is retried before
  dropping it.
• The throttle() option is now implemented, and works for all of the
  aforementioned destination drivers.
• The message delivery loop was optimised to do less sleep/wakeup
  cycles, which should make the drivers not only faster, but more CPU
  friendly too.

Miscellaneous new features

• The multi-line-mode() option gained a new setting:
  prefix-suffix, which works similarly to the prefix-garbage
  (which is the new name for regexp), except it appends the garbage
  part to the message, instead of discarding it.

  This new mode can be used to work around the absence of a timeout.

• Filters default to PCRE matching, instead of the previous POSIX
  regexp default.

• The system() source will now parse @cim marked messages as JSON,
  if the JSON module is available at run-time. This improves
  inter-operation with other software that uses the Common
  Information Model.

• One can now use multiple elements in the key() and exclude()
  options of any value-pairs declaration.

• It is now possible to load not only a single certificate when using
  TLS, but a certificate chain.

Statistics

• The stats counter for PROGRAM counters now includes the timestamp of
  the last update.
• A new stats-lifetime() global option was introduced, which
  controls how often dynamic counters are expired. The timer is not
  exact, some timers may live a little bit longer than the specified
  time.
• Dynamic counters are now cleaned up every stats-lifetime() minutes
  (defaulting to 10 minutes) instead of only on reloads. This change
  was done to reduce the memory used by dynamic counters.
• There is now an internal_queue_length statistic, which shows the
  length of the internal queue. This is most useful to see if the
  internal() source is not connected, or if it is not being emptied
  fast enough (which, again, indicates a more serious error).

MongoDB

• The mongodb() driver now supports authentication, even when using
  replica sets. When re-connecting to another member of the set, the
  driver will automatically re-authenticate.
• The --with-libmongo-client option of the configure script now
  supports auto as a value, and will then detect whether to use the
  system version of the library or the internal copy. We default to
  auto now, which prefers the system library over the internal copy.
• The driver does not automatically add an _id field to the message:
  the server will do that automatically, if none is present. This
  allows users to override the field from within their syslog-ng
  config.
• A new retries() option can be used to tell the driver how many
  times it should try to insert a message into the database before
  giving up (defaults to 3). This fixes the case where a rogue message
  could hold up the entire queue, as it was retried forever.
• The driver now enables safe-mode() by default.
• There is now a one-minute timeout for MongoDB operations. If an
  operation times out, it will be considered failed.
• The driver can now connect to MongoDB via UNIX domain sockets.
• The double() type hint is now supported by the driver.
• In the MongoDB destination, reconnecting in a replica-set
  environment now works correctly, and reliably.
• To build syslog-ng with the MongoDB destination, libmongo-client
  version 0.1.8+ is now required. (The internal copy has been updated
  accordingly.)

SMTP destination changes

• The smtp() destination now supports a retries() option, which
  controls how many times a message delivery will be attempted before
  dropping it.
• The templates used in the destination now honor the time-zone
  settings.
• The driver will abort if required options (any of to(), cc(),
  bcc() and from(), and subject() and body()) are not set.

Unix Domain Sockets

• The unix-dgram() and unix-stream() sources now extract UNIX
  credentials (PID, UID and GID of the sending application) from the
  passed messages, if any. On Linux, and FreeBSD, the path of the
  executable belonging to PID is extracted too, along with
  command-line arguments.

  The extracted values are available in ${.unix.pid},
  ${.unix.uid}, ${.unix.gid}, ${.unix.exe} and
  ${.unix.cmdline}, respectively.

• The system() source will overwrite the PID macro with the value of
  ${.unix.pid}, if present.

JSON

• The json-parser gained an extract-prefix() option, which can be
  used to tell the parser to only extract JSON members from a specific
  subtree of the incoming object.

  Example: json-parser(extract-prefix("foo.bar[5]"));

  Assuming that the incoming object is named msg, this is equivalent
  to the following javascript code: msg.foo.bar[5]

  The resulting expression must be a JSON object, so that syslog-ng
  can extract its members into LogMessage name-value pairs.

  This also works when the top-level object is an array, as
  extract-prefix() allows the use of an array index at the first
  indirection level, for example:
  json-parser(extract-prefix("[5]"));, which translates to msg[5].

• The $(format-json) template function now handles the double()
  type hint.

Debugging

• When sending messages to stderr in debug mode, prepend a timestamp
  to the messages.
• The new $RUNID macro is available for templates, which changes its
  value every time syslog-ng is restarted, but not when reloaded.
• A Valgrind suppression file was added (available under
  contrib/valgrind/), to aid in debugging memory leaks in syslog-ng.
  It supresses a couple of known false positives, and a few other
  things in third-party libraries.
• A new utility, system-expand, was added, which returns what the
  system() source would expand to.

Bugfixes

• The reliability of the usertty() destination driver was greatly
  improved. Previously, some parts of it were not thread-safe, which
  could result in strange behaviour.

• The handling of escape related flags of csvparser() was changed:
  instead of these flags overwriting all other (even non-escape
  related) flags, if the flag to set is an escape-flag, it will keep
  all non-escape flags, and set the new one. If it is a not such a
  flag, then it will clear all flags, and set the previous escape
  flags, and the new flag.

  This, in essence, means that when setting flags on a csvparser(),
  if it is an escape flag, only escape flags will be affected. If not,
  then escape flags will not be affected at all.

• The SQL destination now correctly continues $SEQNUM counting after a
  reload, instead of starting afresh.

• Casting error eliminated in Riemann destination when metric is applied to
  an empty field.

• From now, syslog-ng always exclude attributes that conflict with properties
  in Riemann destination (otherwise value of the attribute would override the
  property).

• When tring to stop syslog-ng while a reload is in progress,
  syslog-ng will now correctly shut down cleanly.

• Reloading a config file containing runtime error now not ends in a crash,
  it is able to fallback to the original config.
  (runtime error: config file is grammatically valid but containing invalid
  value, eg.: wrong database column name)

• When the local hostname is not an FQDN, and the local resolver fails
  to return an FQDN too, syslog-ng does not abort anymore, but
  continues using a non-FQDN hostname after emitting a warning on the
  internal source.

  Furthermore, syslog-ng will try to resolve the FQDN harder: when
  multiple names are returned, it will search for the first FQDN one,
  instead of stopping at the primary name.

• The update-patterndb script will now work correctly when the
  current working directory contains .pdb files.

• Patterndb fixed to apply condition even if context-id is missing.

• We will now correctly handle time going backwards in patterndb: it
  will realign its idea of current time with the system. This corrects
  a bug where timeouts did not function properly when system time was
  set backwards.

• The pdbtool merge command will now generate version 4 patterndb
  files.

• The Linux capability support is now correctly auto-detected by the
  configure script, and defaults to off on FreeBSD 9+, as it should.

• The file() and network() (including tcp() et al) sources will
  now properly set the $SOURCE macro.

• The basicfuncs module was fixed to work correctly on 32-bit
  architectures.

• The stored statistics is no longer incremented by various drivers
  when they mean processed.

• The type hinting feature is now more picky about what kind of type
  hints it accepts, allowing one to use template functions in - for
  example - $(format-json) pairs.

• All the various crypto-related template functions now check that the
  desired length of the digest is not larger than the digest itself.
  If a larger value is requested, they will truncate it to the digest
  length.

• The $(geoip) template function now works with threaded(yes) too.

• The in-list() filter was fixed to look at all elements of the
  list, instead of only the last one.

• Fixed an assertion when using the match() filter under certain
  circumstances.

• The system() source will not add /dev/kmsg (or /proc/kmsg on
  older kernels) to the default sources if using the systemd journal,
  because kernel logs are included in the journal.

• The system() source will not include /dev/kmsg (or /proc/kmsg)
  when running inside a Linux container.

• Various memory leak fixes around the code base.

• Change control socket message from notice to debug

• Opening control socket disabled when syslog-ng is used for only
  syntax-checking.

• Fixes for retries() functionality.
  Retry counter incremented by every message write error
  (including network connection errors) which can lead to message lost.

Miscellaneous changes

• We now ship a "Contributors Guide" in the CONTRIBUTING.md file.

Developer notes

The code base went through a lot of refactoring, too many to list in a
simple NEWS file. Groundwork has been laid out for future features
which are yet to hit the 3.6 branch.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Andres Tamayo, Balazs Scheidler, Csaba Karsai, Daniel
Gados, Evan Rempel, Fabien Wernli, Gergely Nagy, Gyorgy Pasztor,
Igor Ippolitov, Imre Lazar, Jakub Wilk, Laszlo Budai, Lucas McLane,
Martin Bagge, Matyas Koszik, Michael Hocke, Nick Alcock, Otto Berger,
Peter Czanik, Peter Gyongyosi, Robert Fekete, Sebastien Badia,
Sebastiaan Hoogeveen, Tamas Pal, Tibor Benke, Tobias Schwab, Viktor
Juhasz, Viktor Tusa, Xufeng Zhang

3.6.0rc2

This is the second (hopefully last) Release Candidate of the syslog-ng
OSE 3.6 branch. Some release critical bugs found and fixed.

Bugfixes

• Opening control socket disabled when syslog-ng is used for only
  syntax-checking.
• Reloading a config file containing runtime error now not ends in a crash,
  it is able to fallback to the original config.
  (runtime error: config file is grammatically valid but containing invalid
  value, eg.: wrong database column name)
• Casting error eliminated in Riemann destination when metric is applied to
  an empty field.
• From now, syslog-ng always exclude attributes that conflict with properties
  in Riemann destination (otherwise value of the attribute would override the
  property).
• Patterndb fixed to apply condition even if context-id is missing.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai,
Viktor Tusa

3.6.0rc1

This is the first Release Candidate of the syslog-ng OSE 3.6 branch.
Based on our test results this release is almost production ready.

Features

• Added groupset rewrite object.
  Groupset allows the user to modify multiple log message properties at once.
  It also allows referencing the old value of the property as the $_ macro.

Bugfixes

• Fixed a memory leak during configuration parsing when using rewrite().
• Change control socket message from notice to debug
• Fixes for retries() functionality.
  Retry counter incremented by every message write error
  (including network connection errors) which can lead to message lost.

Credits

syslog-ng is developed as a community project, and as such it relies
on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing
feedback are all important contributions, so please if you are a user
of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Balazs Scheidler, Brian De Wolf, Gergely Nagy, Laszlo Budai, Peter Czanik,
Tibor Benke, Viktor Juhasz, Viktor Tusa.

3.6.0beta2

This is the second beta release of the upcoming syslog-ng OSE 3.6 branch. Compared to the previous beta, this release contains a few minor features and bugfixes. We expect the next release to be a release candidate, focusing on stability and bugfixes. Testing is most appreciated!

Features

• It is now possible to load not only a single certificate when using TLS, but a certificate chain.
• The system() source will not include /dev/kmsg (or /proc/kmsg) when running inside a Linux container.

Bugfixes

• The in-list() filter was fixed to look at all elements of the list, instead of only the last one.
• Fixed an assertion when using the match() filter under certain circumstances.
• The system() source will not add /dev/kmsg (or /proc/kmsg on older kernels) to the default sources if using the systemd journal, because kernel logs are included in the journal.

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Balazs Scheidler, Gergely Nagy, Gyorgy Pasztor, Peter Czanik, Tibor Benke.

3.6.0beta1

This is the first beta release of the upcoming syslog-ng OSE 3.6 branch. Compared to the alphas, this release contains a moderate amount of new functionality and bugfixes. Further releases will focus on stability and bugfixes.

Features

• One can now use multiple elements in the key() and exclude() options of any value-pairs declaration.
• A new source driver was added to the syslog-ng: systemd-journal(), which reads from the Journal directly, not via the syslog forwarding socket. The system() source defaults to using this source when systemd is detected.

Bugfixes

• All the various crypto-related template functions now check that the desired length of the digest is not larger than the digest itself. If a larger value is requested, they will truncate it to the digest length.
• The $(geoip) template function now works with threaded(yes) too.
• The unix domain socket credentials code was changed to only build on Linux and FreeBSD. With this change, syslog-ng should compile again on platforms where the OS does not support this, with the feature disabled.

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Fabien Wernli, Gergely Nagy, Laszlo Budai, Michael Hocke, Tibor Benke, Viktor Juhasz, Viktor Tusa.

3.6.0alpha3

This is the third alpha release of the upcoming syslog-ng OSE 3.6 branch. It is expected to be the last alpha release, with the first beta in about two weeks. This release contains a number of important features and bugfixes:

Changed defaults

• The flush-lines() setting now defaults to 100, rather than 1, for increased speed.

Features

• The system() source will now parse @cim marked messages as JSON, if the JSON module is available at run-time. This improves inter-operation with other software that uses the Common Information Model.

Features from the Incubator

• The $(or) template function that returns the first non-empty argument is now included in syslog-ng itself.
• The $(padding) template function, to pad text with custom padding to a given length is also included.
• The $(graphite-output) template function, to be used for sending metrics to Graphite was ported over from the Incubator. The graphite() destination SCL block is also available now, to make it even easier to talk to Graphite.
• The riemann() destination, which allows sending metrics to the Riemann monitoring system was also ported over from the Incubator.

Threaded destinations

A number of features were implemented for all threaded destinations: amqp(), mongodb(), redis(), riemann(), smtp() and stomp().

• The destinations gained support for SEQNUM persistence: the counter will be preserved across reloads and restarts.
• A new option called retries() was implemented for all of these, which controls how many times a message delivery is retried before dropping it.
• The throttle() option is now implemented, and works for all of the aforementioned destination drivers.
• The message delivery loop was optimised to do less sleep/wakeup cycles, which should make the drivers not only faster, but more CPU friendly too.

Bugfixes

• The basicfuncs module was fixed to work correctly on 32-bit architectures.
• The stored statistics is no longer incremented by various drivers when they mean processed.
• The type hinting feature is now more picky about what kind of type hints it accepts, allowing one to use template functions in - for example - $(format-json) pairs.

Miscellaneous changes

• We now ship a "Contributors Guide" in the CONTRIBUTING.md file.

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Balazs Scheidler, Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Robert Fekete, Tibor Benke, Viktor Juhasz, Viktor Tusa.

3.6.0alpha2

This is the second alpha release of the upcoming syslog-ng OSE 3.6 branch, with more internal changes and features compared to the previous 3.6.0alpha1 release. Above the changes in the latest 3.5.6 stable version, this release contains the following noteworthy changes:

Changed defaults

• Threaded mode is now enabled by default. To turn it off, use threaded(no) in the global options section.

• The versioning of the libsyslog-ng internal library has changed: instead of always using the current release number, we will now try to maintain ABI compatibility during the lifetime of a stable branch. Therefore, we use only the first two components of our version as the base of the library version. Another number will be part of the SONAME too, but that will only change when we break compatibility.

  The SONAME is currently set to libsyslog-ng-3.6.so.0, and will remain the same during alpha and beta releases, even when the ABI changes. We will start bumping the version after the first stable release from this branch, if needed.

Features

• The new systemd-syslog() source replaces the former implicit support for the same thing. Users who use systemd are advised to use either the system() source, or this new one when they want to receive logs from systemd via the /run/systemd/journal/syslog socket.

SMTP destination changes

• The smtp() destination now supports a retries() option, which controls how many times a message delivery will be attempted before dropping it.
• The destination no longer counts delivered messages as stored.
• The templates used in the destination now honor the time-zone settings.
• The driver will abort if required options (any of to(), cc(), bcc() and from(), and subject() and body()) are not set.

Bugfixes

• The file() and network() (including tcp() et al) sources will now properly set the $SOURCE macro.
• To build syslog-ng with the MongoDB destination, libmongo-client version 0.1.8+ is now required. (The internal copy has been updated accordingly.)
• The UNIX credential extracting feature was ported to FreeBSD, syslog-ng now compiles, and has support for this feature. (The previous support in alpha1 was sadly incomplete.)
• The pdbtool merge command will now generate version 4 patterndb files.

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Fabien Wernli, Gergely Nagy, Laszlo Budai, Peter Czanik, Tibor Benke, Viktor Juhasz, Viktor Tusa.

This is the sixth bug-fix release for the 3.5.x series. Upgrading from earlier versions is highly recommended, as the changes in this release are very small, yet, also very important for most platforms and workloads.

Bugfixes

• A major memory leak was fixed in the value-pairs framework, which affects $(format-json), MongoDB, AMQP and more. The leak was supposed to be fixed in 3.5.5, but due to a merging mistake, it was missed.
• The configure script now detects Linux capabilities properly, there is no need to use --disable-linux-caps on non-Linux platforms anymore.
• The pdbtool merge command will now generate version 4 patterndb files, instead of version 3.

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

These people have helped in this release:

Fabien Wernli, Gergely Nagy, Peter Czanik

3.6.0alpha1

This is the first alpha release of the upcoming syslog-ng OSE 3.6 branch, a result of about seven months of work, by more than a dozen contributors, touching 379 files, and changing over twenty thousand lines.

Compared to the latest stable release (3.5.5), this alpha release contains the following noteworthy changes:

New dependencies

PCRE is now a required dependency of syslog-ng, and is not optional anymore.

Features

New options

• A new custom-domain() global setting was introduced, which allows the administrator to override the local domain name used by syslog-ng. It affects all locally generated log messages.
• Added a use-rcptid() global option, that tells syslog-ng to assign a reception ID to each message received and generated by syslog-ng. This ID is available as the $RCPTID macro, and is unique on a given host. The counter wraps around at 48 bits and is never zero.

New drivers

• The pseudofile() destination driver is a very simple driver, aimed at delivering messages to special files in /proc or /dev. It opens and closes the file on each message, instead of keeping it open. It does not support templates in the filename, and does not have a queue (and as such, is not adequate in high traffic situations).
• The new nodejs() source driver (implemented as an SCL macro) adds a source driver that allows syslog-ng to accept messages from node.js applications that use the winston logging API.

Miscellaneous new features

• The multi-line-mode() option gained a new setting: prefix-suffix, which works similarly to the prefix-garbage (which is the new name for regexp), except it appends the garbage part to the message, instead of discarding it.

  This new mode can be used to work around the absence of a timeout.

• Filters default to PCRE matching, instead of the previous POSIX regexp default.

Statistics

• The stats counter for PROGRAM counters now includes the timestamp of the last update.
• A new stats-lifetime() global option was introduced, which controls how often dynamic counters are expired. The timer is not exact, some timers may live a little bit longer than the specified time.
• Dynamic counters are now cleaned up every stats-lifetime() minutes (defaulting to 10 minutes) instead of only on reloads. This change was done to reduce the memory used by dynamic counters.
• There is now an internal_queue_length statistic, which shows the length of the internal queue. This is most useful to see if the internal() source is not connected, or if it is not being emptied fast enough (which, again, indicates a more serious error).

MongoDB

• The mongodb() driver now supports authentication, even when using replica sets. When re-connecting to another member of the set, the driver will automatically re-authenticate.
• The --with-libmongo-client option of the configure script now supports auto as a value, and will then detect whether to use the system version of the library or the internal copy. We default to auto now, which prefers the system library over the internal copy.
• The driver does not automatically add an _id field to the message: the server will do that automatically, if none is present. This allows users to override the field from within their syslog-ng config.
• A new retries() option can be used to tell the driver how many times it should try to insert a message into the database before giving up (defaults to 3). This fixes the case where a rogue message could hold up the entire queue, as it was retried forever.
• The driver now enables safe-mode() by default.
• There is now a one-minute timeout for MongoDB operations. If an operation times out, it will be considered failed.
• The driver can now connect to MongoDB via UNIX domain sockets.
• The double() type hint is now supported by the driver.

Unix Domain Sockets

• The unix-dgram() and unix-stream() sources now extract UNIX credentials (PID, UID and GID of the sending application) from the passed messages, if any. On Linux, and FreeBSD, the path of the executable belonging to PID is extracted too, along with command-line arguments.

  The extracted values are available in ${.unix.pid}, ${.unix.uid}, ${.unix.gid}, ${.unix.exe} and ${.unix.cmdline}, respectively.

• The system() source will overwrite the PID macro with the value of ${.unix.pid}, if present.

JSON

• The json-parser gained an extract-prefix() option, which can be used to tell the parser to only extract JSON members from a specific subtree of the incoming object.

  Example: json-parser(extract-prefix("foo.bar[5]"));

  Assuming that the incoming object is named msg, this is equivalent to the following javascript code: msg.foo.bar[5]

  The resulting expression must be a JSON object, so that syslog-ng can extract its members into LogMessage name-value pairs.

  This also works when the top-level object is an array, as extract-prefix() allows the use of an array index at the first indirection level, for example: json-parser(extract-prefix("[5]"));, which translates to msg[5].

• The $(format-json) template function now handles the double() type hint.

Debugging

• When sending messages to stderr in debug mode, prepend a timestamp to the messages.
• The new $RUNID macro is available for templates, which changes its value every time syslog-ng is restarted, but not when reloaded.
• A Valgrind suppression file was added (available under contrib/valgrind/), to aid in debugging memory leaks in syslog-ng. It supresses a couple of known false positives, and a few other things in third-party libraries.
• A new utility, system-expand, was added, which returns what the system() source would expand to.

Bugfixes

• With the MongoDB destination, successfully inserted messages are not counted as "stored" anymore: stored messages are those that are in a memory or disk buffer.

• In the MongoDB destination, reconnecting in a replica-set environment now works correctly, and reliably.

• The reliability of the usertty() destination driver was greatly improved. Previously, some parts of it were not thread-safe, which could result in strange behaviour.

• The handling of escape related flags of csvparser() was changed: instead of these flags overwriting all other (even non-escape related) flags, if the flag to set is an escape-flag, it will keep all non-escape flags, and set the new one. If it is a not such a flag, then it will clear all flags, and set the previous escape flags, and the new flag.

  This, in essence, means that when setting flags on a csvparser(), if it is an escape flag, only escape flags will be affected. If not, then escape flags will not be affected at all.

• The SQL destination now correctly continues $SEQNUM counting after a reload, instead of starting afresh.

• When tring to stop syslog-ng while a reload is in progress, syslog-ng will now correctly shut down cleanly.

• When the local hostname is not an FQDN, and the local resolver fails to return an FQDN too, syslog-ng does not abort anymore, but continues using a non-FQDN hostname after emitting a warning on the internal source.

  Furthermore, syslog-ng will try to resolve the FQDN harder: when multiple names are returned, it will search for the first FQDN one, instead of stopping at the primary name.

• The update-patterndb script will now work correctly when the current working directory contains .pdb files.

• We will now correctly handle time going backwards in patterndb: it will realign its idea of current time with the system. This corrects a bug where timeouts did not function properly when system time was set backwards.

• The Linux capability support is now correctly auto-detected by the configure script, and defaults to off on FreeBSD 9+, as it should.

• Various memory leak fixes around the code base.

Developer notes

The code base went through a lot of refactoring, too many to list in a simple NEWS file. Groundwork has been laid out for future features which are yet to hit the 3.6 branch.

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessary to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andras Mitzki, Andres Tamayo, Balazs Scheidler, Csaba Karsai, Daniel Gados, Evan Rempel, Fabien Wernli, Gergely Nagy, Igor Ippolitov, Imre Lazar, Jakub Wilk, Laszlo Budai, Lucas McLane, Martin Bagge, Matyas Koszik, Nick Alcock, Otto Berger, Peter Czanik, Peter Gyongyosi, Sebastien Badia, Sebastiaan Hoogeveen, Tamas Pal, Tibor Benke, Tobias Schwab, Viktor Juhasz, Viktor Tusa, Xufeng Zhang