osquery

This is a pre-release for FreeBSD's port.

New features in 2.4.2:

Bug fixes

#3157 Use poll over select in inotify (and auditd, udev) publisher
#3169 Adding bounds checks and key checks for appcompat shims table
#3170 Fixing stringToWstring crashes with wide character strings
#3187 Drop permissions properly on Linux (fixes rpm_packages on CentOS7)
#3179 Add database initialization retry

Config options / CLI flags changes

The host_identifier can now take a specified (explicit) identifier.

--host_identifier VALUE Field used to identify the host running osquery (hostname, uuid, instance, ephemeral, specified)
--specified_identifier VALUE Field used to specify the host_identifier when set to "specified"

Table changes (from 2.4.0 to 2.4.2):

Moved table routes to All Platforms
Added table event_taps to Darwin (Apple OS X)
Added table time_machine_backups to Darwin (Apple OS X)
Added table time_machine_destinations to Darwin (Apple OS X)
Added table scheduled_tasks to Microsoft Windows

Added hidden column eid (TEXT_TYPE) to all *_events (used for testing)

New features in 2.4.0:

Important changes

#3073 The Windows registry table was refactored to have a look and feel like the file table.
#3049 Distributed (ad-hoc) queries now support discovery queries.
#3087 & #3091 Improve events tables performance and protect against multiple queries overwriting sliding window optimizations.
#3100 Add globbing support to the Windows registry table.
#3120 Add the auid column to all Audit-based tables.
#3115 Add status logging to AWS-based logger plugins.

Bug fixes

#3065 Set a max size for RocksDB MANIFEST logs, this helps protect against very large transaction logs leading to massive on-disk files.
#3098 Fix crash when sanitizing REG_NONE types from Windows registry.
#3106 Return blank or NULL values for sha, md5 and sha256 when files cannot be hashed.
#3116 Fix potential deadlock with periodic database reset.
#3142 Fix reentry bug with our GLog logger sink leading to potential deadlocks.

Config options / CLI flags changes

--logger_min_status VALUE Minimum level for status log recording 1=INFO, 2=WARNING, 3=ERROR

Table changes (from 2.3.4 to 2.4.0):

Moved table startup_items from Darwin to All Platforms

Added table lldp_neighbors to POSIX-compatible Plaforms
Added table python_packages to POSIX-compatible Plaforms
Added column auid (BIGINT_TYPE) to table process_events
Added column auid (BIGINT_TYPE) to table socket_events
Added column auid (BIGINT_TYPE) to table user_events

Renamed table syslog to syslog_events on Ubuntu, CentOS (the alias syslog still exists)

Breaking Table API changes

Removed column hive (TEXT_TYPE) from table registry
Removed column subkey (TEXT_TYPE) from table registry
The hive and subkey columns have been combined into a path column.

There are several table API changes but this is mostly a bug-fix release to support macOS < 10.12, IPv6-only stacks, and fix a crash on Linux platform_info or smbios_tables tables.

New Features

#2922 The package build now produces a sindle DEB and RPM
#3022 FreeBSD build working again and ports bumped
#3020 IPv6-only stack now supported

Bug fixes

#3018 Fix crash on Linux with platform_info table

New Features in 2.3.2:

This release brings about better behavior to the watcher, config plugin, and extensions framework. Additionally this release brings the Windows Event log publisher to Windows, as well as a more robust Windows chocolatey packaging and experimental support for Windows 7 and Server 2008.

Bug fixes

#2916 A deadlock existed between the decorator and status log messages being written
#2734 Moving the OpenSSL requirement to the utility to additional libraries greatly simplified the building of libosquery
#2912 An update in 0.10.0 of thrift required transport connections to be .closed before an .open would be successful.
#2881 Dependency builds were failing due to a change in our hosted glibc mirror
#2983 The memory_map table in Linux did not check for malformed input

Config options / CLI flags changes

--windows_event_channels Overrides which Windows Event log channels are subscribed to.

Table Changes (from 2.2.0 to 2.3.2):

Added table arp_cache to All Platforms
Added table hash to All Platforms
Added table logged_in_users to All Platforms
Added table windows_events to Microsoft Windows
Added column instance_id (TEXT_TYPE) to table osquery_info
Added column uuid (TEXT_TYPE) to table osquery_info
Added column country_code (TEXT_TYPE) to table wifi_status
Added column country_code (TEXT_TYPE) to table wifi_survey
Removed table arp_cache from POSIX-compatible Plaforms
Removed table logged_in_users from POSIX-compatible Plaforms
Removed table hash from Utility

New features in 2.2.1:

This is a very small release, addressing some bugs in 2.2.0.

#2867 Removed stream name checking for AWS kinesis and firehose
#2878 Fix fault in apt_sources systems without apt and dpkg
#2879 Allow EINTR to continue the Thrift Transport reducing stops when using privilege dropping tables and extensions-based logger plugins
#2880 Fix exception in OS X platform_info table

New features in 2.2.0:

This release is focused on stability.

It improves the backing store database use both by increasing performance and by identifying and fixing logic bugs. Several threading issues have been fixed reducing hard-to-triage errors from complex virtual table and logger plugin use.

Some care has been given to the Linux audit publisher resulting in a safer experience.

Important changes

#2665 For events-based tables the --events_optimize flag is now applied per-query
#2703 The kinesis logger plugin now only sends failed data when retrying
#2181 Add support for augeas configuration parsing, this table needs the supporting lenses installed
#2785 Configuration JSON may include escaped newlines
#2787 "Multi-pack" configuration support
#2792 With Linux process and socket auditing, the Linux audit configuration is now optimized
#2817 The make deps target is now split into make sysprep and deps
#2855 Allow module and extension autoload paths to include directories

Bug fixes

#2751 Fix potential crash in Linux FIM where a patch inode is added without a corresponding path string
#2806 Temporary permissions dropping now uses per-thread syscalls and pthread APIs
#2749 Fix deadline in RocksDB when errors occur and logging is needed
#2847 Use local reference to binary name to avoid memory corruption with syslog
#2865 Fix potential fault when parsing strings in plists

Config options / CLI flags changes

--enroll_always On startup, send a new enrollment request
--watchdog_memory_limit VALUE Override watchdog profile memory limit
--watchdog_utilization_limit VALUE Override watchdog profile CPU utilization limit
--logger_syslog_prepend_cee Prepend "@cee:" tag to logged JSON messages

Table changes (from 2.1.2 to 2.2.0):

Moved table listening_ports to All Platforms from POSIX
Moved table process_open_sockets to All Platforms from POSIX

Added table wifi_status to Darwin (Apple OS X)
Added table wifi_survey to Darwin (Apple OS X)
Added table drivers to Microsoft Windows
Added table patches to Microsoft Windows
Added table augeas to POSIX-compatible Plaforms
Added table sudoers to POSIX-compatible Plaforms
Added table portage_keywords to Ubuntu, CentOS
Added table portage_packages to Ubuntu, CentOS
Added table portage_use to Ubuntu, CentOS
Added column idrops (BIGINT_TYPE) to table interface_details
Added column odrops (BIGINT_TYPE) to table interface_details

New features in 2.1.2:

This is a very small release, addressing some nice-to-haves from 2.1.1.

#2718 OS X will try to build RPMs if fpm exists
#2714 OS X's safari_extensions table uses libxar to improve parsing coverage
#2719 The ./build/darwin folder will be used as the most-recent darwin build folder
#2708 The TLS config plugins uses the correct VERB corresponding to the API documentation
#2700 If the logger_path becomes unusable the daemon will stop with an error to syslog

New features in 2.1.1:

This is a very small release, addressing some nice-to-haves from 2.1.0.

To help improve some debugging scenarios we've added .features and .summary to osqueryi.
#2695 Also adds some color to the osquery> prompt if your terminal emulator supports 256-color output.

#2692 Extensions will activate their distributed plugin
#2694 OS X's preferences table now supports LIKE
#2693 Several Linux publishers are fixed to tearDown during destruction

New features in 2.1.0:

This version adds lots of fixes for Windows.

Please see the important changes and bug fixes section.

Plugin API changes

Version 2.1.0 is backward compatible with extensions SDK to version 1.7.3. A warning will be emitted if an older extension is connected.

Important changes

#2613 The readline implementation is replaced with linenoise-ng
#2624 Drop privileges when reading RPM and DPKG databases
#2650 Update dependency hashes for OS X Sierra
#2670 The distributed APIs may now use custom plugins
#2673 Systems may install a default flagfile at /etc/osquery/osquery.flags.default

Bug fixes

#2593 The table column option INDEX is used to prioritize scanning cost
#2596 Allow custom HTTP redirects within AWS plugins
#2561 RPM packages are saved with the correct filename format
#2612 Linux SMBIOS tables prefer sysfs nodes and content over memory reads
#2616 Promote UUID to version 3 and strip trailing NULL bytes
#2617 Fix pidfile read errors on Windows Server 2012
#2627 TLS logger re-enrollment now includes enrollment data
#2621 Prioritize reading node_invalid when handling TLS response errors
#2634 Ignore pidfile parsing errors when starting osqueryd
#2654 Limit virtual table usage 'help' messages in osqueryi
#2677 Add SQLite prepare lock to fix fast extension load bugs
#2682 Prevent Linux audit from default-enabling socket_events
#2684 Fix Linux routes table inconsistencies

Config options / CLI flags changes

--extensions_require VALUE Comma-separated list of required extensions
--extension VALUE Path to a single extension to autoload

Table changes (from 2.0.0 to 2.1.0):

Added table chrome_extensions to All Platforms
Added table etc_services to All Platforms
Added table kernel_info to All Platforms
Added table platform_info to All Platforms

Added table appcompat_shims to Microsoft Windows
Added table services to Microsoft Windows

Added column codename (TEXT_TYPE) to table os_version
Added column platform (TEXT_TYPE) to table os_version
Added column platform_like (TEXT_TYPE) to table os_version
Added column version (TEXT_TYPE) to table os_version
Added column query_language (TEXT_TYPE) to table wmi_event_filters

New features in 2.0.0:

This release brings osquery to Windows! Currently only Windows 8+ is supported, and many features that exist on POSIX platforms are still being ported. Our Windows support includes an abstraction on top of the WMI architecture provided by Microsoft thus allowing rapid development of new virtual tables. We include a dependency and build management system to statically compile binaries independent of system location or supported platform. More simply, we can support a single build that may be deployed to any Windows version (8+).

This release introduces a new Brew-based build redesign (#2251) for Linux and OS X. This has enabled a variety of new CI features, including:

• clang-format checking to insure style uniformity.
• cppcheck to catch performance bugs via static-analysis.
• Version-pinned LLVM with sanitization frameworks for nightly dynamic-analysis and memory leak checking.
• zzuf for DIY fuzz testing via make fuzz.
• And of course, far-fewer dependency errors due to build host drift.

As always, we hold performance as an essential feature and project goal. With 2.0 we are also including a user-experience focus. The most significant related change is to TablePlugins; we have added a column-attribute and table-attribute description language so SQLite and the osqueryi shell can make decisions based on how tables and columns are used. As a quick example, consider selecting from file-- in 2.0 if you do not include a path or directory the table will emit a warning; if you include an _events-based table in your schedule the set-difference calculations will be skipped.

Plugin API changes

#2412 Rename phys_footprint to total_size in the processes table.
#2525 Promote host UUID to version 2 meaning Linux UUIDs become board UUIDs.
#2527 Add extensions SDK incompatibility checking.

Version 2.0.0 is backward compatible with extensions SDK to version 1.7.3. A warning will be emitted if an older extension is connected.

Important changes

#2523 Refactor events and remove the 10/3600 indexes this yields ~4x speed improvement.
#2500 Improve Thrift exit verbosity on all platforms by forwarding output to Glog.
#2504 Change --utc to default as true.

Bug fixes

#2309 Fix race conditions in Linux inotify publisher configuration
#2316 Add size check to package_bom variable address bounds checking
#2320 Properly intialize BufferedLogForwarder for TLS output plugin
#2345 Avoid constructor ambiguity in table headers for extensions
#2348 Use seconds for --profile_delay precision
#2416 Fix Linux memory_map printing and convert to using IOMEM-mappings
#2417 Handle empty Linux pwd structures members
#2422 Improve status logging when using multi-loggers
#2447 Multiple bug fixes in the crashes virtual table
#2455 Fix minor sandboxes virtual table performance issues and plist parsing exceptions
#2457 Fix potential string casting issue in memory_info virtual table
#2508 Remove time-override for events add API
#2528 Correct config-loaded boolean meaning to become has-run-load
#2529 Create temp directory for exceptional shell uses and fallback to home directories
#2530 Initialize VirtualTableContent attributes for extesnions
#2562 Fix memory leak within osqueryi when using the -A flag

Config options / CLI flags changes

—install flag added for windows to install the daemon with the Windows Service Manager
—uninstall flag added for windows to remove the daemon from the Windows Service Manager

--aws_sts_arn_role AWS STS ARN role
--aws_sts_region AWS STS region
--aws_sts_session_name AWS STS session name
--aws_sts_timeout AWS STS assume role credential validity in seconds (default 3600)

Table changes (from 1.8.2 to 2.0.0):

Added table carbon_black_info to All Platforms
Added table etc_hosts to All Platforms
Added table etc_protocols to All Platforms

Added table kernel_panics to Darwin (Apple OS X)

Added table apt_sources to Ubuntu, CentOS
Added table deb_packages to Ubuntu, CentOS
Added table rpm_package_files to Ubuntu, CentOS
Added table rpm_packages to Ubuntu, CentOS

Added table programs to Microsoft Windows
Added table registry to Microsoft Windows
Added table shared_resources to Microsoft Windows
Added table wmi_cli_event_consumers to Microsoft Windows
Added table wmi_event_filters to Microsoft Windows
Added table wmi_filter_consumer_binding to Microsoft Windows
Added table wmi_script_event_consumers to Microsoft Windows

Renamed column phys_footprint to total_size (BIGINT_TYPE) in table processes
Renamed column restarts to refreshes (INTEGER_TYPE) in table osquery_events

Added column type (TEXT_TYPE) to table logged_in_users
Added column name (TEXT_TYPE) to table memory_map
Added column active (INTEGER_TYPE) to table osquery_packs
Added column threads (INTEGER_TYPE) to table processes
Added column datetime (TEXT_TYPE) to table syslog

Removed column region (INTEGER_TYPE) from table memory_map
Removed column type (TEXT_TYPE) from table memory_map

New features in 1.8.2:

This is a breakfix release for those using the AWS logger plugins on OS X.

The firehose and kinesis logger plugins use the cpp-netlib TLS client libraries, which depend on ASIO, boost, and a TLS implementation provided by OpenSSL or LibreSSL. This release allows the plugins to take advantage of --tls_server_certs and other TLS-related configuration options. If you are using these logger plugins and receiving invalid certificate issue, you need to provide a PEM bundle using the aforementioned flag.

Bug fixes

#2285 Fix 'off the end' potential bug in crashes table
#2287 Use "UTC" for timezone when no timezone is present in the time table
#2299 Use TLSTransport HTTPS client within AWS logger plugins

Config options / CLI flags changes

--buffered_log_max Maximum number of logs in buffered output plugins (0 = unlimited)

Table changes (from 1.8.1 to 1.8.2):

Added table memory_info to Ubuntu, CentOS

New features in 1.8.1:

This is a bugfix release for version 1.8.0.

Critical Bug fixes:

#2259 Fixed memory leak in QueryContext affecting watchdog processes
#2275 Fix regression from 1.7.7 where the Linux daemon searched for a config in /var/osquery

Bug fixes:

#2255 Improved defaults for systemd service unit
#2266 Fix kernel extension build and COMM version requirement
#2267 The process_file_events subscriber was incorrectly added to Linux

Config options / CLI flags changes

--audit_allow_sockets Enable the socket_events on Linux

Table changes (from 1.8.0 to 1.8.1):

Added table user_ssh_keys to POSIX-compatible Plaforms
Added column watcher (INTEGER_TYPE) to table osquery_info

New features in 1.8.0:

This is a release rollup of fixes since 1.7.4.

There is an optional Thrift API change for extensions: the shutdown method.
The osquery core, extension manager, will attempt to call this optionally-implemented method
immediately before it shuts down. This request is blocking and allows an extension to perform
cleanup before its watcher thread quits.

Plugin API changes

#2224 Add shutdown() method to extensions API
#2229 The logger facilities now write catastrophic errors to syslog
#2241 Distributed queries will log verbose events indicating their query requests

Bug fixes:

#2205 Fix milli/micro conversion when waiting for active plugins (regression from 1.7.4)
#2207 Restore extension respawn limits to 20s (regression from 1.7.4)
#2217 Fix SQLite local access after ASIO URL usage (OS X)
#2228 Force RocksDB to sync writes for non-event domains
#2234 Fix various Linux process path parsing errors

Config options / CLI flags changes

--decorations_top_level Add decorators as top level JSON objects