osquery

osquery 3.4.0 Release Notes

This tag is a Windows only release containing various bug and vulnerability fixes, as well as numerous improvements to performance. The processes table has been re-written to no longer make use of WMI and various aspects of the Windows build system has been re-written to make use of the new buck build system. A critical deadlocking bug has been addressed in the thread management system which will allow osquery to make use of the TLS plugins without deadlocking on service restart.

Below are some of the highlights as they relate to the Windows release. This tag contains well over 250 commits, and there is considerably more content added than what is detailed below. Investigate the full commit history since our last tag for greater details on what has changed since the last tag.

Security Vulnerabilities

#5568 CVE-2019-3567 - osquery is now installed to Program Files to prevent a privilege escalation vulnerability

Bug Fixes

#5421 - addressing deadlock regression in windows dispatcher threads
#5304 - key_strength now correctly displays in certificates table

New Features

#5431 - Add Windows product version information to file table
#5400 - logical_drives table has been drastically refactored
#5454 - sid and hive columns added to the logged_in_users table
#5293 - Processes table now selectively generates columns, no longer uses WMI

New features include

#4094 Add opt-in write-support for extensions
#4224 Add SELinux event recording on Linux
#4626 Add number monitoring system concept

Bug fixes

#4416 Added custom version of realpath
#4599 Resource protection for udev structures
#4579 Fix case where regular files were reported as symlinks
#4695 Fix use of incorrect directory separator
#4686 Improve etc_hosts table data
#4647 Improve audit-based table performance

Table changes (from 3.2.7 to 3.2.8)

Added table logon_sessions to Microsoft Windows
Added table winbaseobj to Microsoft Windows
Added table ssh_configs to POSIX-compatible Plaforms
Added table smart_drive_info to SMART
Added table elf_dynamic to Ubuntu, CentOS
Added table elf_info to Ubuntu, CentOS
Added table elf_sections to Ubuntu, CentOS
Added table elf_segments to Ubuntu, CentOS
Added table elf_symbols to Ubuntu, CentOS
Added table selinux_events to Ubuntu, CentOS
Added column socket_designation (TEXT_TYPE) to table cpu_info
Added column encryption_status (TEXT_TYPE) to table disk_encryption
Added column attributes (TEXT_TYPE) to table file
Added column file_id (TEXT_TYPE) to table file
Added column volume_serial (TEXT_TYPE) to table file
Added column ssdeep (TEXT_TYPE) to table hash
Added column cpu_subtype (INTEGER_TYPE) to table processes
Added column cpu_type (INTEGER_TYPE) to table processes

This releases updates some code dependencies and addresses several bugs.

Update libxml to version 2.9.7
Update yara to version 3.7.1
Update openssl to version 1.0.20

Bug fixes

#4561 Fix Dispatcher race conditions
#4597 Fix memory leak in Dispatcher
#4585 Never give up on failed extensions

Table changes (from 3.2.8 to 3.2.9)

Added table ntfs_acl_permissions to Microsoft Windows

This release fixes a serious issue causing dead locks in the Registry. The bug was introduced in the 3.2 release.

Bug fixes

#4538 - Windows Events may drop events due to case-mismatches
#4549 - Writes to /dev/null on macOS caused performance issues
#4359 - Autoloaded extensions could outlive the main process
#4531 - Do note reset audit handle when poll returns EINTR
#4528 - Fix potential local in Registry caused by extensions

Table changes (from 3.2.7 to 3.2.8)

Added table process_namespaces to Linux
Removed column cgroup_namespace (TEXT_TYPE) from table processes
Removed column ipc_namespace (TEXT_TYPE) from table processes
Removed column mnt_namespace (TEXT_TYPE) from table processes
Removed column net_namespace (TEXT_TYPE) from table processes
Removed column pid_namespace (TEXT_TYPE) from table processes
Removed column user_namespace (TEXT_TYPE) from table processes
Removed column uts_namespace (TEXT_TYPE) from table processes

This release is made available to address CVE-2018-6336.
The fix results in the macOS signature table reporting lines for each architecture within FAT bundled executables.

Improvements

We added lite-support for building the dependencies toolchain with GCC7.
The goal is to help folks building dependencies from source on Ubuntu 18.04 builds.

This also removes native compilation optimizations for RapidJSON.

#4437 Update AWS-SDK-CPP to version 1.4.55
#4439 Update libdpkg to version 1.19.0.5
#4440 Update The SleuthKit to version 4.6.1

#4393 Reduce drift time in query schedule

There was a minor unintentional drifting-effect on the query schedule.
This was adding slight delays to when queries are executed.

C++ extensions built using the external make target can now be bundled into a single executable.

Bug fixes

#3307 Various improvements to the python_packages table.
#4525 Address CVE-2018-6336 by making macOS signatures architecture-aware.

Table changes (from 3.2.6 to 3.2.7)

Added table battery to Darwin (Apple OS X)
Added table cpu_info to Microsoft Windows
Added table memory_array_mapped_addresses to POSIX-compatible Plaforms
Added table memory_arrays to POSIX-compatible Plaforms
Added table memory_device_mapped_addresses to POSIX-compatible Plaforms
Added table memory_error_info to POSIX-compatible Plaforms
Added table ulimit_info to POSIX-compatible Plaforms
Added column readonly_rootfs (INTEGER_TYPE) to table docker_containers
Added column directory (TEXT_TYPE) to table python_packages
Added column arch (TEXT_TYPE) to table signature

Lots of bug fixes!

Bug fixes

#4284 Improve yum_sources reporting
#4310 Fix unicode parsing errors in the configuration
#4341 Fix races in plugin methods (caused by extension registrations)
#4321 Improve EventData parsing in Windows Events
#4328 Fix various errors in the system_controls table on MacOS
#4374 Handle placeholder hardware UUIDs by using an ephemeral UUID
#4399 Fix socket-reuse after failed-connection segfault (large-bug!)
#4401 Fix debuginfo build-id paths
#4404 Fix over-release in disk_encryption on MacOS

Table Changes (from 3.2.4 to 3.2.6)

Added table user_groups to All Platforms (moved from POSIX)
Added table cups_destinations to Darwin (Apple OS X)
Added table cups_jobs to Darwin (Apple OS X)
Added table mdfind to Darwin (Apple OS X)
Added table startup_items to MacOS and Windows
Added table powershell_events to Microsoft Windows
Added table wmi_bios_info to Microsoft Windows
Added table memory_devices to POSIX-compatible Plaforms
Added table npm_packages to Linux
Added column encryption_method (TEXT_TYPE) to table bitlocker_info
Added column link_speed (BIGINT_TYPE) to table interface_details
Added column pci_slot (TEXT_TYPE) to table interface_details
Added column service (TEXT_TYPE) to table interface_details
Added column cgroup_namespace (TEXT_TYPE) to table processes
Added column ipc_namespace (TEXT_TYPE) to table processes
Added column is_elevated_token (INTEGER_TYPE) to table processes
Added column mnt_namespace (TEXT_TYPE) to table processes
Added column net_namespace (TEXT_TYPE) to table processes
Added column pid_namespace (TEXT_TYPE) to table processes
Added column user_namespace (TEXT_TYPE) to table processes
Added column uts_namespace (TEXT_TYPE) to table processes

osquery 3.2.4 release notes

This tag represents the first stable release of the osquery 3.0.0 series. The biggest change for 3.0.0 is a migration from boost property trees to Rapid JSON documents. This effects content in our RocksDB persistent store, and JSON interpretation of configuration and logging. Because of this migration we have introduced new database upgrading logic to automatically handle any subsequent database changes. This release also publishes the audit redesign first introduced in 3.1.0, as well as a variety of new tables for all platforms detailed below.

Finally, this release introduces numerous new unit and integration tests for various components of osquery. Going forward, we will be more strict about requiring integration or unit tests for new features introduced to the code base in an effort to make our product more reliable and robust.

New features in osquery 3

• We've migrated away from boost property trees in favor of RapidJSON objects. This migration resulted in massive performance gains for serialization to and from the database.
• The linux audit subsystem has been rearchitected to be more performant, reliably, and extensible.
• The osquery.io website has been overhauled! Use this as a landing portal for table schemas, package downloads, and any news round the product

Bug fixes

#4323 fix HANDLE leak in Windows processes functions
#4325 fix conversion of empty ptree to be empty RJ list
#4305 addressed memory leak in macos sip_config table
#4286 prevent runnable threads from deadlocking Windows service exit
#4276 ensure registry interface is thread safe
#4281 config parser keys are now objects or arrays
#4256 use specific release files in Linux os_version table
#4240 correctly divide uptime on Windows
#4236 ensure accelerated mode handles rapidjson correctly
#4234 filter process open sockets correctly when pid = -1
#4229 continue processing if a namespace lookup fails
#4222 fix crash in parsing stack traces for Windows crashes
#4125 fix leak in darwin disk_encryption table
#4169 correct external plugin name lookup
#4129 add loop detection to fs globbing
#4140 prevent duplicate build linkage by removing WEL as system logger
#4086 address RJ assertion failures in configuration
#4109 address sslv3 handshake failure in carver
#4051 fixes a crash in extended_attributes if file access fails due to permissions
#4047 fixes on_disk entry in processes table for linux

Table changes (from 2.11.2 to 3.2.4)

Added table account_policy_data to Darwin (Apple OS X)
Added table bitlocker_info to Microsoft Windows
Added table disk_info to Microsoft Windows
Added table kva_speculative_info to Microsoft Windows
Added table video_info to Microsoft Windows
Added table apt_sources to POSIX-compatible Plaforms
Added table yum_sources to POSIX-compatible Plaforms
Added table process_file_events to Ubuntu, CentOS

Added column serial (TEXT_TYPE) to table certificates
Added column cgroup_namespace (TEXT_TYPE) to table docker_containers
Added column config_entrypoint (TEXT_TYPE) to table docker_containers
Added column env_variables (TEXT_TYPE) to table docker_containers
Added column finished_at (TEXT_TYPE) to table docker_containers
Added column ipc_namespace (TEXT_TYPE) to table docker_containers
Added column mnt_namespace (TEXT_TYPE) to table docker_containers
Added column net_namespace (TEXT_TYPE) to table docker_containers
Added column path (TEXT_TYPE) to table docker_containers
Added column pid (BIGINT_TYPE) to table docker_containers
Added column pid_namespace (TEXT_TYPE) to table docker_containers
Added column privileged (INTEGER_TYPE) to table docker_containers
Added column security_options (TEXT_TYPE) to table docker_containers
Added column started_at (TEXT_TYPE) to table docker_containers
Added column user_namespace (TEXT_TYPE) to table docker_containers
Added column uts_namespace (TEXT_TYPE) to table docker_containers
Added column signed (INTEGER_TYPE) to table drivers
Added column fd (BIGINT_TYPE) to table listening_ports
Added column net_namespace (TEXT_TYPE) to table listening_ports
Added column path (TEXT_TYPE) to table listening_ports
Added column socket (BIGINT_TYPE) to table listening_ports
Added column net_namespace (TEXT_TYPE) to table process_open_sockets
Added column state (TEXT_TYPE) to table process_open_sockets
Added column disk_bytes_read (BIGINT_TYPE) to table processes
Added column disk_bytes_written (BIGINT_TYPE) to table processes
Added column cpu_microcode (TEXT_TYPE) to table system_info

Removed table apt_sources from Ubuntu, CentOS

See the 3.0.0 release notes about the 3.0 series!

This release includes the Linux Audit redesign. This redesign is faster, more reliable, and more extensible!

Welcome to the 3.0.0 series! In this series we'll be moving fast to incorporate new features that improve performance and safety. Minor releases will indicate new landed features. We'll highlight what to expect for compatibility in the release notes for each version.

In this kick-off tag, we're ratcheting the build "runtime" that is installed with make deps. On macOS and Linux this is completely rebuilt to minimize the final binary size. We have also nitpicked compatibility options for macOS and believe this version is much safer for older versions, below 10.13. Finally, this version pays attention to OS and package manager maintainers. It will be a struggle to find the correct dependencies, but 3.0.0 supports a traditional cmake build if the SKIP_DEPS environment variable exists.

This is a small release that adds mitigations for #3984.

It also includes a new crashes table for Windows, a bugfix #4022 for startup_items not including non-existent paths, and upgraded our internal dependencies for boost (1.66) and thrift (0.11).

This release is also the first using the new ASL2.0 and GPL2 dual license.

This tag includes dependency changes to accommodate Homebrew builds.

New features in 2.11.0

This version adds more features to osquery extensions. For a few examples, the Thrift API
calls now enforce a 5 minutes maximum execution time to protect osquery from hung
extensions (#3847); extension processes that are autoloaded, will respawn if they exit
prematurely (#3944).

We now depend on the newest libaugeas and have altered our integration to achieve
much better performance (#3911). Several changes in the new Augeas version were designed for
osquery's use cases.

Finally, along with the bug and features below, this version adds more care to Windows
Services and MSI packaging (#3927).

#3921 Kafka SSL support
#3814 Hash table cache
#3887 Windows Event Log (as a logger plugin) support
#4005 Non-blacklistable queries

Bug fixes

#3909 Print correct address family id for AF_UNIX sockets
#3938 Remove 'removed' results correctly
#3943 Stop renaming worker and extension argv[0]
#3958 Fix header calculation with HTTP client and AWS Firehose
#3979 Only daemon-reload if systemd is running
#3985 Removing newline from Windows Event Log lines
#4001 Remove invalid assumptions about status logging (refactor status logging)

Table changes (from 2.10.2 to 2.11.0)

Added table groups to All Platforms
Added table intel_me_info to Linux and Windows
Added table shadow to Linux
Added column blacklisted (INTEGER_TYPE) to table osquery_schedule
Added column install_location (TEXT_TYPE) to table programs
Added column type (TEXT_TYPE) to table users
Renamed table key_events to user_interaction_events on MacOS