q

Commits

• a1da26c: Create FUNDING.yml (Harel Ben-Attia)

New major version with lots of additions:

• Automatic Immutable Caching - Automatic caching of data files (into <my-csv-filename>.qsql files), with huge speedups for medium/large files. Enabled through -C readwrite or -C read

• Direct querying of standard sqlite databases - Just use it as a table name in the query. Format is select ... from <sqlitedb_filename>:::<table_name>, or just <sqlitedb_filename> if the database contains only one table. Multiple separate sqlite databases are fully supported in the same query.

• Direct querying of the qsql cache files - The user can query directly from the qsql files, removing the need for the original files. Just use select ... from <my-csv-filename>.qsql. Please wait for the next version before deciding that you don't need your original files anymore... This version contains major changes and might include subtle bugs in it.

• Revamped .qrc mechanism - allows opting-in to caching without specifying it in every query. By default, caching is disabled, for backward compatibility and for finding usability issues.

• Save-to-db is now reusable for queries - --save-db-to-disk option (-S) has been enhanced to match the new capabilities. You can query the resulting file directly through q, using the method mentioned above (it's just a standard sqlite database).

Breaking Changes:

• Support for file concatenation (filenameA+filenameB+...) has been dropped. Glob support (myfilename*.csv) is still supported.

• Some cases of "one column only" warnings have been dropped as well

• Only python3 is supported from now on - Shouldn't be an issue, since q is a self-contained binary executable which has its own python embedded in it. Internally, q is now packaged with Python 3.8. After everything cools down, I'll probably bump this to 3.9/3.10.

• Minimal Linux Version Bumped - Works with CentOS 8, Ubuntu 18.04+, Debian 10+. Currently only for x86_64. Depends on glibc version 2.25+. Haven't tested it on other architectures. Issuing other architectures will be possible later on

For details on the changes and the new usage, see here

The version is now built using the awesome pyoxidizer project.

• Automatic Immutable Caching - Automatic caching of data files (into <my-csv-filename>.qsql files), with huge speedups for medium/large files. Enabled through -C readwrite or -C read
• Direct querying of standard sqlite databases - Just use it as a table name in the query. Format is select ... from <sqlitedb_filename>:::<table_name>, or just <sqlitedb_filename> if the database contains only one table. Multiple separate sqlite databases are fully supported in the same query.
• Direct querying of the qsql cache files - The user can query directly from the qsql files, removing the need for the original files. Just use select ... from <my-csv-filename>.qsql. Please wait until the non-beta version is out before thinking about deleting any of your original files...
• Revamped .qrc mechanism - allows opting-in to caching without specifying it in every query. By default, caching is disabled, for backward compatibility and for finding usability issues.
• Save-to-db is now reusable for queries - --save-db-to-disk option (-S) has been enhanced to match the new capabilities. You can query the resulting file directly through q, using the method mentioned above (it's just a standard sqlite database).
• Only python3 is supported from now on - Shouldn't be an issue, since q is a self-contained binary executable which has its own python embedded in it. Internally, q is now packaged with Python 3.8. After everything cools down, I'll probably bump this to 3.9/3.10.
• Minimal Linux Version Bumped - Works with CentOS 8, Ubuntu 18.04+, Debian 10+. Currently only for x86_64. Depends on glibc version 2.25+. Haven't tested it on other architectures. Issuing other architectures will be possible later on

For details on the changes and the new usage, see here

The version is still in early testing, for two reasons:

• Completely new build and packaging flow - Using pyoxidizer
• It's a very large change in functionality, which might surface issues, new and backward compatibility ones

Please don't use it for production, until the final non-beta version is out

If you're testing it out, I'd be more than happy to get any feedback. Please write all your feedback in this issue, instead of opening separate issues. That would really help me with managing this.

There's a newer beta version available, please look at the releases page and take the latest beta version.

• Automatic Immutable Caching - Automatic caching of data files (into <my-csv-filename>.qsql files), with huge speedups for medium/large files. Enabled through -C readwrite or -C read
• Direct querying of standard sqlite databases - Just use it as a table name in the query. Format is select ... from <sqlitedb_filename>:::<table_name>, or just <sqlitedb_filename> if the database contains only one table. Multiple separate sqlite databases are fully supported in the same query.
• Direct querying of the qsql cache files - The user can query directly from the qsql files, removing the need for the original files. Just use select ... from <my-csv-filename>.qsql. Please wait until the non-beta version is out before thinking about deleting any of your original files...
• Revamped .qrc mechanism - allows opting-in to caching without specifying it in every query. By default, caching is disabled, for backward compatibility and for finding usability issues.
• Save-to-db is now reusable for queries - --save-db-to-disk option (-S) has been enhanced to match the new capabilities. You can query the resulting file directly through q, using the method mentioned above (it's just a standard sqlite database).
• Only python3 is supported from now on - Shouldn't be an issue, since q is a self-contained binary executable which has its own python embedded in it. Internally, q is now packaged with Python 3.8. After everything cools down, I'll probably bump this to 3.9/3.10.
• Minimal Linux Version Bumped - Works with CentOS 8, Ubuntu 18.04+, Debian 10+. Currently only for x86_64. Depends on glibc version 2.25+. Haven't tested it on other architectures. Issuing other architectures will be possible later on

For details on the changes and the new usage, see here

The version is still in early testing, for two reasons:

• Completely new build and packaging flow - Using pyoxidizer
• It's a very large change in functionality, which might surface issues, new and backward compatibility ones

Please don't use it for production, until the final non-beta version is out

If you're testing it out, I'd be more than happy to get any feedback. Please write all your feedback in this issue, instead of opening separate issues. That would really help me with managing this.

2.0.17 and 2.0.18 haven't been officially released, so this release contains both of their deliveries:

From 2.0.17:
Multiple usability additions and fixes

• Fixed multiline quoting issue with -W minimal
• Added -p/-P convenience flags for pipe-delimited files
• Added -L flag to list all user functions
• Added warning when -p/-t is used along with -d, or -P/-T along with -D
• Generalized sha() user function to provide other algorithms (old sha1() remains for backward compat)
• Added md5() user function
• md5 and sha both require encoding to be provided. Will be automated in the future
• Added sqrt() and power() user functions
• Added stdev_sample() and stdev_pop() user functions

From 2.0.18:

• q benchmark logic and results

Multiple usability additions and fixes

• Fixed multiline quoting issue with -W minimal
• Added -p/-P convenience flags for pipe-delimited files
• Added -L flag to list all user functions
• Added warning when -p/-t is used along with -d, or -P/-T along with -D
• Generalized sha() user function to provide other algorithms (old sha1() remains for backward compat)
• Added md5() user function
• md5 and sha both require encoding to be provided. Will be automated in the future
• Added sqrt() and power() user functions
• Added stdev_sample() and stdev_pop() user functions

Please download release 2.0.9 from the web site http://harelba.github.io/q/ which contains rpm and deb packages as well. Another option would be to download version 2.0.9 from the "releases" list here, but it doesn't contain any rpm or deb packaging, just the windows installation.

This release has one feature - Being able to write the data into an sqlite3 disk database.

Use -S <db-filename> in order to save the data, and use any sqlite3 client to analyze/process it.

This is the first actual release of 1.6.x (previous 1.6.x were used to test the new release process).

• Added parameter for controlling maximum field size and providing info when failed
• Added a flag to disable automatic column type detection
• Fixed beautify when output headers exist
• Solved the following bugs:
  • #89 - Output formatting attempts to apply to header row
  • #115 - Fixed bug in escaping double-quotes
  • #116 - Test breaks when running on a Mac
  • #126 - Fixed bug with handling of null values in relaxed mode (exposed by #126)
  • #122 - Empty data and missing header behavior fixed
  • #132 - -b -D'|' not applied to -O header line
  • #131 - -b flag fails when no results match query
  • #114 - "Crashes" when conditions don't match any rows.
• Merged the following PRs:
  • #137 - Unit tests with Travis CI - Thanks @serima !
  • #107 - update query parameter - Thanks @jungle-boogie !
  • #103 - Simplified mac homebrew version bump instructions - Thanks @bfontaine !
• Version release process changes
• Doc changes

• Full input/output support for double-quoting fields with delimiters
• Multiple query support in one command line, reusing previously loaded data in subsequent queries
• Support literal SELECT statements (e.g. SELECT 5+12.5/3)
• Full code restructuring (Internally working now using a full python API which will become public in the next version)
• Added sha1 function. You can use 'select sha1(c3) from ...'
• Solved the following bugs/pull-requests:
  • #10 - Reuse of previously loaded data when running multiple queries in one command line
  • #64 - Full support for literal SELECT statements without a table (e.g. SELECT 5+12.5)
  • #56 - Proper double quote handling, including multiline fields, for both input and output according to csv standards
  • #69 - Added warning suppression when the user provides a specific column count
  • #40 - Code restructuring cleaning, creating a full python API (will be published in the next version)
  • #60 - Fixed RPM packaging
  • #68 - UTF-8 with BOM files cause column naming issues
  • #63 - Unicode string support in regexp function