SCARY is a software tool aimed at providing infrastructure owners the ability to mimic production workloads to a secondary server under test. The outcomes are recorded data on the execution of production queries and the execution on the target server. With this data, infrastructure owners are able to make informed choices about changing their production environments on the only benchmark that matters, their production workload.
The target server can be different hardware, configuration, maybe a version bump, or maybe a compatible product.
Types of comparisons can include:
Base/Base Secondary:
Target Version:
Recording:
With some testing around table syntax compatibility, this limitation may be removed.
The initial implementation does a SELECT FROM INFORMATION_SCHEMA.PROCESSLIST
on the Base server. It's envisaged that after some time, the sampling will be sufficient to be considered a full representation of the Base server.
Limitations:
The SCARY Processor take the data from the agent, stores this to the Recording server, and dispatches queries to the Target Server. As data on the Base may be incomplete, missing results and complete timing/query information.
Communication from the Agent to the SCARY Processor is via a Kafka Message Log. This is to ensure a reliable persistent message data that makes the Agent free of complexity and delays, and provides a work-queue buffer for the SCARY processor.
It's hoped other proxy implementations, like ProxySQL and MaxScale, can be extended to provide a rich set of data on proxied queries, which will limit the SCARY Processor to performing the Target-based queries and recording.
A message queue (via Kafka) from SCARY to the Agent and other sources will include a message digest of queries that have sufficient sampling and need not be recorded further.
Data is stored on the Recording Server in a number of tables.
Test: start/stop time and description test being run. Globalvars: global variable information at the start/finish of the test (for consistency checks), of both Base and Target servers Globalstatus: global status information at the start/finish of the test of both Base and Target servers Queries: the recorded queries, their execution times and query plan on both Base and Target servers. Query: digest information of the query and the number of further desired samples of this query.
Q. Why SCARY? A. SQL Comparator Analysis, Reassure Yourself (SCARY), but really software changes are SCARY, so lets own the word.
Q. How did it happen? A. Its been on the list as something that was needed for a while. As many know, production workloads are hard to reproduce, so we have to accept them as being the only valid test. With funding from Wikimedia Foundation the work started from the ground up under an open source license.
Q. How is it licensed? A. SCARY components in this repository are available under the APACHE-2.0 license.
Q. Who developed this? A. MariaDB Foundation.
Q. Can I contribute? A. Yes! Create an issue and or pull request on what you would like to change.