pash

High-level changes

Significant restructuring and rearchitecting of PaSh components into their own libraries.

What's Changed

• Support for spec by @angelhof in https://github.com/binpash/pash/pull/660
• Python dependency by @angelhof in https://github.com/binpash/pash/pull/661
• Reorganize pash_runtime by @angelhof in https://github.com/binpash/pash/pull/665
• Extend pash to support variable passing for spec by @angelhof in https://github.com/binpash/pash/pull/666
• whitespace to trigger CI by @angelhof in https://github.com/binpash/pash/pull/667
• Spec loop support by @angelhof in https://github.com/binpash/pash/pull/670
• Clean up shell AST by @angelhof in https://github.com/binpash/pash/pull/673
• Clean up ast modules and logging by @angelhof in https://github.com/binpash/pash/pull/674
• Use typed ASTs for all nodes throughput PaSh by @angelhof in https://github.com/binpash/pash/pull/675
• Prepare expand to be its own library by decoupling imports by @angelhof in https://github.com/binpash/pash/pull/676
• Move ast_node under shasta directory to prepare librarification by @angelhof in https://github.com/binpash/pash/pull/677
• Use shasta-lib from PyPI by @angelhof in https://github.com/binpash/pash/pull/678
• Clean up some obsolete code by @angelhof in https://github.com/binpash/pash/pull/679
• Make evaluation/intro more user-friendly by @nikpag in https://github.com/binpash/pash/pull/680
• Remove obsolete echo from installation by @angelhof in https://github.com/binpash/pash/pull/681
• Separate expand to its own library by @angelhof in https://github.com/binpash/pash/pull/682
• Simplify tests and installation by @angelhof in https://github.com/binpash/pash/pull/683
• New PaSh release by @angelhof in https://github.com/binpash/pash/pull/684

New Contributors

• @nikpag made their first contribution in https://github.com/binpash/pash/pull/680

Full Changelog: https://github.com/binpash/pash/compare/v0.12.2...v0.13

What's Changed

• Update installation script up.sh and README

Full Changelog: https://github.com/binpash/pash/compare/v0.12...v0.12.2

Changes

• Separate PaSh's preprocessor and add an alternative entrypoint
• Add alternative entrypoints to PaSh
• Remove golang dependencies

What's Changed in detail

• Cleanup by @angelhof in https://github.com/binpash/pash/pull/650
• Allow preprocessor to extract sequence of commands by @angelhof in https://github.com/binpash/pash/pull/651
• Add df regions to their own files and only edges to the po file by @angelhof in https://github.com/binpash/pash/pull/652
• Add a speculative entrypoint to PaSh by @angelhof in https://github.com/binpash/pash/pull/653
• Remove golang code to avoid dependabot alerts by @angelhof in https://github.com/binpash/pash/pull/655
• PaSh version 0.12 by @angelhof in https://github.com/binpash/pash/pull/656

Full Changelog: https://github.com/binpash/pash/compare/v0.11...v0.12

Big Changes

• PaSh now uses the new annotations-as-code library (https://github.com/binpash/annotations)
• Remove a lot of obsolete files and restructure PaSh code in subdirectories

Detailed view of what's Changed

• First PR for new annotations and new model for DFG nodes by @festutz in https://github.com/binpash/pash/pull/583
• Add support for eager-nodes by @festutz in https://github.com/binpash/pash/pull/589
• Add reduce/aggregation trees by @festutz in https://github.com/binpash/pash/pull/590
• Support for round-robin parallelization with new annotations by @festutz in https://github.com/binpash/pash/pull/591
• Fix unwrap and commutative interplay by @festutz in https://github.com/binpash/pash/pull/593
• Add support for dgsh_tee nodes by @festutz in https://github.com/binpash/pash/pull/594
• Modify dgsh-wrapper ... by @festutz in https://github.com/binpash/pash/pull/595
• Remove dgsh-tee wrapper and call dgsh-tee directly by @festutz in https://github.com/binpash/pash/pull/596
• Separated parallelization into choose and apply phases by @festutz in https://github.com/binpash/pash/pull/597
• Fuse cat and subsequent split by @festutz in https://github.com/binpash/pash/pull/599
• Fuse r_merge and subsequent r_split by @festutz in https://github.com/binpash/pash/pull/600
• Fuse r_merge and subsequent commutative command by @festutz in https://github.com/binpash/pash/pull/601
• Support round-robin parallelization for commutative commands by @festutz in https://github.com/binpash/pash/pull/602
• Install annotations lib using pip by @angelhof in https://github.com/binpash/pash/pull/603
• Refactored to remove future from annotations library by @festutz in https://github.com/binpash/pash/pull/609
• Cover all but one test case from compiler/test_evaluation_scripts.sh by @festutz in https://github.com/binpash/pash/pull/612
• Add support for bigrams by @festutz in https://github.com/binpash/pash/pull/614
• Refactor to have defaults for AnnotationInfo by @festutz in https://github.com/binpash/pash/pull/615
• Future annotations minor typing anno repo by @festutz in https://github.com/binpash/pash/pull/622
• Integrate New Annotation Library by @festutz in https://github.com/binpash/pash/pull/623
• Annotations as package by @festutz in https://github.com/binpash/pash/pull/634
• Remove custom aggregators by @angelhof in https://github.com/binpash/pash/pull/639
• Trigger CI on scripts changes by @angelhof in https://github.com/binpash/pash/pull/642
• Delete files and references to old annotations by @angelhof in https://github.com/binpash/pash/pull/640
• Refactor components by @angelhof in https://github.com/binpash/pash/pull/636
• Remove more obsolete files in compiler, move tests to the tests directory by @angelhof in https://github.com/binpash/pash/pull/643
• Remove the last annotation files by @angelhof in https://github.com/binpash/pash/pull/644
• Refactor all shell ast related code to its own directory (ideally being mostly self contained) by @angelhof in https://github.com/binpash/pash/pull/645
• Refactor orchestor runtime by @angelhof in https://github.com/binpash/pash/pull/646
• PaSh version 0.11 by @angelhof in https://github.com/binpash/pash/pull/647

Full Changelog: https://github.com/binpash/pash/compare/v0.10.2...v0.11

What's Changed

• Docker image build fixes by @angelhof in https://github.com/binpash/pash/pull/631

Full Changelog: https://github.com/binpash/pash/compare/v0.10.1...v0.10.2

What's Changed

Removed installation script dependencies and made their output more verbose to improve debuggability.

• Remove pexpect and add a deprecation warning by @angelhof in https://github.com/binpash/pash/pull/628
• Cleanup setup scripts by @angelhof in https://github.com/binpash/pash/pull/627
• Update debian docker image to have python3.8 by @angelhof in https://github.com/binpash/pash/pull/630

Full Changelog: https://github.com/binpash/pash/compare/v0.10...v0.10.1

Biggest change

Removed libdash from the PaSh repo instead installing it through pip

What's Changed

• Run tests in Ubuntu 18.04 and 20.04 to completion by @mgree in https://github.com/binpash/pash/pull/617
• Don't run PaSh Github CI on Ubuntu 18.04 due to deprecation by @angelhof in https://github.com/binpash/pash/pull/618
• Temporarily remove Ubuntu 22.04 from the CI until it is debugged by @angelhof in https://github.com/binpash/pash/pull/621
• Add a python script to contact the POSIX test server by @angelhof in https://github.com/binpash/pash/pull/624
• Remove compiler/parser subtree, use libdash pip module by @mgree in https://github.com/binpash/pash/pull/616

Full Changelog: https://github.com/binpash/pash/compare/v0.9...v0.10

What's Changed

At a high level, several bugfixes with respect to argument handling and escaping. Installation is also simplified.

• Added annotation for rev by @tammam1998 in https://github.com/binpash/pash/pull/563
• Clean up and refactor input generation to avoid several issues with artifact by @angelhof in https://github.com/binpash/pash/pull/564
• Exclamation escaping by @angelhof in https://github.com/binpash/pash/pull/574
• Fix a fd leak issue by @angelhof in https://github.com/binpash/pash/pull/575
• Remove an outdated TODO comment by @a5ob7r in https://github.com/binpash/pash/pull/580
• Append to pythonpath instead of setting it by @angelhof in https://github.com/binpash/pash/pull/579
• Installation fixes by @nvasilakis in https://github.com/binpash/pash/pull/581
• Fix redir variable parsing bug by @angelhof in https://github.com/binpash/pash/pull/582
• add --depth=1 flag to git clone command by @averykhoo in https://github.com/binpash/pash/pull/588
• More installation fixes by @nvasilakis in https://github.com/binpash/pash/pull/584
• Fix argument handling by @angelhof in https://github.com/binpash/pash/pull/578
• Drop function kw by @mgree in https://github.com/binpash/pash/pull/604
• Remove an invalid trailing comma in an annotation by @a5ob7r in https://github.com/binpash/pash/pull/605
• PaSh version 0.9 by @angelhof in https://github.com/binpash/pash/pull/611

New Contributors

• @averykhoo made their first contribution in https://github.com/binpash/pash/pull/588

Full Changelog: https://github.com/binpash/pash/compare/v0.8...v0.9

Big changes

• Shellcheck is now part of the CI
• PaSh early expansion is now correctly pure
• OSDI 22 Artifact
• Various fixes

Full Changelog: https://github.com/binpash/pash/compare/v0.7.2...v0.8

This is the branch that contains the artifact for the paper "Practically Correct, Just-in-Time Shell Script Parallelization", published at OSDI 22.

What's Changed

• CI Fix to not push multiple images on Github Images by @angelhof in https://github.com/binpash/pash/pull/546

Full Changelog: https://github.com/binpash/pash/compare/v0.7.1...v0.7.2

What's Changed

Minor installation/CI changes

• Auto-build docker images on new releases and push to dockerhub by @dkarnikis in https://github.com/binpash/pash/pull/521
• correct repo name by @dkarnikis in https://github.com/binpash/pash/pull/522
• Pipelines changes by @dkarnikis in https://github.com/binpash/pash/pull/523
• Trigger performance tests only by @dkarnikis in https://github.com/binpash/pash/pull/524
• fetch correct posix table by @dkarnikis in https://github.com/binpash/pash/pull/525
• Add graphviz in setup_pash by @angelhof in https://github.com/binpash/pash/pull/541

Full Changelog: https://github.com/binpash/pash/compare/v0.7...v0.7.1

This is the state of the repo (version 0.7) as of April 2022

Changelog - Commits

• [594fe3cf97b19cea0f8e4d72b9a0c31134544af7] - Add Individual aggregation tests
• [650b184cf6e8686b8cc0ee3aa8fc54505f5b1b56] - Address a bug related to auto-split
• [e954e30c8eeb38cb3da93dfe1727685d4e98521b] - Fix setup and installation process
• [8f455fd0645eb36b0322a8be84cf790539d2d76b] - Custom pipeline to issue POSIX test
• [11bab5a681acfb53ec05724225bf1621dec609c0] - Upload the execution log artifact of the POSIX tests on the Github Servers
• [c11b67ec53af94c03ecf5a0a7502837fba465d07] - Fix execute_pash_and_check_diff on evaluation scripts
• [4f2c79d80b85ec68934d57fed21b810bf6078516] - Add small input dataset for all benchmarks (--small flag)
• [8752f5836e4aa7fc430abd8178fd82befdc289aa] - Kill all child processes when triggering a signal
• [4f005775ef47c9b1cd5c0a1024781042249e242c] - Fix bug caused by early kill of processes
• [10a842f6ed4f49b91a046de8d7078d7a1237749f] - Evaluation script to run all the benchmarks and generate PDF plots
• [5a4e3f652780661f6b7bed9ba548d606cde3bd2d] - Add usage and parse argument functions
• [c1cb6a633569d167495297fa7b219cf3f52930e0] - DFGs are output as dot graphs
• [37071ab2af26b4fe5821fdec146fe5c29c926c6e] - Change the label of r_wrap dot nodes to include the wrapped command
• [8d0a90491bb49c4e256ea62ead9fb6404362f801] - Update Dockerfiles to match the new setup process
• [033ee4e626ffaa6fbabe0da379f28bf7073f7a69] - Add action badges for pages,build and correctness tests for master/develop (main/future)
• Other CI, bug fixes and documentation updates

This is the state of the repo (version 0.6) as of December 2021

Changelog - Commits

• [3495cab0e9f972853250571829967c3226e2409c] - Parallel pipelines implementation
• [65ed28b43cfd43c1de887ad164bd3703b9d13442] - Make sure that pash finishes execution if the daemon crashes
• [3ed1a38718c5f9cce4d003a66f9ef593bdc32b05] - Add buffer to scheduler to deal with multiple inputs
• [751a2612c0c38a8d86bb2f0c2cd64e08295eadef] - Make a first iteration of a profile-driven width lowering
• [8308421] - Gather pids to kill when spawning
• [9b18a84d333addc59dda99bda2f10eea49c5b19e] - Added logic to make sure to improve performance when input is small
• [65e0de6b7219ab2d65bdb649fc269d6bd0d7135b] - Daemon communicates through a socket instead of fifos
• [8edcbaa2e164a0226f777c393f94bc2f8e914ace] - Add a mini-experiment that checks the runtime overhead in a tight loop
• [7704d79d2659a2f2e64ec9a57ab87ca3f4ab28da] - Exposing the exit status of pash-compiled pipelines
• [fc5ba6fdb375ab1e1ad5f15cf33d92504438df59] - Add CI jobs for future branch
• [487cc93ae9d02b2fa7b90c957329e694cf6e2747] - For loops benchmark set
• [3182e4da85b7db7f9f6e71534392338fdf1d6930] - Alternative syntax for poets
• [49052eba222edbce31610049c01a81608e12a3ca] - Fix an issue when umask is very restrictive when invoking pash
• Other CI and Bug Fixes

This is the state of the repo (version 0.5) as of October 2021

Changelog - Commits

• [59f6c92] - Redirect the output of pash_runtime to the log
• [a598bf6] - Extend PaSh to be able to handle Redir shell ast nodes
• [6969672] - Make parsing errors non-silent
• [bcfc1e1] - Fix an issue with set
• [9cad2ca] - PaSh support for interactive execution
• [d1d086a] - Add support for break
• [3612702] - Improve performance of tempfile
• [912865f] - Detached runtime worker to avoid python startup costs
• [aea588b] - Reduce the number of calls related to distro discovery
• [c8b463d] - Pull part of the runtime processes in pa.sh to happen once
• [355bca4] - Python optimizations to reduce startup time
• Other Bug Fixes

This is the state of the repo (version 0.4) as of September 2021

Changelog - Commits

• [2b4db9ddb1616ce6fe56575724c966177bb950f7] - Create Code of Conduct
• [92cbe089fcd839d889ba6b19c6dc0686d5f266d1] - Correct tr in poets
• [4dbc0a76cd7f68f566286c4cc2fbeb1d97df080f] - Remove old testing infrastructure for aggregators
• [348cc7820470ce86ad1cca5cbd35b7dda6391e96] - Add new aggregators and custom tests
• [154d9fc42f5f34d4ef681401e252abf54517be3b] - Fix space in wf's tr
• [46bb9da11b6371c1f51f45a816aad6ae40b2682e] - Add a custom aggregator tutorial
• [9b2d76c48aa0509aaaf5e0428e2ae09440c4d069] - Fixed CI/CD for forks
• [5ec7666f9d44d03aa544cd769f8991f795d98bfc] - Fix paths CI bot
• [f2a181ae6b3d70119d71d13653125ebaa474d6cf] - docs: Hide comment
• [5f75442e6cd110aa73b4a0ac5b67ed3656ea78fc] - Update docs
• [a58e927e0ad5f60eebab02290af6eb39dee8990a] - Docker docs
• [48ef8825140e66457c2607fa940e0f1e3d320879] - Update tutorial.md
• [555c3eb40a76a7c1afa94c180e3a8beea20b8b31] - Annotation updates
• [234fbd44c22653e42e91f425e5e238624c6ad3dc] - Dockerfiles for ubuntu, debian, fedora
• [958a2cd1f14db6b3fa8c392b684a66b2c6768796] - Fix to match new CI/CD interface
• [73695f5501e6350bd0b074985f01be78e1eb4301] - CI/CD test parsing fixes. WIP
• [75d02a5bcb1734b4be77b9b5805c8a10aac4fad3] - Move package installation to distro-deps
• Other Bug Fixes

This is the final state of the repo at the time of the EuroSys 2021 AEC acceptance.