scipipe

A small release adding two quite minor enhancements:

• be6953 - Include the actual command in error message for failing commands
• bdba4d - Add GetSize() method to ips, to get file size for targets

This release adds two big improments:

1. The audit log feature will now include upstream audit info from tasks that are already finished when running the workflow, by reading in their (existing) audit log files. The field names have also been made shorter and more succinct.

2. Ports now support connecting multiple upstream ports to in-ports. This makes it much easier to create components for various merge and reduce operations, among other things. Note though that it will require to go-run port.RunMergeInputs() on all input ports, which contains code that will merge from multiple inputs into the ports InChan channel, which can still be used to do range operations on, for simplicity.

Full list of new commits in this release

• a584425e3387798026cecb48671d30df29f8adef Implement new feature: Allow multiple inputs in ports
• 067df02e52a6424271b0dff8a2396346af7c511b Add ReadAuditFile() method in ip
• 5a38927672b01bd25900c4b8c2f7793ac4613b1e New feature: Read upstream audit info from existing audit log files, for files that exist
• ff94caf9c950afe39cfe1d3339671c37ceb018af Shorten method name: Port.{ConnectStrings -> ConnectStr}()
• 4433cf55b1a3e1c7967cad11b8129073b17f0863 API change: Shorter more sensible field names in audit log
• 55a193aea2c2e965c79b56a5f8136d00501630e6 Back to not create channel for new paramports, except in ConnectStrings(), to not trip up connect logic
• 0a4fb82784d0be51395c2c3f69268a8db9ffeb3a API change in task: new Param() accessor method
• 0d38b89c07d91e9935da66fe8c706a7bbce43ddd API change in task: GetInPath()->InPath()
• 3dd17bae89633ac833a96db6907ec73f54dcb673 New convenience method ParamPort.ConnectStrings() for simply sending one or more strings
• 183eab934d3f86e93789926b22cdb33085788acc Create chan in NewParamPort

This release contains two major new API changes:

1. NewWorkflow() now takes a number telling the max number of tasks to run concurrently, so that your computer's CPU is not overbooked. So, now you might want to do e.g. NewWorkflow("some_wf", 4), on a computer with 4 CPU cores.

2. The NewProc() command, and constructor methods for most tasks now take the workflow as its first parameter (this is not the case if using the workflow.NewProc() version, which does this automatically anyway).

This is to resolve a very commonly occuring and annoying issue that one too easily forgets to add a component to a workflow, which there is no warning for either (checking this is quite hard to implement without resorting to global variables).

Finally, some bugs were resolved in how ports are connected, especially in some of the components in the components lib, which were not properly updated to use the new multi-send ports, and thus were closing the ports in the wrong way. So, now you can connect one out-port to multiple downstream in-ports without bugs and quirks.

This is a first release to start tracking changes using semantically versioned releases, for better clarity.

The current status is that the API seems to be stabilizing, but there might still be breaking changes as we put scipipe into production in the coming week.s