#+TITLE: Brish

#+begin_html

• Guide
  ** Installation

pip install -U brish

Or install the latest master (recommended, as I might have forgotten to push a new versioned update):

pip install git+https://github.com/NightMachinary/brish

You need a recent Python version, as Brish uses some of the newer metaprogramming APIs. Obviously, you also need zsh installed.

** Quickstart

#+begin_src python :session p1 :results silent :tangle tests/test_tangled1.py from brish import z, zp, Brish #+end_src

#+begin_src python :session p1 :results silent :exports none :tangle tests/test_tangled1.py NI = True #+end_src

#+begin_src python :session p1 :results silent :exports none NI = False #+end_src

#+name: t1 #+begin_src python :session p1 :results value :exports both :tangle tests/test_tangled1.py name="A$ron" z("echo Hello {name}") #+end_src

#+RESULTS: t1 #+begin_example Hello A$ron #+end_example

#+begin_src python :session p1 :var t1=t1 :results value :exports none :tangle tests/test_tangled1.py def test1(): assert t1 == "Hello A$ron" return True NI or test1() #+end_src

#+RESULTS: #+begin_example True #+end_example

z automatically converts Python lists to shell lists: #+name: t2 #+begin_src python :session p1 :results value :exports both :tangle tests/test_tangled1.py alist = ["# Fruits", "1. Orange", "2. Rambutan", "3. Strawberry"] z("for i in {alist} ; do echo $i ; done") #+end_src

#+RESULTS: t2 #+begin_example

Fruits

1. Orange
2. Rambutan
3. Strawberry
   #+end_example

#+begin_src python :session p1 :var t2=t2 :results value :exports none :tangle tests/test_tangled1.py def test2(): assert t2 == """# Fruits

1. Orange
2. Rambutan
3. Strawberry"""
   NI or test2()
   #+end_src

#+RESULTS: #+begin_example None #+end_example

z returns a CmdResult (more about which later):

#+begin_src python :session p1 :results value :exports both res = z("date +%Y") repr(res) #+end_src

#+RESULTS: #+begin_example CmdResult(retcode=0, out='2021\n', err='', cmd=' date +%Y ', cmd_stdin='') #+end_example

You can use zp as a shorthand for print(z(...).outerr, end=''):

#+begin_src python :session p1 :results output :exports both for i in range(10): cmd = "(( {i} % 2 == 0 )) && echo {i} || {{ echo Bad Odds'!' >&2 }}" # Using {{ and }} as escapes for { and } zp(cmd) print(f"Same thing: {z(cmd).outerr}", end='') #+end_src

#+RESULTS: #+begin_example 0 Same thing: 0 Bad Odds! Same thing: Bad Odds! 2 Same thing: 2 Bad Odds! Same thing: Bad Odds! 4 Same thing: 4 Bad Odds! Same thing: Bad Odds! 6 Same thing: 6 Bad Odds! Same thing: Bad Odds! 8 Same thing: 8 Bad Odds! Same thing: Bad Odds! #+end_example

CmdResult is true if its return code is zero: #+name: t3 #+begin_src python :session p1 :results output :exports both :tangle tests/test_tangled1.py if z("test -e ~/"): print("HOME exists!") else: print("We're homeless :(") #+end_src

#+RESULTS: t3 #+begin_example HOME exists! #+end_example

#+begin_src python :session p1 :var t3=t3 :results value :exports none :tangle tests/test_tangled1.py assert t3 == "HOME exists!" #+end_src

#+RESULTS:

CmdResult is smart about iterating: #+name: t4 #+begin_src python :session p1 :results output :exports both :tangle tests/test_tangled1.py for path in z("command ls ~/tmp/"): # command helps bypass potential aliases defined on ls zp("du -h ~/tmp/{path}") # zp prints the result #+end_src

#+RESULTS: t4 #+begin_example 524K /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97 524K /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82 1.3M /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr/dreamcorp420 1.3M /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr 0B /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/34cc02221710caf309bff5ca96808d7a 520K /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1 2.9M /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43 0B /Users/evar/tmp/8826215ac0ed61d617906f658322fce7 348K /Users/evar/tmp/IMG_0396.PNG 44K /Users/evar/tmp/a2.jpg 40K /Users/evar/tmp/a4.jpg 8.0K /Users/evar/tmp/bills 0B /Users/evar/tmp/garden 468K /Users/evar/tmp/image-14000213234237913.png 40K /Users/evar/tmp/photo_2021-05-08_00-35-24.jpg 152K /Users/evar/tmp/photo_2021-05-08_00-55-29.jpg 8.0K /Users/evar/tmp/tumblr 4.0M /Users/evar/tmp/tumblr_2c0ad7a3fba563996c9abaedc5e8d4f7_356ef3d9_1280.gif 576K /Users/evar/tmp/tumblr_5a2868650b058c42a7d141b8a2f474bc_eac04dc0_1280.jpg 976K /Users/evar/tmp/tumblr_5cc2e0e48418ec3c9eb200d151daf647_e44e419b_1280.jpg 44K /Users/evar/tmp/tumblr_6c90d77a676cf20fc096cc19220af4ab_e124dbec_540.gif.mp4 0B /Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg 4.0K /Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg.aria2 656K /Users/evar/tmp/tumblr_bc2259b471f792065eb6707b7c29d27e_97f97d94_1280.jpg 1.0M /Users/evar/tmp/tumblr_ec36fde70ee0264cdc2f61f394181c61_575227e7_1280.jpg 84K /Users/evar/tmp/view.php #+end_example

#+begin_src python :session p1 :results value :exports both res = z("""echo This is stdout echo This is stderr >&2 (exit 6) # this is the return code""") repr(res.out) #+end_src

#+RESULTS: #+begin_example This is stdout\n #+end_example

CmdResult.outrs strips the final newlines:

#+begin_src python :session p1 :results value :exports both repr(res.outrs) #+end_src

#+RESULTS: #+begin_example This is stdout #+end_example

#+begin_src python :session p1 :results value :exports both repr(res.err) #+end_src

#+RESULTS: #+begin_example This is stderr\n #+end_example

#+begin_src python :session p1 :results value :exports both res.retcode #+end_src

#+RESULTS: #+begin_example 6 #+end_example

#+begin_src python :session p1 :results value :exports both res.longstr #+end_src

#+RESULTS: #+begin_example

cmd: echo This is stdout echo This is stderr >&2 (exit 6) # this is the return code stdout: This is stdout

stderr: This is stderr

return code: 6 #+end_example

By default, z doesn't fork. So we can use it to change the state of the running zsh session: #+begin_src python :session p1 :results value :exports both z(""" (($+commands[imdbpy])) || pip install -U imdbpy imdb() imdbpy search movie --first "$*" """) z("imdb Into the Woods 2014") #+end_src

#+RESULTS:
#+begin_example
Movie

Title: Into the Woods (2014) Genres: Adventure, Comedy, Drama, Fantasy, Musical. Director: Rob Marshall. Writer: James Lapine, James Lapine. Cast: Anna Kendrick (Cinderella), Daniel Huttlestone (Jack), James Corden (Baker / Narrator), Emily Blunt (Baker's Wife), Christine Baranski (Stepmother). Runtime: 125. Country: United States. Language: English. Rating: 5.9 (134093 votes). Plot: A witch tasks a childless baker and his wife with procuring magical items from classic fairy tales to reverse the curse put on their family tree. #+end_example

We can force a fork. This is useful to make your scripts more robust. #+begin_src python :session p1 :results output :exports both print(z("exit 7", fork=True).retcode) zp("echo 'Still alive!'") #+end_src

#+RESULTS: #+begin_example 7 Still alive! #+end_example

Working with stdin: #+begin_src python :session p1 :results value :exports both

the intuitive way

a="""1 2 3 4 5 """ z("<<<{a} wc -l") #+end_src

#+RESULTS: #+begin_example 6 #+end_example

#+begin_src python :session p1 :results value :exports both z("wc -l", cmd_stdin=a) #+end_src

#+RESULTS: #+begin_example 5 #+end_example

** More Details The stdin will by default be set to the empty string: #+begin_src python :session p1 :results output :exports both zp("cat") zp("echo 'As you see, the previous command produced no output. It also did not block.'") #+end_src

#+RESULTS: #+begin_example as you see, the previous command produced no output. It also did not block. #+end_example

z escapes your Python variables automagically: #+begin_src python :session p1 :results value :exports both python_var = "$HOME" z("echo {python_var}") #+end_src

#+RESULTS: #+begin_example $HOME #+end_example

Turning off the auto-escape: #+begin_src python :session p1 :results value :exports both z("echo {python_var:e}") #+end_src

#+RESULTS: #+begin_example /Users/evar #+end_example

Working with Python bools from the shell: #+begin_src python :session p1 :results value :exports both z("test -n {True:bool}").retcode #+end_src

#+RESULTS: #+begin_example 0 #+end_example

#+begin_src python :session p1 :results value :exports both z("test -n {False:bool}").retcode #+end_src

#+RESULTS: #+begin_example 1 #+end_example

Working with NUL-terminated output: #+begin_src python :session p1 :results output :exports both for f in z("fd -0 . ~/tmp").iter0(): zp("echo {f}") #+end_src

#+RESULTS: #+begin_example /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43 /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/34cc02221710caf309bff5ca96808d7a /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82 /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82/tumblr_9527f4f6d2f1a39ef2b839780831f38f_859e5e2b_2048.jpg /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82/tumblr_dd64a6ced93d19ffe78b47cf3439373d_e8e18fb0_2048.jpg /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1 /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1/tumblr_9527f4f6d2f1a39ef2b839780831f38f_859e5e2b_2048.jpg /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1/tumblr_dd64a6ced93d19ffe78b47cf3439373d_e8e18fb0_2048.jpg /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97 /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97/tumblr_9527f4f6d2f1a39ef2b839780831f38f_859e5e2b_2048.jpg /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97/tumblr_dd64a6ced93d19ffe78b47cf3439373d_e8e18fb0_2048.jpg /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr/dreamcorp420 /Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr/dreamcorp420/tumblr_dreamcorp420_650543836474589184_01.gif /Users/evar/tmp/8826215ac0ed61d617906f658322fce7 /Users/evar/tmp/IMG_0396.PNG /Users/evar/tmp/a2.jpg /Users/evar/tmp/a4.jpg /Users/evar/tmp/bills /Users/evar/tmp/garden /Users/evar/tmp/image-14000213234237913.png /Users/evar/tmp/photo_2021-05-08_00-35-24.jpg /Users/evar/tmp/photo_2021-05-08_00-55-29.jpg /Users/evar/tmp/tumblr /Users/evar/tmp/tumblr_2c0ad7a3fba563996c9abaedc5e8d4f7_356ef3d9_1280.gif /Users/evar/tmp/tumblr_5a2868650b058c42a7d141b8a2f474bc_eac04dc0_1280.jpg /Users/evar/tmp/tumblr_5cc2e0e48418ec3c9eb200d151daf647_e44e419b_1280.jpg /Users/evar/tmp/tumblr_6c90d77a676cf20fc096cc19220af4ab_e124dbec_540.gif.mp4 /Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg /Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg.aria2 /Users/evar/tmp/tumblr_bc2259b471f792065eb6707b7c29d27e_97f97d94_1280.jpg /Users/evar/tmp/tumblr_ec36fde70ee0264cdc2f61f394181c61_575227e7_1280.jpg /Users/evar/tmp/view.php #+end_example

You can bypass the automatic iterable conversion by converting the iterable to a string first: #+begin_src python :session p1 :results value :exports both z("echo {' '.join(map(str,alist))}") #+end_src

#+RESULTS: #+begin_example

Fruits 1. Orange 2. Rambutan 3. Strawberry

#+end_example

Normal Python formatting syntax works as expected:

#+begin_src python :session p1 :results value :exports both z("echo {67:f}") #+end_src

#+RESULTS: #+begin_example 67.0 #+end_example

#+begin_src python :session p1 :exports both :results verbatim z("echo {[11, 45]!s}") #+end_src

#+RESULTS: #+begin_example [11, 45] #+end_example

You can obviously nest your z calls: #+begin_src python :session p1 :results value :exports both z("""echo monkey$'\n'{z("curl -s https://www.poemist.com/api/v1/randompoems | jq --raw-output '.[0].content'")}$'\n'end | sed -e 's/monkey/Random Poem:/'""") #+end_src

#+RESULTS: #+begin_example Random Poem: ’Tis said that the Passion Flower, With its figures of spear and sword And hammer and nails, is a symbol Of the Woe of our Blessed Lord. So still in the Heart of Beauty Has been hidden, since Life drew breath, The sword and the spear of Anguish, And the hammer and nails of Death. end #+end_example

*** The Brish Class z and zp are just convenience methods:

#+begin_example bsh = Brish() z = bsh.z zp = bsh.zp zq = bsh.zsh_quote zs = bsh.zstring #+end_example

You can use Brish instances yourself (all arguments to it are optional). The boot command boot_cmd allows you to easily initialize the zsh session:

#+begin_src python :session p1 :results value :exports both my_own_brish = Brish(boot_cmd="mkdir -p ~/tmp ; cd ~/tmp") my_own_brish.z("echo $PWD") #+end_src

#+RESULTS: #+begin_example /Users/evar/tmp #+end_example

Brish.z itself is sugar around Brish.zstring and Brish.send_cmd: #+begin_src python :session p1 :results value :exports both cmd_str = my_own_brish.zstring("echo zstring constructs the command string that will be sent to zsh. It interpolates the Pythonic variables: {python_var} {alist}") cmd_str #+end_src

#+RESULTS: #+begin_example echo zstring constructs the command string that will be sent to zsh. It interpolates the Pythonic variables: '$HOME' '# Fruits' '1. Orange' '2. Rambutan' '3. Strawberry' #+end_example

#+begin_src python :session p1 :results value :exports both my_own_brish.send_cmd(cmd_str) #+end_src

#+RESULTS: #+begin_example zstring constructs the command string that will be sent to zsh. It interpolates the Pythonic variables: $HOME # Fruits 1. Orange 2. Rambutan 3. Strawberry #+end_example

You can restart a Brish instance: #+begin_src python :session p1 :results output :exports both my_own_brish.z("a=56") my_own_brish.zp("echo Before restart: $a") my_own_brish.restart() my_own_brish.zp("echo After restart: $a") my_own_brish.zp("echo But the boot_cmd has run in the restarted instance, too: $PWD") #+end_src

#+RESULTS: #+begin_example Before restart: 56 After restart: But the boot_cmd has run in the restarted instance, too: /Users/evar/tmp #+end_example

Brish is threadsafe. I have built [[https://github.com/NightMachinary/BrishGarden][BrishGarden]] on top of Brish to provide an HTTP REST API for executing zsh code (if wanted, in sessions). Using BrishGarden, you can embed zsh in pretty much any programming language, and pay no cost whatsoever for its startup. It can also function as a remote code executor.

**** Parallel Execution Using =server_count= =server_count= allows the underlying =zsh= instance of a =Brish= object to fork that many times, and so serve that many clients in parallel. This will not increase the startup time, as the forking happens after loading the =zsh= interpreter completely.

I have combined this with =GNU parallel= to easily parallelize my =zsh= functions.

#+begin_src python :session p1 :results output :exports both n = 32 my_parallel_brish = Brish(server_count=n)

import logging import threading import time

def thread_function(name): logging.info("Thread %s: starting", name) my_parallel_brish.zp("echo Started {name} at $EPOCHREALTIME ; sleep 10 ; echo Finished {name} at $EPOCHREALTIME") logging.info("Thread %s: finishing", name)

if name == "main": format = "%(asctime)s: %(message)s" logging.basicConfig(format=format, level=logging.INFO, datefmt="%H:%M:%S")

threads = list()
now = float(z("echo $EPOCHREALTIME").outrs)
for index in range(32):
    logging.info("Main    : create and start thread %d.", index)
    x = threading.Thread(target=thread_function, args=(index,))
    threads.append(x)
    x.start()

for index, thread in enumerate(threads):
    logging.info("Main    : before joining thread %d.", index)
    thread.join()
    logging.info("Main    : thread %d done", index)

end = float(z("echo $EPOCHREALTIME").outrs)
print(f"Took {(end - now)}")

#+end_src

#+RESULTS: #+begin_example 17:25:26: Main : create and start thread 0. 17:25:26: Thread 0: starting 17:25:26: Main : create and start thread 1. 17:25:26: Thread 1: starting 17:25:26: Main : create and start thread 2. 17:25:26: Thread 2: starting 17:25:26: Main : create and start thread 3. 17:25:26: Thread 3: starting 17:25:26: Main : create and start thread 4. 17:25:26: Thread 4: starting 17:25:26: Main : create and start thread 5. 17:25:26: Thread 5: starting 17:25:26: Main : create and start thread 6. 17:25:26: Thread 6: starting 17:25:26: Main : create and start thread 7. 17:25:26: Thread 7: starting 17:25:26: Main : create and start thread 8. 17:25:26: Thread 8: starting 17:25:26: Main : create and start thread 9. 17:25:26: Thread 9: starting 17:25:26: Main : create and start thread 10. 17:25:26: Thread 10: starting 17:25:26: Main : create and start thread 11. 17:25:26: Thread 11: starting 17:25:26: Main : create and start thread 12. 17:25:26: Thread 12: starting 17:25:26: Main : create and start thread 13. 17:25:26: Thread 13: starting 17:25:26: Main : create and start thread 14. 17:25:26: Thread 14: starting 17:25:26: Main : create and start thread 15. 17:25:26: Thread 15: starting 17:25:26: Main : create and start thread 16. 17:25:26: Thread 16: starting 17:25:26: Main : create and start thread 17. 17:25:26: Thread 17: starting 17:25:26: Main : create and start thread 18. 17:25:26: Thread 18: starting 17:25:26: Main : create and start thread 19. 17:25:26: Thread 19: starting 17:25:26: Main : create and start thread 20. 17:25:26: Thread 20: starting 17:25:26: Main : create and start thread 21. 17:25:26: Thread 21: starting 17:25:26: Main : create and start thread 22. 17:25:26: Thread 22: starting 17:25:26: Main : create and start thread 23. 17:25:26: Thread 23: starting 17:25:26: Main : create and start thread 24. 17:25:26: Thread 24: starting 17:25:26: Main : create and start thread 25. 17:25:26: Thread 25: starting 17:25:26: Main : create and start thread 26. 17:25:26: Thread 26: starting 17:25:26: Main : create and start thread 27. 17:25:26: Thread 27: starting 17:25:26: Main : create and start thread 28. 17:25:26: Thread 28: starting 17:25:26: Main : create and start thread 29. 17:25:26: Thread 29: starting 17:25:26: Main : create and start thread 30. 17:25:26: Thread 30: starting 17:25:26: Main : create and start thread 31. 17:25:26: Thread 31: starting 17:25:26: Main : before joining thread 0. Started 0 at 1620651326.2126729488 Finished 0 at 1620651336.2229239941 17:25:36: Thread 0: finishing 17:25:36: Main : thread 0 done 17:25:36: Main : before joining thread 1. Started 1 at 1620651327.2022259235 Finished 1 at 1620651337.2120540142 17:25:37: Thread 1: finishing 17:25:37: Main : thread 1 done 17:25:37: Main : before joining thread 2. Started 30 at 1620651327.2101778984 Finished 30 at 1620651337.2140960693 17:25:37: Thread 30: finishing Started 2 at 1620651328.2068090439 Finished 2 at 1620651338.2182691097 17:25:38: Thread 2: finishing 17:25:38: Main : thread 2 done 17:25:38: Main : before joining thread 3. Started 31 at 1620651328.2222359180 Finished 31 at 1620651338.2338199615 17:25:38: Thread 31: finishing Started 7 at 1620651329.2063989639 Finished 7 at 1620651339.2115590572 17:25:39: Thread 7: finishing Started 15 at 1620651330.2087130547 Finished 15 at 1620651340.2192440033 17:25:40: Thread 15: finishing Started 21 at 1620651331.2160348892 Finished 21 at 1620651341.2246019840 17:25:41: Thread 21: finishing Started 23 at 1620651332.2160398960 Finished 23 at 1620651342.2200219631 17:25:42: Thread 23: finishing Started 9 at 1620651333.2236700058 Finished 9 at 1620651343.2359619141 17:25:43: Thread 9: finishing Started 18 at 1620651334.2257950306 Finished 18 at 1620651344.2365601063 17:25:44: Thread 18: finishing Started 12 at 1620651335.2241439819 Finished 12 at 1620651345.2335329056 17:25:45: Thread 12: finishing Started 20 at 1620651336.2342200279 Finished 20 at 1620651346.2429049015 17:25:46: Thread 20: finishing Started 16 at 1620651337.4859669209 Finished 16 at 1620651347.4899230003 17:25:47: Thread 16: finishing Started 22 at 1620651338.2339038849 Finished 22 at 1620651348.2375440598 17:25:48: Thread 22: finishing Started 19 at 1620651339.2459530830 Finished 19 at 1620651349.2504169941 17:25:49: Thread 19: finishing Started 13 at 1620651340.2416980267 Finished 13 at 1620651350.2485001087 17:25:50: Thread 13: finishing Started 10 at 1620651340.2490129471 Finished 10 at 1620651350.2568130493 17:25:50: Thread 10: finishing Started 29 at 1620651341.2439520359 Finished 29 at 1620651351.2504179478 17:25:51: Thread 29: finishing Started 25 at 1620651342.2465701103 Finished 25 at 1620651352.2498950958 17:25:52: Thread 25: finishing Started 17 at 1620651343.2493131161 Finished 17 at 1620651353.2571830750 17:25:53: Thread 17: finishing Started 28 at 1620651344.2550890446 Finished 28 at 1620651354.2586359978 17:25:54: Thread 28: finishing Started 14 at 1620651345.2569661140 Finished 14 at 1620651355.2659308910 17:25:55: Thread 14: finishing Started 5 at 1620651346.2559928894 Finished 5 at 1620651356.2631940842 17:25:56: Thread 5: finishing Started 4 at 1620651347.2538421154 Finished 4 at 1620651357.2619009018 17:25:57: Thread 4: finishing Started 3 at 1620651347.2638580799 Finished 3 at 1620651357.2686970234 17:25:57: Thread 3: finishing 17:25:57: Main : thread 3 done 17:25:57: Main : before joining thread 4. 17:25:57: Main : thread 4 done 17:25:57: Main : before joining thread 5. 17:25:57: Main : thread 5 done 17:25:57: Main : before joining thread 6. Started 26 at 1620651348.2553079128 Finished 26 at 1620651358.2628009319 17:25:58: Thread 26: finishing Started 27 at 1620651348.2706210613 Finished 27 at 1620651358.2781529427 17:25:58: Thread 27: finishing Started 24 at 1620651349.2586579323 Finished 24 at 1620651359.2646100521 17:25:59: Thread 24: finishing Started 11 at 1620651350.2648739815 Finished 11 at 1620651360.2702779770 17:26:00: Thread 11: finishing Started 8 at 1620651351.2621378899 Finished 8 at 1620651361.2658278942 17:26:01: Thread 8: finishing Started 6 at 1620651352.4786870480 Finished 6 at 1620651362.4896230698 17:26:02: Thread 6: finishing 17:26:02: Main : thread 6 done 17:26:02: Main : before joining thread 7. 17:26:02: Main : thread 7 done 17:26:02: Main : before joining thread 8. 17:26:02: Main : thread 8 done 17:26:02: Main : before joining thread 9. 17:26:02: Main : thread 9 done 17:26:02: Main : before joining thread 10. 17:26:02: Main : thread 10 done 17:26:02: Main : before joining thread 11. 17:26:02: Main : thread 11 done 17:26:02: Main : before joining thread 12. 17:26:02: Main : thread 12 done 17:26:02: Main : before joining thread 13. 17:26:02: Main : thread 13 done 17:26:02: Main : before joining thread 14. 17:26:02: Main : thread 14 done 17:26:02: Main : before joining thread 15. 17:26:02: Main : thread 15 done 17:26:02: Main : before joining thread 16. 17:26:02: Main : thread 16 done 17:26:02: Main : before joining thread 17. 17:26:02: Main : thread 17 done 17:26:02: Main : before joining thread 18. 17:26:02: Main : thread 18 done 17:26:02: Main : before joining thread 19. 17:26:02: Main : thread 19 done 17:26:02: Main : before joining thread 20. 17:26:02: Main : thread 20 done 17:26:02: Main : before joining thread 21. 17:26:02: Main : thread 21 done 17:26:02: Main : before joining thread 22. 17:26:02: Main : thread 22 done 17:26:02: Main : before joining thread 23. 17:26:02: Main : thread 23 done 17:26:02: Main : before joining thread 24. 17:26:02: Main : thread 24 done 17:26:02: Main : before joining thread 25. 17:26:02: Main : thread 25 done 17:26:02: Main : before joining thread 26. 17:26:02: Main : thread 26 done 17:26:02: Main : before joining thread 27. 17:26:02: Main : thread 27 done 17:26:02: Main : before joining thread 28. 17:26:02: Main : thread 28 done 17:26:02: Main : before joining thread 29. 17:26:02: Main : thread 29 done 17:26:02: Main : before joining thread 30. 17:26:02: Main : thread 30 done 17:26:02: Main : before joining thread 31. 17:26:02: Main : thread 31 done Took 36.33210492134094 #+end_example

** Running in the Background The =z_background= function allows you to execute shell commands asynchronously in a new thread. It starts a new Zsh instance and runs the given command without blocking your main Python thread. This is particularly useful when you want to perform non-blocking operations or execute long-running shell commands without interrupting your Python program's flow.

#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both from brish import z_background

msg = "You can’t make an omelet without breaking a few eggs." result_future = z_background( "say {msg}", # Needs the say command, available by default on macOS ) result_future #+end_src

#+RESULTS: : <Future at 0x1435b51e0 state=running>

The =z_background= function returns a =Future= object.

#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both result_future.result() #+end_src

#+RESULTS: : CmdResult(retcode=0, out='', err='', cmd=" say 'You can’t make an omelet without breaking a few eggs.' ", cmd_stdin='')

Here is another example:

#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both from brish import z_background import concurrent.futures

Define multiple commands

commands = [ "sleep 5 && echo 'First command completed.'", "sleep 3 && echo 'Second command completed.'", "sleep 1 && echo 'Third command completed.'", ]

Execute all commands asynchronously

futures = [z_background(cmd) for cmd in commands]

for future in concurrent.futures.as_completed(futures): result = future.result() print(result.out) #+end_src

#+RESULTS: : Third command completed. : : Second command completed. : : First command completed. :

• Security Considerations
  I am not a security expert, and security doesn't come by default in these situations. So be careful if you use untrusted input in the commands fed to zsh. Nevertheless, I can't imagine any (non-obvious) attack vectors, as the input gets automatically escaped by default. Feedback by security experts will be appreciated.

Note that you can create security holes for yourself, by, e.g., running =eval= on user input:

#+begin_src python :session p1 :results value :exports both untrusted_input = " ; echo do evil | cat" z("eval {untrusted_input}") # unsafe #+end_src

#+RESULTS: #+begin_example do evil #+end_example

#+begin_src python :session p1 :results value :exports both z("echo {untrusted_input}") # safe #+end_src

#+RESULTS: #+begin_example ; echo do evil | cat #+end_example

One thing to keep in mind is that Brish purposely uses the zsh from your PATH. That zsh will load its dotfiles as usual.

• Known Issues

• Piping binary (non-text) output from zsh to Python does not work

• Nonstandard encodings (non UTF-8) are corrupted #+begin_src python :session p1 :results value :exports both z("echo 'sth × another (ver.-)'") #+end_src

  #+RESULTS: #+begin_example sth Ã\xb7 another (ver.-) #+end_example

• There is always sth piped to the standard input (an empty string by default). This can alter the behavior of some commands such as =ripgrep=; Using =</dev/null= or =<&-= can be a suitable workaround.

• Future Features
  I like to add a mode where the zsh session inherits the stderr from the parent Python process. This allows usage of interactive programs like fzf.

If you have any good design ideas, create an issue!

• Related Projects

• [[https://github.com/sharkdp/pysh][pysh]] uses comments in bash scripts to switch the interpreter to Python, allowing variable reuse between the two.
• [[https://github.com/tomerfiliba/plumbum][plumbum]] is a small yet feature-rich library for shell script-like programs in Python. It attempts to mimic the shell syntax ("shell combinators") where it makes sense, while keeping it all Pythonic and cross-platform. I personally like this one a lot. A robust option that is also easy-to-use.
• [[https://github.com/timofurrer/shellfuncs][shellfuncs]]: Python API to execute shell functions as they would be Python functions. (Last commit is in 2017.)
• [[https://github.com/xonsh/xonsh][xonsh]] is a superset of Python 3.5+ with additional shell primitives.
• [[https://github.com/terrycojones/daudin][daudin]] [[https://github.com/terrycojones/daudin#how-commands-are-interpreted][tries]] to eval your code as Python, falling back to the shell if that fails. It does not currently reuse a shell session, thus incurring large overhead. I [[https://github.com/terrycojones/daudin/issues/11][think]] it can use Brish to solve this, but someone needs to contribute the support.
• [[https://github.com/oconnor663/duct.py][duct.py]] is a library for running child processes. It's quite low-level compared to the other projects in this list.
• python -c can also be powerful, especially if you write yourself a helper library in Python and some wrappers in your shell dotfiles. An example:
  #+BEGIN_EXAMPLE
  alias x='noglob calc-raw'
  calc-raw () {
  python3 -c "from math import ; print($)"
  }
  #+END_EXAMPLE
• [[https://github.com/danylo-dubinin/zsh-jupyter-kernel][Z shell kernel for Jupyter Notebook]] allows you to do all sorts of stuff if you spend the time implementing your usecase; See [[https://github.com/nnicandro/emacs-jupyter#org-mode-source-blocks][emacs-jupyter]] to get a taste of what's possible. [[https://github.com/jupyter/kernel_gateway][Jupyter Kernel Gateway]] also sounds promising, but I haven't tried it out yet. Beware the completion support in this kernel though. It uses a pre-alpha proof of concept [[https://github.com/Valodim/zsh-capture-completion][thingy]] that was very buggy when I tried it.
• Finally, if you're feeling adventurous, try Rust's [[https://github.com/rust-shell-script/rust_cmd_lib][rust_cmd_lib]]. It's quite beautiful.

• Licenses
  Dual-licensed under MIT and GPL v3 or later.