Shell commands¶

In any script, a line that starts with $ is run as an external command. cliffy uses pybash to translate these lines into subprocess calls, so you can move between Python and shell freely.

The basics¶

  hello: $ echo "hello from the shell"

  mixed:
    help: Mix Python and shell lines
    params:
      - name: str!
    run: |
      print("python first")
      $ echo "then shell"
      if name == "ada":
          $ echo "shell lines can be indented"

$ echo "then shell" becomes subprocess.run(["echo", "then shell"]). The command is split into arguments the way a shell would split it, and is then started directly, without a shell in between. Shell lines follow Python indentation, so they can sit inside if blocks and loops.

$ lines work in run, pre_run, post_run, in callbacks, in functions written as a list, and as steps in tests. Run cli render to see the translation.

Using Python values¶

f{expression} inserts the value of a Python expression into the command. The value is passed as a single argument, whatever it contains: spaces, quotes and ; are just text.

  say:
    help: Pass a Python value to a shell command
    params:
      - message: str!
    run: $ echo f{message} f{len(message)}

$ cli run shell.yaml say "two words"
two words 9

Everything an f-string accepts after the expression works too, and f{ } can be used inside quotes:

  price:
    help: f{ } takes f-string format specs, also inside quotes
    params:
      - amount: float!
    run: |
      out = $ echo "total: f{amount:,.2f}"
      print(out.decode().strip())

Double braces are Jinja's¶

pybash also has a {{ expression }} form, which inserts str(expression). In a manifest, cliffy renders Jinja first, so {{ name }} is replaced with a var when the manifest is loaded and never reaches pybash. That is what you want for vars:

vars:
  data_file: tasks.json
commands:
  backup: $ cp {{ data_file }} {{ data_file }}.backup

For a value that only exists when the command runs, such as a param, use f{ }. If you prefer the double-brace form, put the lines in a Jinja raw block so they are left alone:

  whoami:
    help: Double braces belong to Jinja unless they are inside a raw block
    params:
      - user: str!
    run: |
      {% raw %}
      out = $ echo {{ user.title() }}
      {% endraw %}
      print(out.decode().strip())

To write the markers themselves as text, escape them: \f{ and, inside a raw block, \{{.

Capturing output¶

Assign a shell line to a variable, or wrap it in parentheses, to get its output as bytes. $(command) does the same in the middle of an expression:

  upper:
    help: Capture shell output in Python
    run: |
      out = $ echo captured
      print(out.decode().strip().upper())

  inline:
    help: Capture inline with parentheses
    run: print(($ echo inline).decode().strip())

  words:
    help: Capture in the middle of an expression with $( )
    run: print(len($(echo one two three).split()), "words")

$ cli run shell.yaml upper
CAPTURED
$ cli run shell.yaml words
3 words

A capture that is not wrapped in brackets runs to the end of the line.

Pipes and redirection¶

Pipes and redirects are carried out by the generated CLI itself, without a shell:

  sorted:
    help: Pipes work without a shell
    run: $ printf 'b\na\n' | sort

  save:
    help: So does redirection
    run: |
      $ echo "saved" > shell-demo.txt
      $ cat shell-demo.txt

  errors-to-file:
    help: Redirect stderr on its own, together with stdout, or into a pipe
    run: |
      $ ls no-such-file 2> shell-demo.txt
      $ ls . no-such-file &> shell-demo.txt
      $ ls . no-such-file > shell-demo.txt 2>&1
      lines = $ ls no-such-file |& wc -l
      print(int(lines), "line of errors")

Operator	Meaning
`a \| b`	Pipe the output of `a` into `b`
`a \|& b`	Pipe output and errors
`< file`	Read input from a file
`> file`, `>> file`	Write or append output
`2> file`, `2>> file`	Write or append errors
`&> file`, `&>> file`	Write or append both
`2>&1`, `>&2`	Send errors to the output, or the output to the errors

Chaining commands¶

&& runs the next command when the previous one succeeded, || when it failed, and ; in either case:

  chain:
    help: "&&, || and ; decide what runs next, like in a shell"
    run: |
      out = $ true && echo "first worked" || echo "first failed"
      print(out.decode().strip())
      out = $ false || echo "fallback" ; echo "always"
      print(out.decode().split())

Capturing a chain captures the output of every command that ran.

Long commands and comments¶

A $ or > line that ends with a backslash continues on the next line. An unquoted # starts a comment that runs to the end of the line:

  longest:
    help: "A trailing backslash continues a command, a # starts a comment"
    run: |
      $ printf '%s\n' pear fig banana \
          | sort \
          | tail -n 1 > shell-demo.txt  # the last one in alphabetical order
      $ echo "a # inside quotes is text" >> shell-demo.txt
      print(open("shell-demo.txt").read())

A captured command (out = $ ...) has to fit on one line.

When you need a real shell¶

$ lines never expand glob patterns (*.txt), ~ or environment variables ($HOME), and they do not support background jobs (&) or here-documents (<<). For those, start the line with > instead of $. The line is handed to the shell as written:

  texts:
    help: Glob patterns need a real shell
    run: "> ls *.txt"

This becomes subprocess.run("ls *.txt", shell=True). A one-line command that starts with > must be quoted, because > also has a meaning in YAML.

Values inserted with f{ } are quoted for the place they appear in, so a value is always one word of data and never shell syntax. Here name can be ; rm -rf ~ and nothing is removed:

  find:
    help: A value in a > line is quoted for the shell
    params:
      - name: str!
    run: |
      > ls | grep -c f{name} || echo "nothing is called" f{name}

  glob:
    help: Splice a value in as shell syntax, here a glob pattern
    params:
      - pattern: str!
    run: |
      {% raw %}
      > ls {{! pattern }}
      {% endraw %}

{{! expression }} is the exception: it splices the value in unquoted, so that it can be a glob pattern or several arguments. Like every double-brace form it has to sit in a raw block.

Never pass user input to {{! }}

A spliced value is interpreted by the shell. A parameter value such as ; rm -rf ~ would be executed. Use f{ } for anything that comes from a user, and prefer $ lines, which have no shell to inject into. See the Python documentation on subprocess security.

Errors¶

A shell line that fails does not stop the script. Check the result yourself when it matters:

  careless:
    help: A failing command does not stop the script
    run: |
      $ false
      print("still running")

  careful:
    help: Check the exit status yourself
    run: |
      result = subprocess.run(["false"])
      if result.returncode != 0:
          print("command failed")
          raise typer.Exit(code=result.returncode)

Two exceptions:

A command that does not exist raises FileNotFoundError.
A captured command (out = $ ...) raises subprocess.CalledProcessError when it exits with a non-zero status.

A shell line that cannot be translated is reported when the manifest is loaded, with the command it is in, the line and the column:

$ cli load deploy.yaml
~ error in deploy.yaml: command `ship`, run, line 2, column 11: running in the background with & is not supported in a $ command; use a > shell command
    $ sleep 5 &
              ^ 💔

Things to know¶

Output of $ lines goes straight to the terminal. It is not part of result.output in manifest tests; capture it and print it if you want to assert on it.
A CLI that only uses plain commands imports nothing but subprocess. Pipes, redirects, chains and values in > lines go through a small helper, pybash.runtime, which the generated code imports as _pybash. cli build copies that helper into the file it builds, so a built CLI does not need pybash.
Quoting follows POSIX shell rules: single quotes are literal, double quotes allow \" and \\, and a backslash escapes the next character. Quote or escape an operator to pass it as text: $ echo 'a && b' \| c.

The complete example¶

shell.yaml

name: sh
version: 0.1.0
help: Shell commands in run scripts

functions:
  - |
    def current_branch() -> str:
        out = $ git rev-parse --abbrev-ref HEAD
        return out.decode().strip()

commands:
  hello: $ echo "hello from the shell"

  mixed:
    help: Mix Python and shell lines
    params:
      - name: str!
    run: |
      print("python first")
      $ echo "then shell"
      if name == "ada":
          $ echo "shell lines can be indented"

  say:
    help: Pass a Python value to a shell command
    params:
      - message: str!
    run: $ echo f{message} f{len(message)}

  price:
    help: f{ } takes f-string format specs, also inside quotes
    params:
      - amount: float!
    run: |
      out = $ echo "total: f{amount:,.2f}"
      print(out.decode().strip())

  whoami:
    help: Double braces belong to Jinja unless they are inside a raw block
    params:
      - user: str!
    run: |
      {% raw %}
      out = $ echo {{ user.title() }}
      {% endraw %}
      print(out.decode().strip())

  upper:
    help: Capture shell output in Python
    run: |
      out = $ echo captured
      print(out.decode().strip().upper())

  inline:
    help: Capture inline with parentheses
    run: print(($ echo inline).decode().strip())

  words:
    help: Capture in the middle of an expression with $( )
    run: print(len($(echo one two three).split()), "words")

  sorted:
    help: Pipes work without a shell
    run: $ printf 'b\na\n' | sort

  save:
    help: So does redirection
    run: |
      $ echo "saved" > shell-demo.txt
      $ cat shell-demo.txt

  errors-to-file:
    help: Redirect stderr on its own, together with stdout, or into a pipe
    run: |
      $ ls no-such-file 2> shell-demo.txt
      $ ls . no-such-file &> shell-demo.txt
      $ ls . no-such-file > shell-demo.txt 2>&1
      lines = $ ls no-such-file |& wc -l
      print(int(lines), "line of errors")

  chain:
    help: "&&, || and ; decide what runs next, like in a shell"
    run: |
      out = $ true && echo "first worked" || echo "first failed"
      print(out.decode().strip())
      out = $ false || echo "fallback" ; echo "always"
      print(out.decode().split())

  longest:
    help: "A trailing backslash continues a command, a # starts a comment"
    run: |
      $ printf '%s\n' pear fig banana \
          | sort \
          | tail -n 1 > shell-demo.txt  # the last one in alphabetical order
      $ echo "a # inside quotes is text" >> shell-demo.txt
      print(open("shell-demo.txt").read())

  texts:
    help: Glob patterns need a real shell
    run: "> ls *.txt"

  find:
    help: A value in a > line is quoted for the shell
    params:
      - name: str!
    run: |
      > ls | grep -c f{name} || echo "nothing is called" f{name}

  glob:
    help: Splice a value in as shell syntax, here a glob pattern
    params:
      - pattern: str!
    run: |
      {% raw %}
      > ls {{! pattern }}
      {% endraw %}

  careless:
    help: A failing command does not stop the script
    run: |
      $ false
      print("still running")

  careful:
    help: Check the exit status yourself
    run: |
      result = subprocess.run(["false"])
      if result.returncode != 0:
          print("command failed")
          raise typer.Exit(code=result.returncode)

tests:
  - hello: assert result.exit_code == 0
  - mixed ada: assert "python first" in result.output
  - say hi: assert result.exit_code == 0
  - upper: assert result.output == "CAPTURED\n"
  - inline: assert result.output == "inline\n"
  - words: assert result.output == "3 words\n"
  - price 1234.5: 'assert result.output == "total: 1,234.50\n"'
  - whoami ada: assert result.output == "Ada\n"
  - sorted: assert result.exit_code == 0
  - save: assert result.exit_code == 0
  - errors-to-file: assert result.output == "1 line of errors\n"
  - chain: assert result.output == "first worked\n['fallback', 'always']\n"
  - longest: assert result.output.splitlines()[0] == "pear" and "inside quotes" in result.output
  - find "; echo injected": assert result.exit_code == 0
  - glob "*.yaml": assert result.exit_code == 0
  - texts: assert result.exit_code == 0
  - careless: assert result.output == "still running\n"
  - careful: assert result.exit_code == 1
  - $ rm -f shell-demo.txt