Safety features

Several features are included in Subprocess to improve the safety and correctness of programs which launch other programs. It is not perfect in this regard, but it does try to alleviate many of the common pitfalls when working with processes.

No Shell

Commands in Subprocess never receive a shell. We've had enough injection CEVs in the past several decades that it's time to stop giving our commands a shell. I realize redirection tends to be a bit easier to be a bit easier with a shell, but Subprocess provides combinators to make it as easy as possible (but no easier!) to do IO redirection--and indeed, this is the main point of the library.

The user is, of course free to do the equivalent of /bin/sh -c "echo foo", i.e. Subprocess.cmd ["/bin/sh"; "-c"; "echo foo"] but it's your own affair at that point--and you're probably better off using functions provided by the standard library or the Unix library--but I believe you will find that Subprocess provides a better way to do most things.

Non-zero exit status is (normally) an error

Another "feature not a bug" of Subprocess is that, by default, a non-zero exit status is considered a failure and will produce an error in OCaml. In the top-level functions of the Subprocess module, this means raising an exception. If you, like me, prefer to handle errors with result types, the Subprocess.Results module is provided to represent non-zero exit status as an Error of Subprocess.Exit.t. A combinator is provided convert this to Error of string for better composition with other result types in the context of monadic binding. If you don't need the exit status at all, there is also a Subprocess.StringResults module which only deals in Error of string

I consider treating non-zero exit status as an error to be a "sane default" like using set -eo pipefail in a bash script is a sane default. However, I realize that a non-zero exit status is not always an error, and for this, the Subprocess.Unchecked module is provided. This module is also useful if you are interested in the output of a process regardless of the exit status.

Cleanup is abstracted

Subprocess attempts to avoid making the user handle closing processes and pipes explicitly by providing several functions which simply read process output, close all pipes and wait for the process to exit before returning the output (or error). There is also family of fold functions which fold over line output from a running process, but clean everything up before returning.

For running process interaction, the running process is passed to a user-provided function and everything is properly closed when the function exits--either by being fully evaluated or by raising an exception.

If for some reason you need to pass around a live process in a wider context, Subprocess.Exec.exec makes this possible, but you have to clean up your own mess at that point.

Extras

Subprocess generally does not use many any opaque types and practically everything has a pretty printer for easier debugging. The lack of opaque types is not an invitation to go digging into the implementation details which are subject to change, but rather an acknowledgment that no abstraction is perfect and there may at times valid reasons to dig into them.

Opt-in non-blocking IO

Subprocess supports non-blocking IO, but it does not provide direct compatibility layers with popular OCaml asynchronous IO libraries like lwt and eio, which typically provide their own process abstractions. Neither does Subprocess provide its own event loop. The user must handle non-blocking IO operations and process polling manually where non-blocking behavior is desired. It is not different from handling non-blocking IO in the OCaml standard library and examples will be provided here where relevant.

Perhaps compatibility with lwt and / or eio will be a goal for a future version of Subprocess.

Introductory examples

Commands

The most fundamental abstraction in Subprocess is Subprocess.Cmd.t It is simply a data representation of a yet-to-be-executed command, and contains information about arguments, I/O redirection, environment variables and blocking.

  # open Subprocess;;
  # let my_cmd = cmd ["echo"; "foo"];;
  val my_cmd : (stdin, stdout, stderr) Cmd.t = cmd(`echo foo`)
  # Format.printf "%a\n" Cmd.pp my_cmd;;
  cmd(`echo foo`)

As you can see, the command also has type parameters related to each of the standard streams, which both gives us more information as programmers, but also turns certain failure cases for I/O into type errors.

A number of combinators are defined after Subprocess.Core.cmd which may be used to redirect the standard streams, as well as set environment variables and use non-blocking I/O for pipes. Subprocess will either unblock all pipes or none. More on that later. I typically use the pipeline operator with this combinators because it "feels right", but obviously you can use regular function application or the @@ application operator.

For example:

(* similar to `my_cmd 2> /dev/null` *)
# my_cmd |> devnull_err;;
- : (stdin, stdout, devnull) Cmd.t = cmd(`echo foo`, stderr: /dev/null)
(* similar to `my_cmd < input.txt *)
# my_cmd |> file_in "input.txt";;
- : (file, stdout, stderr) Cmd.t = cmd(`echo foo`, stdin: file "input.txt")
(* similar to `USER=app my_cmd` *)
# my_cmd |> env ["USER=app"];;
- : (stdin, stdout, stderr) Cmd.t = cmd(`echo foo`, env:["USER=app"])

One thing that cannot be expressed very well with these combinators is the shell idiom 2>&1, which combines stdout and stderr into a single stream.

# my_cmd |> channel_err Out_channel.stdout;;
- : (stdin, stdout, channel) Cmd.t = cmd(`echo foo`, stderr: channel)

This works well enough if your desire is that the output simply be printed, but we run into problems in this case:

# my_cmd |> channel_err Out_channel.stdout |> pipe_out;;
- : (stdin, pipe, channel) Cmd.t =
cmd(`echo foo`, stdout: pipe, stderr: channel)`

In this case, the child's stdout is redirected to a pipe, but stderr is still printed to the parent process's stdout.

What we have for this case is a family of execution helpers with _joined in the name such as Subprocess.read_joined.

Execution Helpers

Note that we have not yet executed any process. These combinators simply produce a new instance of Subprocess.Cmd.t. How do we execute commands? There are a variety of helper functions for this.

# run my_cmd;;
foo
- : unit = ()
# read my_cmd;;
- : string = "foo\n" 
# lines my_cmd;;
- : string list = ["foo"]
# fold my_cmd ~init:0 ~f:(fun acc line -> acc + String.length line);;
- : int = 3

There are many more such functions. See Subprocess.run and following for more. Remember that all the functions demonstrated here will throw an exception on non-zero exit status. However, they all have analogous versions in Subprocess.Results and Subprocess.Unchecked which handle this case in different ways.

# Results.read my_cmd;;
- : (string, Exit.t) result = Ok "foo\n"
# Unchecked.read my_cmd;;
- : Exit.t * string =
((exited: 0, pid: 13762, cmd(`echo foo`, stdout: pipe)), "foo\n")

If you look closely at the exit information in the previous example, you will also get a glimpse of how the sausage is made. my_cmd has its stdout set to the default (inheriting from the spawning process) but the read function sets it to pipe internally which is necessary to read the output into an OCaml string.

on the benefits of expressive types

As already seen, commands carry type-level information about their I/O streams. Whether you regard this as expressive or excessive may be a matter of taste, but a consequence of this is that not all commands instances are valid with all of these process executor functions.

# read (my_cmd |> file_out "out.txt");;
Error: This expression has type (stdin, file, stderr) Cmd.t
       but an expression was expected of type (stdin, stdout, 'a) Cmd.t
       Type file is not compatible with type stdout

Here, stdout would be redirected twice, and from the perspective of someone reading the code, this is ambiguous in its meaning. Rather than let inscrutable bugs creep into our code, we simply make such ambiguities unrepresentable.

Interacting with Running Processes (or exec and let&)

At times you may wish to interact with a running process, reading from pipes iteratively or polling the process as you go. For iterating over output, you may use Subprocess.fold and related functions, as we have already seen. However as an illustrative example lets make the world's most inefficient capitalize function.

# 
let capitalize str = 
  exec (cmd ["tr"; "a-z"; "A-Z"] |> pipe_in |> pipe_out)
    ~f:(fun proc ->
      Out_channel.output_string (stdin proc) str;
      Out_channel.close (stdin proc);
      In_channel.input_all (stdout proc)
    );;
val capitalize : string -> string = <fun>
# capitalize "foo";;
- : string = "FOO"`

Several important things here.

• Subprocess.exec takes a command as input in addition to a function. The function will take a handle to the process as input and will automatically close everything when the function exits.
• Subprocess.stdin and Subprocess.stdout are functions which will return Stdlib.out_channel and Stdlib.in_channel respectively if these streams have been set to pipe. As you might expect, there is also a Subprocess.stderr function for accessing the process's stderr. Trying to access a stream that is not a pipe with these functions is simply a type error.
• I/O with pipes is tricky. Though exec will automatically close any open pipes, it this we still have to explicitly close or flush stdin in this this case or reading from stdout will block because the string will just be sitting in a buffer otherwise.

Also note that for longer inputs and outputs, we would want to use non-blocking I/O to avoid these kinds of problems--a good case for using Subprocess.fold_with, which handles reads and writes asynchronously.

# 
let capitalize str =
  fold_with (cmd ["tr"; "a-z"; "A-Z"]) ~lines:(Seq.return str) ~init:""
    ~f:( ^ );;
val capitalize : string -> string = <fun>
# capitalize "foo";;
- : string = "FOO"

creating process pipelines

To simplify creating pipelines, let& another way to execute commands with similar semantics to exec. & is a mnemonic for backgrounding process in the shell, which is sort of similar to what is going on here in the sense that the process is launched in the background as your OCaml code continues to execute, but different in the sense that you have a handle to the process and any open pipes.

# let& p1 = cmd ["echo"; "foo"] |> pipe_out in
  let& p2 = cmd ["tr"; "a-z"; "A-Z"] |> channel_in (stdout p1) in
  ();;
FOO
- : unit = ()

Note that this example actually crashes utop for reasons I don't fully understand, but it works correctly in programs.

Also be aware that Subprocess.Results.(let&) composes in a way similar to monadic binding, so a result instance must be returned. Subprocess.Results.exec does not behave in the same way.

# Results.(
let& p1 = cmd ["echo"; "foo"] |> pipe_out in
let& p2 = cmd ["tr"; "a-z"; "A-Z"] |> channel_in (stdout p1) in
Ok ());;
FOO
- : (unit, Exit.t) result = Ok ()
(* yes, this one also crashes utop. *)

Passing around processes

If working processes inside of callback functions cramps your style for some reason, you can also use Subprocess.Exec.exec to pass them around. It is simply your responsibility to ensure they are cleaned up eventually.

# let proc = Exec.exec (cmd ["echo"; "foo"]);;
foo
val proc : (stdin, stdout, stderr) t = process(pid: 17508, cmd(`echo foo`))
# proc.close ();;
- : Exit.t = (exited: 0, pid: 17508, cmd(`echo foo`))

Each instance of Subprocess.t has a close property, which is a closure that closes any file descriptors which need to be closed and waits for the process to exit, returning an instance of Subprocess.Exit.t.

Note that this includes pipes and any redirects of the type file. However, channels bound to a command are not automatically closed, since Subprocess didn't open them.

Non-blocking I/O or "working with multiple pipes for one process"

Subprocess is not fundamentally about efficiently multiplexing I/O operations. Nonetheless working with a process with multiple pipes open Is a problem which is best solved with non-blocking I/O. Using blocking I/O in such a case can lead to deadlocks if you are waiting for I/O on one pipe while the process you've created is waiting for I/O on another pipe. Unless you've written the program yourself, you really have no control over how other processes do I/O or when they flush their buffers.

First, if you are working with multiple pipes for one process, use Subprocess.no_block to set the non-blocking flag.

# let tr_upcase =
    cmd ["tr"; "a-z"; "A-Z"] |> pipe_in |> pipe_out |> no_block;;
val tr_upcase : (pipe, pipe, stderr) Cmd.t =
  cmd(`tr a-z A-Z`, stdin: pipe, stdout: pipe, non-blocking)

Note that there is only one flag to set, and it is applied to all pipes. In any scenario where you want to do non-blocking operations on one pipe, you will want non-blocking on all of them.

With OCaml channels, when they are set to non-blocking and you try to I/O when it's not ready, it raises Sys_blocked_io, so you have to catch that for any I/O operation.

# let proc = Exec.exec tr_upcase;;
val proc : (pipe, pipe, stderr) t =
  process(pid: 18679,
          cmd(`tr a-z A-Z`, stdin: pipe, stdout: pipe, non-blocking))
# In_channel.input_line (stdout proc);;
Exception: Sys_blocked_io.
[stack trace omitted]

As an example of how programming with non-blocking pipes looks, here follows the implementation of Subprocess.fold_with:

let fold_with ?(sep="\n") cmd ~lines ~f ~init =
  let f' t =
    let write_line line lines =
      match Out_channel.output_string (stdin t) line with
      | exception Sys_blocked_io -> Seq.cons line lines
      | () ->
        match Out_channel.output_string (stdin t) sep with
        | exception Sys_blocked_io -> Seq.cons "" lines
        | () -> lines in
    let rec go lines_opt acc =
      let lines_opt' = Option.bind lines_opt @@ fun lines ->
        match lines () with
        | Seq.Nil -> Out_channel.close (stdin t); None
        | Seq.Cons (line, tl) -> Some (write_line line tl) in
      match In_channel.input_line (stdout t) with
      | exception Sys_blocked_io -> go lines_opt' acc
      | None -> acc
      | Some line ->
        go lines_opt' (f acc line) in
    go (Some lines) init in
  exec (cmd |> pipe_in |> pipe_out |> no_block) ~f:f'

To summarize, you have to check for Sys_blocked_io after every I/O operation and decide what action to take if I/O was not available on the channel you tried to communicate with.

Another approach which is not technically non-blocking, but achieves the same thing is using Unix.select. This would require using the Unix module to retrieve the file descriptors from the channels.

Any helper function that ends with _both uses non-blocking I/O internally, though the helper function itself will block until all I/O is finished.

One could conceivably execute such functions in worker threads, of course, so as not to block the entire program.