Release-test a macOS CLI beyond a successful build

Building a command-line program on a cloud Mac is only the first step. Before delivery, check the contract its callers rely on: help text must not leak into data output, failures must not return zero, and the tool must work outside the developer’s current directory. This small Swift tool provides a release test that needs no graphical session and can run in an unattended build job.

Define the invocation contract first

The example tool, taskprobe, accepts --output and a piece of text, then writes the text to the specified file. The file path is required. A help request exits successfully; invalid arguments fail. Define how callers will judge each outcome before implementing the tool.

Standard output is reserved for machine-consumable success results; standard error is reserved for diagnostics. A calling script must check both the exit code and the expected file. A “written” message in a log is not enough to establish success.

An atomic write prevents partial new content from replacing the target file. It does not guarantee that the data will survive a power failure; durability and recovery require a separate design.

Build a minimal testable tool

Save the source and compile it in a separate directory. The program does not hard-code an output location: the caller supplies an absolute path so the tests can deliberately change the working directory. Foundation’s atomic write option avoids exposing partial content at the target path during a normal write.

WORK="$(mktemp -d)"
mkdir -p "$WORK/bin" "$WORK/home"
cat > "$WORK/main.swift" <<'SWIFT'
import Foundation

func fail(_ message: String) -> Never {
    FileHandle.standardError.write(Data((message + "\n").utf8))
    exit(2)
}

let args = Array(CommandLine.arguments.dropFirst())
if args == ["--help"] {
    print("usage: taskprobe --output PATH TEXT")
    exit(0)
}
guard args.count == 3, args[0] == "--output",
      !args[1].isEmpty else {
    fail("expected --output PATH TEXT")
}

let target = URL(fileURLWithPath: args[1])
do {
    try Data(args[2].utf8).write(to: target, options: .atomic)
    print("written")
} catch {
    fail("cannot write output: \(error.localizedDescription)")
}
SWIFT
xcrun swiftc -O "$WORK/main.swift" -o "$WORK/bin/taskprobe"

xcrun uses the currently selected developer toolchain. You can run xcode-select -p before compiling to confirm the selection, but that does not replace a run on the target macOS version. Test the same binary you intend to deliver. Do not rebuild after acceptance and then deliver a different artifact.

Test success and failure in an isolated directory

The script below uses WORK from the preceding shell session. Each check captures the two output streams separately. The test deliberately changes to a temporary working directory and clears most inherited environment variables. If the delivered tool needs a particular variable, make it part of the contract rather than relying on an interactive shell to provide it.

set -eu
BIN="$WORK/bin/taskprobe"
cd "$WORK/home"

env -i PATH=/usr/bin:/bin HOME="$WORK/home" \
  "$BIN" --help >"$WORK/help.out" 2>"$WORK/help.err"
grep -Fx 'usage: taskprobe --output PATH TEXT' "$WORK/help.out"
test ! -s "$WORK/help.err"

if "$BIN" --output >"$WORK/bad.out" 2>"$WORK/bad.err"; then
  echo "missing arguments returned success" >&2
  exit 1
fi
test ! -s "$WORK/bad.out"
grep -F 'expected --output' "$WORK/bad.err"

"$BIN" --output "$WORK/home/result.txt" 'hello 云端 Mac' \
  >"$WORK/good.out" 2>"$WORK/good.err"
grep -Fx 'written' "$WORK/good.out"
test ! -s "$WORK/good.err"
grep -Fx 'hello 云端 Mac' "$WORK/home/result.txt"

if "$BIN" --output "$WORK/home/absent/result.txt" test \
  >"$WORK/fail.out" 2>"$WORK/fail.err"; then
  echo "unwritable path returned success" >&2
  exit 1
fi
test ! -e "$WORK/home/absent/result.txt"
test ! -s "$WORK/fail.out"
test -s "$WORK/fail.err"

The first failure case checks invalid arguments; the second checks a write error. Do not use chmod to manufacture an “unwritable directory”: a test process with special privileges may behave differently. A nonexistent parent directory is more direct, but it still covers only one kind of I/O failure and cannot prove that every disk error will be handled as expected.

Avoid three common false positives

A pipeline hides the exit code

The exit code of taskprobe ... | tee output.log usually comes from the last command in the pipeline, not from the tool. Prefer redirecting both streams directly in acceptance tests, then check the command’s status. If a pipeline is necessary, enable pipefail in a shell that supports it and record which shell you used.

The current directory happens to be right

Relative paths depend on the launch directory. The test above runs the binary from home and supplies an absolute output path to expose hidden working-directory assumptions. If a real tool needs bundled resources, locate them through an explicit resource path; do not assume the task always starts in the source root.

Error text is mistaken for data

Checking only that “the command printed something” proves little. A downstream job might save standard output directly as its result file. Acceptance tests should assert all three failure conditions: empty standard output, nonempty standard error, and a nonzero exit code. On success, assert that standard error is empty and check the generated file’s actual contents. If the tool emits structured data, validate its format rather than comparing file sizes alone.

Check the boundaries before delivery

Keep the acceptance script alongside the build artifact and run it in the same build job. Once it passes, record the binary’s checksum and the selected toolchain information for the handoff. The account running the test should have the same directory access as the account that will run the actual task. Use a disposable directory for file tests so an old artifact cannot make test -e pass unexpectedly.

These tests cannot simulate a power failure, a full disk, or another process modifying the target file concurrently. They also cannot establish that the program runs on an untested system version. If you need those guarantees, add fault injection, concurrency coordination, and acceptance tests on the target system, respectively. Before the rental period ends, move the source, scripts, and acceptance records out, and have the next owner rerun the full suite in the new environment. “It passed before” is not an acceptance result for a new machine.

Frequently asked questions