I Added pipefail and a Passing Check Turned Red: The False Failure That Starts Once Output Passes 8 KB
Adding pipefail can turn a passing check red. I reproduced the false failure that appears once output passes 8 KB on 1,122 files, then built a file-based runner and a canary.
In an earlier post I wrote about how a pipe can erase a failure. The remedy I gave was set -o pipefail.
This time I applied that remedy to the front-matter check for my own site and tested it again. The medicine, it turned out, has a side effect.
On a repository with no defects at all, the check went red. Nothing in the check had changed. Only the amount of output had.
A pipe can get an exit code wrong in two directions: it can hide a failure, or it can turn a success into one. The first is well known. The second shows up only on the day your inputs grow, which is why I found it easy to miss. Below are the measurements and the full text of the runner I ended up with.
The check, and the environment I used
The target is the MDX of the 1,122 Japanese articles on this site, which I run as an indie developer. The check looks at three things in the front matter: a misspelled level, a period in slug, and an unquoted colon in highlights. Each one causes a real 404 or 500 in production.
Here is the whole script.
#!/usr/bin/env python3"""MDX front-matter check. Prints each violation and exits 1 if there is any.If there are zero target files, exit 2 so that a green that looked at nothing cannot happen."""import reimport sysimport pathlibLEVELS = {"beginner", "intermediate", "advanced"}args = [a for a in sys.argv[1:] if not a.startswith("-")]verbose = "-v" in sys.argvroot = pathlib.Path(args[0])files = sorted(root.rglob("*.mdx"))if not files: print(f"NOFILES {root}") sys.exit(2)bad = 0for p in files: text = p.read_text(encoding="utf-8") m = re.match(r"^---\n(.*?)\n---\n", text, re.S) if not m: print(f"NOFM {p}") bad += 1 continue fm = m.group(1) if verbose: print(f"OK {p}") lv = re.search(r'^level:\s*"?([a-z]+)"?', fm, re.M) if not lv or lv.group(1) not in LEVELS: print(f"LEVEL {p}") bad += 1 sl = re.search(r'^slug:\s*"([^"]+)"', fm, re.M) if sl and "." in sl.group(1): print(f"SLUG {p} ({sl.group(1)})") bad += 1 for h in re.findall(r'^ - ([^"\n][^\n]*:\s[^\n]*)$', fm, re.M): print(f"COLON {p} ({h[:30]})") bad += 1print(f"checked={len(files)} violations={bad}")sys.exit(1 if bad else 0)
It has three exit codes: 1 for any violation, 2 if there is not a single file to look at, and 0 otherwise.
I added the 2 because an empty directory would otherwise report "0 violations" and go green. Run on an empty directory, the script prints NOFILES and stops with 2. A green that looked at nothing is more dangerous than a red.
Item
Value
Shell
GNU bash 5.1.16
Python
3.10.12
Target
1,122 Japanese articles (a copy of the real repository)
Result on the real repository
checked=1122 violations=0 (exit code 0)
Run time
0.13 s (identical across five runs)
Three planted defects, five ways to call the check
The real repository has no defects, so I could not see the check go red. In a copied directory I planted three on purpose: level set to intermediat, .v2 appended to a slug, and 手順: added to the start of a highlights item.
Against that same copy, only the way of calling changed.
How it was called
Exit code
Verdict
Run directly
1
Reported
Trimmed with tail -1
0
Lost
Same, then read PIPESTATUS
1 and 0
Reported if you read it
set -o pipefail plus tail -1
1
Reported
tail -1 || true
0
Lost (on purpose)
So far this matches the earlier post. With three violations present, looking only at the tail gives a green, and pipefail brings the 1 back.
I thought that settled it. Then I tried to carry the same pattern into another call, and tripped.
✦
Thank you for reading this far.
Continue Reading
What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.
WHAT YOU'LL LEARN
✦You can now tell, with a few lines in your own shell, whether a piped check is hiding a failure or turning a success into one
✦You can now prevent the false red that appears the day a check's output grows, using a file-based runner, before you start hunting for a cause
✦You can now confirm in one command that every new check really goes red on a known-bad input, so a green result stays trustworthy
Secure payment via Stripe · Cancel anytime
✦
Unlock This Article
Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.
To see what the check had looked at, I used -v, which prints one OK line per file. I only wanted the first line, so I piped it into head -1. The repository had no defects.
( set -o pipefail; python3 check_frontmatter.py corpus-clean -v | head -1 )echo "rc=$?"
The exit code was 1, and standard error showed BrokenPipeError: [Errno 32] Broken pipe. The check had succeeded, yet the pipeline as a whole was reported as a failure.
The cause is that head -1 exits after reading one line. Python then tries to write to a pipe with no reader and the write fails. pipefail faithfully carries that failure into the result.
How much output does it take? I varied the number of files and counted over 30 runs each, with no defects planted.
#!/usr/bin/env bash# Vary the output size and count how often pipefail + head -1 turns a healthy check red.# Beforehand, put N .mdx files in sweep/xN/ (no planted defects).for n in 100 103 104 106 300 1122; do d="sweep/x$n" bytes=$(python3 check_frontmatter.py "$d" -v | wc -c) red=0 for i in $(seq 30); do ( set -o pipefail; python3 check_frontmatter.py "$d" -v 2>/dev/null | head -1 >/dev/null ) || red=$((red+1)) done echo "files=$n output=${bytes}B red=$red/30"done
Files
Output size
Healthy runs that went red
100
8,042 B
0 / 30
103
8,304 B
1 / 30 (0 / 30 in another round)
104
8,383 B
30 / 30 (29 / 30 in another round)
106
8,540 B
30 / 30
300
24,234 B
30 / 30
1,122
90,950 B
30 / 30
On my machine the switch happened between 103 and 104 files. Python's standard output buffer is 8,192 B. Once the output exceeds it and is written in several chunks, every write after the first can hit a head that has already finished.
Not one character of the check changed, and yet once the inputs grew it went red every time. A false failure that flips on output size does not yield to staring at the check itself.
The threshold also moves with the length of the printed paths. The 103-file row was 0 in one round and 1 in another, so the neighborhood is unstable. I would remember it as "somewhere around 8 KB" rather than trust my exact numbers.
grep -q did the same, since it stops reading at the first match. With 300 files or more, it went red in 40 of 40 runs.
The exit code is not one value
The value on a red run depended on the setup.
Situation
Exit code
Standard error discarded, piped into head -1
1
Standard error also sent into head -1
120
Python restored the default SIGPIPE behavior
141
Whether you get 1, 120, or 141, you cannot read it as "the check failed". All you can read is "something in the pipe returned non-zero". Trying to guess the cause from the value was not worth the effort.
The fix: write to a file first, then display
I did not give up pipefail. I stopped using pipes for the verdict. The runner writes the output to a file, fixes the check's exit code first, and only then displays it.
#!/usr/bin/env bash# Write the check output to a file, fix the exit code first, then display it.# Usage: run_gate.sh <log-name> -- <command...>set -uname="$1"; shift; [ "${1:-}" = "--" ] && shiftlog="${GATE_LOG_DIR:-$HOME/.gate-logs}/${name}.log"mkdir -p "$(dirname "$log")""$@" >"$log" 2>&1rc=$? # <- no pipe in between, so this is the check's own exit codeecho "[gate:$name] rc=$rc lines=$(wc -l <"$log")"tail -n "${GATE_TAIL:-5}" "$log" # the display-only tail does not affect the exit codeexit "$rc"
The mechanism is simple. The check runs behind a file redirect instead of on the left side of a pipe, so rc=$? receives the check's own exit code. The tail used for display runs afterward as a separate command.
Here are the results on the same copy and the same check.
Situation
Runner exit code
Copy with three planted defects
1 (three violations shown)
Real repository, no defects
0
-v printing 90,950 B, ten runs
0 all ten times
At an output size where head -1 went red 30 times out of 30, the runner returned 0 ten times out of ten.
I also print rc= on the first line. If a caller only reads the end of the output, the exit code still sits at the top of what is displayed. The number of lines shown from the tail can be changed with an environment variable.
A canary proves the check can go red
Fixing how the exit code travels does not help if the check itself is broken, because a green then guarantees nothing. When I wrote about hooks that were loaded but never ran, I came away convinced that "it is running" can only be confirmed by evidence that it ran. I apply the same idea to checks.
This script runs the check on a directory holding exactly one known-bad file and confirms that it returns non-zero.
#!/usr/bin/env bash# Plant one known-bad file and confirm the check always goes red. If it does not, the check itself is broken.here="$(cd "$(dirname "$0")" && pwd)"./run_gate.sh canary -- python3 "$here/check_frontmatter.py" "$here/canary" >/dev/nullrc=$?if [ "$rc" -eq 0 ]; then echo "CANARY BROKEN: planted a defect but got rc=0"; exit 2; fiecho "canary ok (rc=$rc)"
canary/bad.mdx holds a single file with an invalid level and a period in its slug. On my machine it printed canary ok (rc=1), and the log recorded two violations.
I add one canary every time I add a check. If I later edit a regular expression and the check quietly stops detecting anything, I find out right after the edit.
Pitfalls
Do not add set -e or pipefail to the runner. The script would end before reaching rc=$?, and the rc= line would never appear. The runner behaved best when it only received the result.
The log is overwritten under the same name. If you run the same check in parallel, add something unique per run to the name. I give parallel runs distinct names so I cannot misread them.
Standard error is mixed into the same file. All diagnostics land in one log, which helps later reading. It is a poor fit if you need to pass standard output alone to a later stage.
Do not try to read meaning into 1, 120, and 141. Moving to a form without pipes cost fewer checks than hunting for meaning in the differences.
The rules I now follow
Three rules came out of this.
I never use a pipe for the verdict. I write to a file and take rc=$?, using tail only for display.
I place one canary next to every check I add, so that "goes red on a known-bad input" lives in the same place as the check.
I do not allow a green over zero targets. If there is nothing to look at, the check stops with exit code 2.
The next time you add a check, try planting one defect that does not exist yet and see whether it goes red. Making the green trustworthy before adding more greens ends up costing less effort.
I hope this saves someone from stumbling in the same place. Thank you for reading to the end.
Share
Thank You for Reading
Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.