Everything on this site is produced by two scheduled jobs on a Mac Mini in Jakarta. One runs at 09:02 WIB and builds the news brief. One runs at 19:00 WIB, after the Indonesia Stock Exchange has closed and settled, and builds the liquidity screener. Neither has a server, a queue, or a container. Both end by pushing a folder of static files to a git remote.
That architecture is deliberately unfashionable, and it has survived every failure mode thrown at it so far. Here is what actually matters in it.
Use launchd, not cron
macOS still runs cron, but it is a compatibility shim. If the Mac is asleep at the scheduled minute, a cron job silently does not happen and you find out days later. launchd is the supported scheduler, and its StartCalendarInterval has the behaviour you want: if the machine was asleep at the trigger time, the job fires when it wakes.
A minimal agent lives in ~/Library/LaunchAgents/ and looks like this:
<key>StartCalendarInterval</key>
<dict>
<key>Hour</key><integer>9</integer>
<key>Minute</key><integer>2</integer>
</dict>
<key>WorkingDirectory</key>
<string>/path/to/project</string>
<key>EnvironmentVariables</key>
<dict>
<key>PATH</key><string>/opt/miniconda3/bin:/usr/local/bin:/usr/bin:/bin</string>
<key>HOME</key><string>/Users/you</string>
</dict>
Three of those lines are load-bearing, and every one of them cost a broken run to learn:
PATH — launchd does not source your shell profile. Your job gets a minimal environment. If any step shells out to git, ffmpeg, or a conda Python, it will not be found. Set PATH explicitly and use absolute paths for the interpreter.
HOME — if your git remote is an SSH host alias defined in ~/.ssh/config, ssh needs HOME to find that file. Without it the push fails with a confusing "could not resolve hostname" for a host that resolves fine in your terminal.
WorkingDirectory — this one is the sneakiest. python-dotenv's load_dotenv() with no argument searches upward from the current working directory. Under launchd, the working directory is / unless you set it, so .env is never found, and the API key is None. The code does not crash at import — it crashes later, inside the API call, with an authentication error that sends you looking at the wrong thing entirely.
The robust fix is to not depend on the working directory at all:
from pathlib import Path
from dotenv import load_dotenv
load_dotenv(Path(__file__).parent.parent / ".env")
Set WorkingDirectory anyway. Defence in depth is cheap here.
Never claim a job is scheduled without checking
This is the most expensive lesson here, and it was not a technical failure.
The project's own documentation stated that a particular collection step ran automatically every night at 22:00. It said so in three places. For two days the data behind a dashboard was stale and nothing alerted, because there was no job. No crontab entry, no LaunchAgent, nothing had ever been installed. The claim had been written down once as an intention and then read back forever as a fact.
The check takes one second:
launchctl list | grep yourjob
crontab -l
If your notes say a thing is scheduled, that command is the only evidence that counts. Treat every "this runs automatically" sentence in your own documentation as unverified until those two lines have been re-run. A scheduled job that does not exist is worse than a manual step, because a manual step is at least visibly manual.
One lock, shared by both jobs
Two jobs, one git repository, one output directory. If they ever overlap — a late morning run colliding with a re-run of the evening job — they will interleave commits and stage each other's half-written files.
The fix is one exclusive flock on a lock file, taken by both jobs under the same name:
import fcntl
from contextlib import contextmanager
@contextmanager
def run_lock(name):
path = PROJECT_DIR / "storage" / f".{name}.lock"
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w") as fh:
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
yield
The interesting part is what happens on contention, and it depends on who is running.
Run by hand, the right behaviour is to ask: wait, proceed anyway, or abort. Run by launchd, there is nobody to ask — and a scheduled job that blocks on a prompt does not fail, it hangs, holding the lock and blocking every subsequent run until someone notices. So the contention path branches on whether there is a terminal attached:
if sys.stdin.isatty():
choice = input("[w]ait / [p]roceed / [a]bort? ")
else:
print("[LOCK] another run holds the lock — exiting")
sys.exit(0)
The same rule applies to anything that can prompt. A library that asks for a phone code, a CLI that asks to confirm — under a scheduler, all of them are hangs waiting to happen. Guard every one of them with isatty().
Exiting 0 rather than 1 on contention is intentional: another process holding the lock is a normal outcome, not a failure worth alerting on.
Make the last step a git push
The publishing step is git add, git commit, git push. Cloudflare Pages watches the repository and deploys. There is no build command and no framework — the output directory contains exactly what the browser receives.
This has a few properties that are easy to underrate until you have them.
Deploys are atomic and free. A push either lands or it does not. There is no half-deployed state.
History is the backup. Every day's published output is a commit. Recovering yesterday's brief is git show, not a database restore.
A no-op day costs nothing. The commit step checks git status --porcelain first and returns early when the output is byte-identical. That means serialisation has to be deterministic — same input, same bytes. json.dump(..., indent=2, ensure_ascii=False) with insertion-ordered keys gives you that for free, and it is the difference between a quiet day and a pointless deploy.
Scope the git add. Use git add -- output_dir/ rather than git add -A. The narrow path means a stray file at the repository root — a .env, a session file, a downloaded CSV — is not merely gitignored but unstageable by the automation. Gitignore is a rule you can forget to write. A scoped add is a rule you cannot forget to apply.
Let steps fail without failing the run
Not every step deserves to stop the pipeline. Losing one optional data source should degrade the output, not delete it. Every step goes through one wrapper that decides:
def run_step(name, func, date_str=None, fatal=True):
try:
func()
return True
except Exception as e:
print(f"{name} error: {e}", file=sys.stderr)
if fatal:
write_run_log(date_str, "failed", f"{name}: {e}")
sys.exit(1)
return False
Fetching the feeds is fatal; there is nothing to summarise without them. Publishing is fatal. An optional enrichment source is not. The important half is write_run_log — under a scheduler, an unattended traceback goes to a log file nobody opens. A structured run log with a status and a note is the thing you can actually grep three weeks later when you ask "when did this last work?"
What stays
Rebuilt tomorrow, these parts would stay: static output, git as the deploy mechanism, one lock shared by every job that touches the repository, isatty() guards on anything that can prompt, and a run log with one line per run.
The part that would change: the launchctl list check belongs in the documentation itself, as a command to run rather than a sentence to believe.