The Indonesia Stock Exchange publishes a daily trading summary for every listed stock — volume, value, frequency, closing price, split by board. It is public, it is free, and it is the raw material behind the liquidity screener on this site.
Two things stand between that endpoint and a clean panel. One is a networking problem with a well-known fix. The other is a domain problem that produces numbers which look completely reasonable and are wrong by roughly a third.
Your HTTP client is the problem, not your headers
The first attempt is always the same. You call the endpoint with requests, you get a Cloudflare challenge page, and you assume it is your User-Agent. So you copy the full header set out of devtools — UA, Accept, Accept-Language, Referer, sec-ch-ua, all of it. You still get the challenge page.
The headers were never the signal. The signal is the TLS handshake.
Before any HTTP header is sent, your client offers a ClientHello: a specific ordered list of cipher suites, extensions, elliptic curves, and ALPN values. That ordering differs between TLS implementations, and it is stable enough per client to act as a fingerprint — the technique is usually called JA3 or its successor JA4. Chrome's OpenSSL/BoringSSL build produces one fingerprint. Python's requests, sitting on urllib3 and the system OpenSSL, produces a very different one. A bot-management layer compares the fingerprint against the claimed User-Agent, sees a client claiming to be Chrome while handshaking like Python, and serves the challenge. No header you set can change this, because the mismatch is decided before your headers exist.
The fix is to use a client that reproduces the browser handshake. curl_cffi binds to curl-impersonate, a curl build patched to emit browser-identical ClientHellos:
from curl_cffi import requests as cffi
session = cffi.Session(impersonate="chrome")
r = session.get(url, timeout=30)
That is the entire change. impersonate="chrome" sets the cipher order, the extension order, the ALPN list and the HTTP/2 settings frame to match a real Chrome build, and the challenge stops.
Two operational notes from running this daily for months:
Recycle the session. Long-lived sessions accumulate state and start getting challenged again after a few dozen requests. Constructing a fresh Session every N requests is cheaper than debugging why hour six of a backfill started failing.
Be a good client. This is public end-of-day data, fetched once per trading day, in a loop with a real timeout and a delay between requests. A backfill that walks eighteen months of history should be paced, not parallelised. Impersonating a browser handshake is how you get a correct response from a CDN; it is not a licence to hammer a public exchange's infrastructure. Cache locally and never re-fetch a date you already have.
The part that actually corrupts the data
Now the domain fact, which matters far more than the networking.
The IDX runs more than one order book, and the daily summary reports them together. The two that matter:
- Reguler — the continuous auction. Standard lots, price-time priority. This is what people mean by "the market".
- Negosiasi — the negotiated board. Bilaterally agreed trades, crossings, block deals, reported to the exchange rather than matched by it.
Both are real trades. They are not comparable, and they should never be summed into a single "volume" figure. A single negotiated crossing can be many multiples of a stock's normal daily regular turnover. If you blend them, a quiet stock with one block trade looks like it exploded, and any anomaly detector you build on top will fire on it constantly.
Here is the trap that costs a rebuild to discover. In the payload, the field named Value is already regular-board only, and the NonRegular* fields are a disjoint set covering the negotiated board. They are siblings, not a total and a part.
The instinct is to write:
regular = row["Value"] - row["NonRegularValue"] # WRONG
That subtracts the negotiated book from a number that never included it. The result is silently plausible: still positive for most rows, still ordered roughly sensibly, so nothing throws and no test that only checks types will catch it. It is simply about a third too small on the names where it matters most, which are exactly the names a liquidity screener exists to find.
The correct read is that there is nothing to reconcile:
reg = row["Value"] # regular board, already isolated
nonreg = row["NonRegularValue"] # negotiated board, disjoint
total = reg + nonreg # only if you genuinely want both
Track them as two separate series from ingestion onward. Every downstream metric — turnover ratio, moving averages, liquidity score, anomaly thresholds — is computed per board and never mixed. The ratio of non-regular to total is itself a useful signal: a stock whose turnover is suddenly 80% negotiated is telling you something specific, and you can only see it if you never merged the two.
Verify against the source, not against yourself
The general lesson is that a schema tells you a field's type, not its meaning. Value and NonRegularValue are both integers, both in rupiah, both plausible, and their relationship is a fact about the exchange's reporting convention that exists nowhere in the payload.
Two habits that catch this class of error:
Reconcile a single day by hand. Pick one date, take the ten largest names, and check your computed figures against the exchange's own published daily summary. If the total does not tie out on ten rows, it does not tie out on nine hundred. This is tedious exactly once.
Store raw, compute derived. Keep the untouched payload per day, immutable, and rebuild the panel from it. When a definitional error surfaces eighteen months in — and it will — a rebuild is one command instead of a re-scrape that may no longer be possible.
The networking problem is the one everybody writes about. The definitional one is the one that quietly ships wrong numbers to a dashboard you then trust.