A trading system is a function of its input. Almost all of the attention goes to the logic and almost none to whether the data feeding it is what it claims to be.
That is the wrong allocation, because bad data does not produce an error. It produces a confident, wrong answer that looks exactly like a right one.
The four ways historical data misleads
Unadjusted prices. When a company splits its shares or issues bonus shares, the price changes without any economic event. Unadjusted history therefore contains an enormous overnight fall that never happened. A backtest reading it sees a crash, and any rule reacting to sharp falls will produce a result that is entirely an artefact. This is the single most common way an Indian equity backtest is quietly wrong.
Survivorship. If your universe is today's index constituents tested over ten years, you have tested the companies that survived and prospered enough to still be in the index. The ones removed along the way are absent, and their absence flatters every result.
Missing candles. Gaps happen — a feed interruption, a halted instrument, a provider's own error. Code that assumes consecutive rows are consecutive periods will compute an indicator across a gap without noticing, and the value it produces is meaningless rather than merely inaccurate.
The wrong series after a symbol change. Companies rename, merge and get reclassified, and instrument tokens change. History stitched across such a change can silently contain two different companies.
The defence for all four is the same and it is dull: validate the input before trusting the output. Check row counts against the expected number of sessions, check for impossible values, and check that a split-adjusted series does not contain unexplained overnight moves of implausible size.
Live data fails differently
Historical data is wrong consistently. Live data is wrong occasionally, which is harder.
Bad ticks. A single quote far away from the surrounding prices, from a feed glitch or a genuinely odd trade in a thin instrument. A system that treats every tick as truth can trigger an entry or a stop on a price that existed for a fraction of a second. A simple sanity filter — reject any tick beyond a plausible distance from the recent price, and require confirmation before acting on an extreme value — prevents most of this.
Stale data. Worse than missing data, because it looks fine. A feed that stops updating leaves your last known price sitting there, and a system reading it believes the market is flat while it is actually moving. Every live system should track the age of its last update and refuse to act on anything older than a defined threshold.
Out-of-order or duplicated messages. Streams do not guarantee ordering. Use the exchange timestamp rather than arrival time, and discard anything older than what you have already processed.
Reconnection gaps. When a connection drops and re-establishes, the period in between is missing. A system that resumes as though nothing happened is operating on an incomplete picture. On reconnect, fetch the interval that was missed rather than assuming continuity.
The Indian specifics that catch people
The trading calendar is not a weekday calendar. Indian markets close for a long list of holidays that changes every year, and there are occasional special sessions. Any code that generates expected dates by skipping weekends will be wrong several times a year, and the errors show up as apparent gaps or as indicators computed over the wrong number of periods. Use the exchange's published calendar rather than deriving one.
Derivative contracts expire, and the series must be built deliberately. A continuous futures series is constructed by joining successive contracts, and the joining method changes the results. Splicing raw prices creates artificial jumps at every roll. How you handle that is a decision to make explicitly rather than inherit from whatever a library does by default.
Lot sizes and strike intervals get revised. A system caching contract specifications will keep using stale values after a revision, which produces orders of the wrong size. Refresh the instrument master regularly rather than once.
Circuit limits and surveillance measures change what trades. An instrument locked at a circuit has quotes on one side only, and a share moved to periodic call auctions does not trade continuously at all. Data from those sessions is genuine and does not mean what normal data means.
Corporate action dates matter as much as the adjustment. Knowing a split happened is not enough; the system needs to know when, so historical analysis and live positions are treated correctly on either side of it.
A validation layer worth building once
The useful architectural idea is to stop treating data ingestion as plumbing. Put a validation step between your source and your strategy, and let it be the only path in.
What it should do, in rough order of value:
Check completeness. Expected sessions against actual rows, using the exchange calendar. Fail loudly on a mismatch rather than continuing with a gap.
Check plausibility. High not below low, no zero or negative prices, no volume where the instrument did not trade, and no single-period move beyond a threshold you have chosen deliberately.
Check freshness, live. Timestamp on every update, with a maximum age beyond which the system stops acting rather than acts on old information.
Log what was rejected. Rejections are the most informative thing the layer produces. A sudden rise in rejected ticks usually means something upstream has changed, and that is worth knowing before it reaches a position.
Fail closed. When data cannot be validated, the correct behaviour is to stop trading and alert, not to proceed on a best guess. A system that continues on unvalidated input is making decisions with unknown inputs, which is worse than making none.
Built once, this layer serves every strategy afterwards. It is also the part of the system that nobody enjoys building and everybody wishes they had when a result turns out to have been an artefact.
An account with data access
Instrument master, candles and a live feed to validate against. Free to open
API behaviour, instrument masters, calendars and adjustment conventions differ between providers and are revised from time to time — read your provider's current documentation rather than relying on this description. This is educational material, not advice. We run no tips group and no signal service, we manage nobody's money, and no return is promised. Automated trading carries a real risk of loss.
Disclosure: the account-opening link on this page is under Atul Shrivastava's Zerodha Authorised Person registration (NSE AP Reg: AP2516003481; Zerodha Broking Ltd. SEBI Reg: INZ000031633) and earns a revenue share. TheFinBaba is not a SEBI-registered Investment Adviser — this content is educational, not investment advice.
Frequently Asked Questions
Why does my backtest show a crash that never happened?
Almost certainly unadjusted data around a split or bonus issue. The price changes mechanically without any economic event, and unadjusted history records it as an enormous overnight fall. Any rule reacting to sharp moves will produce results that are entirely an artefact.
How do I handle missing candles?
Detect them first, by comparing expected sessions from the exchange calendar against the rows you actually have. Then decide deliberately whether to fill, skip or refuse to trade that instrument. The dangerous case is code that assumes consecutive rows are consecutive periods.
What is a bad tick and how do I filter it?
A quote far away from surrounding prices, usually from a feed glitch or an odd trade in a thin instrument. Reject values beyond a plausible distance from the recent price and require confirmation before acting on an extreme one, rather than treating every tick as truth.
Why is stale data worse than missing data?
Because it looks valid. A stopped feed leaves the last price in place, so the system believes the market is flat while it is moving. Track the age of the last update and stop acting when it exceeds a threshold you have set.
Can I generate the trading calendar by skipping weekends?
No. Indian markets close for a long list of holidays that changes each year, and there are occasional special sessions. Use the exchange's published calendar, or your date logic will be wrong several times a year in ways that look like data gaps.
Related Reading
- Fetching the history in the first place
- Where a backtest lies to you
- The adjustments that break a series
- The execution side of the same problem
- Receiving the live feed
- Data handling, taught as a layer
Disclaimer: TheFinBaba provides educational content only - this is not investment advice. Trading involves risk of loss.
