Building a reliable market data pipeline

Downloading prices into a notebook works for a quick experiment. It stops working the moment several people, strategies or reports depend on the same data. At that point you need a pipeline: a repeatable process that collects data, checks it, stores it in a consistent shape and makes it available to whoever needs it.

The good news is that the principles are the same whether you are handling a few hundred symbols of daily bars or a large tick-level dataset.

The stages of a pipeline

1. Ingest: pull data from vendors, exchanges or internal systems.
2. Land raw: store exactly what was received, unmodified.
3. Validate: run automated checks and quarantine anything that fails.
4. Normalize: convert to a standard schema, identifiers and time zone.
5. Store: write clean data in a format optimized for analysis.
6. Serve: expose it to research, dashboards and applications through a consistent interface.

Keep an immutable raw layer

Always keep a copy of the data exactly as it arrived. If a cleaning rule turns out to be wrong, or a vendor later revises its history, you can reprocess from the raw layer instead of re-downloading or guessing what changed. Storage is usually cheaper than the cost of an unrecoverable mistake.

Make loads idempotent

An idempotent job produces the same result whether it runs once or five times. This matters because jobs fail, get retried and occasionally run twice. A common pattern is to partition data by date and have each run fully replace its own partition rather than append to it.

from pathlib import Path
import pandas as pd

def write_partition(df: pd.DataFrame, root: str, dataset: str, day: str) -> Path:
    """Write one day's data; re-running replaces the same partition safely."""
    out_dir = Path(root) / dataset / f"date={day}"
    out_dir.mkdir(parents=True, exist_ok=True)
    tmp = out_dir / "part.parquet.tmp"
    final = out_dir / "part.parquet"
    df.to_parquet(tmp, index=False)
    tmp.replace(final)  # atomic rename on the same filesystem
    return final

Writing to a temporary file and renaming it means readers never see a half-written partition.

Get identifiers and time right

Ticker symbols change, get reused and differ between venues. Map every record to a stable internal identifier and keep a history of symbol changes. Store timestamps in UTC with explicit time zone information, and keep both the event time (when a trade happened) and the ingestion time (when you received it). The difference between those two is often essential for avoiding look-ahead bias.

Validate automatically

Checks should run on every load, not only when someone notices a problem.

def validate_bars(df: pd.DataFrame) -> list[str]:
    errors = []
    if df.duplicated(subset=["instrument_id", "timestamp"]).any():
        errors.append("duplicate instrument/timestamp rows")
    if (df[["open", "high", "low", "close"]] <= 0).any().any():
        errors.append("non-positive prices")
    if (df["high"] < df[["open", "close", "low"]].max(axis=1)).any():
        errors.append("high below open, close or low")
    if df["timestamp"].isna().any():
        errors.append("missing timestamps")
    return errors

Other useful checks include row counts compared with the previous day, the share of instruments with no update, and returns beyond a plausible range. Failed data should be quarantined and reported, never silently dropped.

Think point-in-time

Research needs to know what was known at each moment. Index constituents change, fundamentals are restated and vendors correct history. Rather than overwriting old values, record each version with the time it became valid. This is more work up front but it is the foundation for backtests free of look-ahead bias.

Choose storage for the access pattern

Columnar file formats such as Parquet, partitioned by date and possibly by instrument, work well for analytical workloads that scan many rows but few columns. Time-series databases help when many users query recent data interactively. Many teams use both: files for bulk research and a database for serving.

Monitor and document

• Track data freshness and alert when a feed is late.
• Log every run with row counts, check results and duration.
• Version schemas and transformation code so results can be reproduced.
• Maintain a short data dictionary describing each field, its units and its source.

A good pipeline is boring in the best sense: it runs every day, fails loudly when something is wrong and lets you reproduce any past result.

See how Quantzivo Studio's planned developer tools approach data access, or learn to build data-driven applications in the Building Financial Dashboards & Applications course.