Data makes or breaks your research and your strategy. What started as a simple data pull to enable local backtesting turned into the foundation the rest of the platform runs on.
View on GitHubWhere it started
Before any of this there was a Python toolkit I kept adding to. API wrappers for news and market data, sentiment scoring over the headlines, and Monte Carlo for valuation and risk across equities, options and whole portfolios. Most of that logic ended up somewhere in the platform, the sentiment work in the factor engine and the rest in research. The toolkit is where I first ran into the real problem, which was that getting trustworthy data in front of the analysis was harder than anything I was doing with it afterwards.
The original goal was to run backtests on hardware more powerful than the cloud nodes available on QuantConnect. With QuantConnect's LEAN engine running locally as the plumbing, that meant pulling price history from a vendor, storing it, and converting it into the CSV-inside-zip layout LEAN expects, checking local coverage first so the same data is never paid for twice.
That still works, and it is still the fastest way to run a backtest here: point a runner at a standard, unmodified LEAN algorithm and it resolves the tickers, finds the gaps, fetches only what is missing, and runs. Everything below is what grew around it once more than one strategy depended on the data.
What it ingests
Market reference data is the whole scope: OHLCV, fundamentals, corporate actions, analyst estimates, macro series and entity identity. Fundamentals and macro come from commercial and academic providers; equity price history comes from a tick-level vendor.
That is all it does. Factor scoring, portfolio state and the dashboard each live in their own code location, so a change to how sentiment is scored cannot break how prices are ingested.
Orchestration
Everything runs as Dagster assets on a multi-tenant Dagster deployment hosted on the cluster, four code locations covering data, factor scoring, portfolio and the PortDash dashboard, each deployed from its own repository and reconciled by ArgoCD from Git.
Because the pipeline is built out of assets and not scripts, the dependency graph is explicit. If a fundamentals table is stale, everything downstream of it is visibly stale too. Long backfills run as separate GPU scheduled jobs, so a multi-year rebuild never blocks the daily run.
Point-in-time correctness
Fundamental data is keyed on filing date, not period end, so a backtest never sees a figure before the date it was actually published.
None of this is interesting to build. It is also what decides whether a backtest is worth anything. Look-ahead bias through restated or early-visible fundamentals is the most common reason a backtest shows alpha that disappears in live trading, and you cannot see it happening unless the data layer stops it.
Storage: layered and dual-served
Data lands in layers, raw then conformed then curated, so I can reprocess from the layer below instead of going back to the vendor.
The curated layer is dual-served. An Iceberg lakehouse behind a REST catalog is the single source of truth for tabular data across engines, so Dagster, DuckDB and Spark all read the same tables; PostgreSQL with TimescaleDB serves low-latency reads when a strategy or the factor API needs one day's slice. A time-series store and a Kafka pipeline handle streaming market data, and object storage sits underneath the lakehouse.
Compute, and where it is going
The cluster runs a highly available control plane with GPU-capable workers, managed as code and reconciled by ArgoCD, alongside the usual production apparatus. Prometheus and Grafana with long-term metrics, log aggregation, scheduled backups, sealed secrets and load-balanced ingress.
Distributed work is single-node DuckDB today. Spark is on the roadmap and I have written down what it has to beat before I bring it in. A second cluster at another site is up and being wired for cross-site compute, and I am starting to move some workloads to the cloud when they outgrow the hardware I have.
What it feeds
The platform publishes prices, point-in-time fundamentals, corporate actions, macro series and entity identity. Factor scoring reads from those tables, the portfolio layer reads from both, and LEAN reads exported data straight off disk for backtests.
The code
The code is public now. The schema ships and the vendor data does not, because none of it is mine to redistribute.
The part worth reading is the point-in-time handling. Join a fundamental to the period it describes instead of the date it was filed, and you leak weeks of future information into every value and quality signal. The backtest that comes out looks good, which is what makes it expensive.