Data makes or breaks your research and your strategy. What started as a simple data pull to enable local backtesting turned into the foundation the rest of the platform runs on.
View on GitHubWhere it started
Before any of this there was a Python toolkit I kept adding to. API wrappers for news and market data, sentiment scoring over the headlines, and Monte Carlo for valuation and risk across equities, options and whole portfolios. Most of that logic ended up somewhere in the platform, the sentiment work in the factor engine and the rest in research. The toolkit is where I first ran into the real problem, which was that getting trustworthy data in front of the analysis was harder than anything I was doing with it afterwards.
The original goal was to run backtests on hardware more powerful than the cloud nodes available on QuantConnect. With QuantConnect's Lean Local engine as the plumbing, that meant pulling price history from a vendor, storing it, and converting it into the CSV-inside-zip layout Lean expects, checking local coverage first so the same data is never paid for twice.
That still works, and it is still the fastest way to run a backtest here: point a runner at a standard, unmodified Lean algorithm and it resolves the tickers, finds the gaps, fetches only what is missing, and runs. Everything below is what grew around it once more than one strategy depended on the data.
What it ingests
Market reference data is the whole scope: OHLCV, fundamentals, corporate actions, analyst estimates, macro series and entity identity. Fundamentals and macro come from commercial and academic providers; equity price history comes from a tick-level vendor.
That is all it does. Factor scoring, portfolio state and document extraction each live in their own code location, so a change to how sentiment is scored cannot break how prices are ingested.
Orchestration
Everything runs as Dagster assets on a multi-tenant Dagster deployment hosted on the cluster, four code locations covering data, factor scoring, portfolio and document extraction, each deployed from its own repository and reconciled by ArgoCD from Git.
Because the pipeline is built out of assets and not scripts, the dependency graph is explicit. If a fundamentals table is stale, everything downstream of it is visibly stale too. Long backfills run as separate GPU scheduled jobs, so a multi year rebuild never blocks the daily run.
Point-in-time correctness
Fundamental data is keyed on filing date, not period end, so a backtest never sees a figure before the date it was actually published.
None of this is interesting to build. It is also what decides whether a backtest is worth anything. Look ahead bias through restated or early visible fundamentals is the most common reason a backtest shows alpha that disappears in live trading, and you cannot see it happening unless the data layer stops it.
Storage: layered and dual-served
Data lands in layers, raw then conformed then curated, so I can reprocess from the layer below instead of going back to the vendor.
The curated layer is dual-served. An Iceberg lakehouse behind a REST catalog is the single source of truth for tabular data across engines, so Dagster, DuckDB and Spark all read the same tables; PostgreSQL with TimescaleDB serves low-latency reads when a strategy or the factor API needs one day's slice. A time-series store and a Kafka pipeline handle streaming market data, and object storage sits underneath the lakehouse.
Compute, and where it's going
The cluster runs a highly-available control plane with GPU-capable workers, managed as code and reconciled by ArgoCD, alongside the usual production apparatus. Prometheus and Grafana with long-term metrics, log aggregation, scheduled backups, sealed secrets and load-balanced ingress.
Distributed work is single-node DuckDB today. Spark is on the roadmap and I have written down what it has to beat before I bring it in. A second cluster at another site is up and being wired for cross-site compute, and I am starting to move some workloads to the cloud when they outgrow the hardware I have.
What it feeds
The platform publishes prices, point-in-time fundamentals, corporate actions, macro series and entity identity. Factor scoring reads from those tables, the portfolio layer reads from both, and Lean reads exported data straight off disk for backtests.