> For the complete documentation index, see [llms.txt](https://optiview-1.gitbook.io/optiview-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://optiview-1.gitbook.io/optiview-docs/tech/data-and-etl-pipeline.md).

# Data & ETL Pipeline

<figure><img src="https://2222255890-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FywRbPvnDR41TjbXWb0nX%2Fuploads%2F42yGSiM42M2XMiw5IUHy%2Funknown.png?alt=media&amp;token=e5d12aa6-b74d-476f-93d4-1a1ac6e57fb5" alt=""><figcaption></figcaption></figure>

* **Ingestion sources**: on-chain nodes, protocol adapters, external price feeds, NFT data, news, and social signals
* **Streaming backbone** via Kafka / Redpanda
* **Batch orchestration** using Airflow / dbt
* **Data quality and lineage tracking** to prevent propagation of corrupted data
* **Storage layer**:
  * Operational databases (PostgreSQL, ClickHouse)
  * Data lake (Parquet on object storage)
  * Feature store for ML models
* End-to-end observability covering **latency, throughput, failure rate, and data freshness**