↓Skip to main content

Field-to-Cloud Data Pipeline

·595 words·3 mins

Systems architecture decisions get made early, before the data is real or the assumptions are tested. By the time you understand what you actually need, the right sampling rate, the right granularity, the right query patterns for your model, the structure is already set. And if that structure runs through a managed platform, you don’t fully own what comes out the other side: the data has been shaped by someone else’s schema, throttled by someone else’s rate limits, and priced by someone else’s usage tiers.

Metric Link is a platform using proven technologies, built on the premise that the data belongs to you, completely and from the start. A $6 microcontroller handles the edge; a self-hosted time-series database handles storage and querying, on your terms, at your resolution, with no intermediary deciding what fidelity you get. The architecture is intentionally decoupled so the edge stays simple, complexity lives in infrastructure you control, and your ML pipeline connects to data that hasn’t been touched by a managed service.

Metric Link
The field device, a Raspberry Pi Pico W running Zephyr RTOS

The Decoupled Architecture #

The system draws a strict line between the physical edge, data logistics, and storage. This decoupling means the field device remains unaware of the database backend, allowing the cloud infrastructure to evolve without forcing fleet-wide firmware updates.

%%{init: {"look": "handDrawn"}}%% flowchart TD subgraph Field ["Field Components"] direction TB S[/Sensors/] --> T[[Data Link]] end subgraph Infrastructure ["Infrastructure Components"] direction TB Broker[[Message Broker]] Persistence[(Data Persistence)] Broker -- "Ingestion" --> Persistence end subgraph Observability ["Observability Components"] direction TB Monitoring[[Monitoring & Alerts]] end T -- "Secure Ingress" --> Broker Persistence -- "Query/Alert" --> Monitoring
  • Field Edge: Built on a real-time operating system, the firmware isolates physical sensor collection from network transport. It interacts exclusively with a single, secured ingress point.
  • Infrastructure Core: A central message broker manages data routing, ensuring new downstream consumers can be added without modifying the field devices.
  • Observability: VictoriaMetrics powers the storage layer. Relational time-series databases frequently create severe disk bottlenecks and underutilize compute resources under heavy workloads; native time-series storage drastically reduces long-term operational costs.[1]

victoria-metrics
Observability dashboards powered by VictoriaMetrics

The Scalability Path #

System growth should not require a fundamental rewrite. The broker and storage layers can independently migrate from single-node instances to distributed, highly available clusters. The message broker remains the fixed point in the flow, making enterprise scaling a localized infrastructure task rather than an edge engineering problem.

Architectural Gaps for Enterprise Deployments #

  • Hardware Roots of Trust: Internal microcontroller security features are vulnerable to physical fault-injection attacks.[2] External, hardened secure elements are required to protect cryptographic identities.[3]
  • Lifecycle Management: Remote firmware updates must account for silicon-specific limitations, requiring precise memory management to prevent execution crashes during flash swaps.
  • Semantic Interoperability: Adopting structured standards like Sparkplug B prevents data silos; decoding complex formats should be offloaded to the central broker to keep the edge lightweight.[4]
  • Data Resilience: Local non-volatile buffering on the device to survive network partitioning without metric loss.
  • Regulatory Alignment: Mapping controls to IEC 62443 and NIS2 — enforcing transit encryption, tamper resistance, and continuous audit logging.[5]

The complete implementation is on GitHub. For silicon limitations, library selection, and the full scalability path, refer to the Technical Overview.

References #

  1. Valialkin, A. “High-cardinality TSDB benchmarks: VictoriaMetrics vs TimescaleDB vs InfluxDB.” Medium, https://valyala.medium.com/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b
  2. “Security Highlight: RP2350 Hardware Attacks.” Keysight Blogs, March 26, 2025, https://www.keysight.com/blogs/en/tech/nwvs/2025/03/26/security-highlight-rp2350-hardware-attacks
  3. “Your Keys, Your Fortress on a Budget: Building a DIY Hardware Security Module with PicoKeys.” GitHub/PicoKeys, https://www.picokeys.com/
  4. “Introduction to MQTT Sparkplug” HiveMQ, https://www.hivemq.com/blog/mqtt-sparkplug-essentials-part-1-introduction/
  5. “Achieving NIS2 Compliance Through IEC 62443: A Practical Guide.” Shieldworkz OT security, https://shieldworkz.com/regulatory-playbooks/nis2-directive-achieving-nis2-compliance-through-iec-62443