Ingest
Newton's first law applies to data. Data at rest tends to stay at rest, while data in motion tends to stay in motion. For a time-series platform, ingest is not an afterthought. It is the foundation that determines whether data ever becomes usable in the first place.
PredictiveGrid was architected to ingest virtually every form of relevant time-series data, at the scales physical systems actually produce. The platform was benchmarked several years ago at a sustained ingest rate of over 125 million measurements per second. That benchmark was not the upper bound; it was the point at which cloud infrastructure costs limited further testing.
Streaming protocols
PredictiveGrid supports streaming ingest from the protocols that dominate utility and industrial telemetry: IEEE C37.118 for synchrophasor data, STTP (IEEE 2664-2024) for Streaming Telemetry Transport Protocol deployments, IEC 61850 for substation automation, Modbus for SCADA and industrial control systems, and DNP3 for distribution automation. Additional streaming ingesters have been built for proprietary continuous point-on-wave sensors, FNET Universal Grid Analyzer devices, PowerSide PQube power quality monitors, and the U.S. Geological Survey's magnetic observatories.
The ingest framework is modular by design. Adding support for a new streaming protocol typically requires hours of engineering work, not weeks. This modularity reflects six years of accumulated experience integrating diverse utility data sources and direct relationships with the open-source maintainers of the most widely used protocol libraries.
Batch and file ingest
For data that doesn't stream natively, PredictiveGrid supports asynchronous ingest from PQDIF, COMTRADE, CSV, Parquet, JSON, Apache Arrow, Apache Feather, AVRO, and PI Historian archives. We will happily support other formats not mentioned here as needed. These formats cover the working set of utility data engineering: power quality reports filed in PQDIF, fault recordings exported as COMTRADE, bulk historical loads from existing historians, and the columnar formats used by modern data engineering pipelines.
Industrial system integration
The ingest architecture integrates with the systems that already exist in utility IT and OT environments: Energy Management Systems (EMS), Advanced Distribution Management Systems (ADMS), substation automation gateways, and time-synchronized data concentrators including OpenPDC, EPG's ePDC, and GE's PhasorPoint. PingThings has more than half a decade of operational deployment experience with OpenPDC specifically, working in close collaboration with the Grid Protection Alliance.
This is the integration work that vendors often underestimate. Supporting a protocol is necessary but not sufficient. The actual deployment involves negotiating firewall rules, mapping device naming conventions, validating timestamp synchronization across the fleet, and handling the inevitable edge cases that arise when data sources have been operating in isolation for decades. The ingest layer is designed to absorb that complexity rather than push it onto the customer.
Resilience and replay
The ingest system is built on Docker containers with replay and backfill capabilities. When connectivity to a data source is interrupted, the platform can replay missed data once the source reconnects, with no loss of fidelity or temporal precision. Out-of-order data is handled natively by the storage layer; ingest does not need to enforce strict ordering to maintain correctness.
Storage, Query, and Context
The storage and query layer is the technical core of the platform. It is built on BTrDB, the Berkeley Tree Database, originally developed at UC Berkeley with funding from ARPA-E and published at FAST '16 (USENIX Conference on File and Storage Technologies). PingThings operates an enterprise-grade implementation of BTrDB tuned for the workloads physical-system observability actually demands.
The performance envelope is concrete and benchmarked. PredictiveGrid's time-series API sustains 1 to 2 million points per second on single-stream writes, 5 to 10 million points per second on single-stream reads, and 10 to 20 million points per second on multi-stream reads in tested configurations. All performance is horizontally scalable by adding nodes. The API is regularly benchmarked to ensure that platform updates do not introduce regressions.
Time-series store
The data structure underneath BTrDB is a time-partitioning, copy-on-write, version-annotated k-ary tree. Raw measurements and timestamps live in the leaves. Each higher level of the tree summarizes the level below with statistical aggregates: min, mean, max, standard deviation, sample count, and version number. As data is written, the relevant aggregates are recomputed in memory.
The practical consequence is that aggregations over arbitrary time intervals (a year, a month, a week, a day, a second) are answered nearly instantaneously. Querying a higher level of the tree is, structurally, asking the database for a rollup over a larger time range. The database does not compute aggregates at query time; it has already computed them at write time.
The platform supports both continuous and sparse time series without compromising performance, at resolutions up to 1 billion samples per second per stream. Sensors with dynamic sample rates (those that normally capture at 1 Hz but accelerate to 100 Hz on a trigger condition) are handled natively, without configuration changes.
Versioning and reproducibility
The tree is copy-on-write. Every write produces a new version. Previous versions remain accessible at virtually zero additional cost; there is no performance penalty for querying historical state. A data stream can be queried at any point in its past, much like checking out a previous commit in Git. The database can also be queried for change sets, the data that has arrived since a particular version, enabling efficient answers to questions like "what data has arrived since yesterday?"
For scientific reproducibility, this is the architectural difference between "we ran the analysis last March" and "we can re-run the exact analysis on the exact data as it existed last March." For audit and forensic workflows, it is the difference between operational logs and a versioned record of system state.
Contextualization engine
Raw time series alone is not enough. A sensor measurement without context is a number without meaning. PredictiveGrid treats metadata as a first-class subsystem rather than an annotation layer, providing the semantic foundation that turns telemetry into engineering objects.
Every signal ingested into PredictiveGrid is accompanied by a structured set of metadata fields describing its origin, measurement context, technical attributes, and operational lineage: source device ID, measurement type, unit of measure, sampling rate, substation location, asset identifier. The metadata system is extensible. Users can add ad-hoc key-value pairs on demand. The baseline schema itself can be customized per-deployment to include asset hierarchy levels, maintenance zones, sensor firmware versions, or regulatory compliance domains.
Performance matters at scale. As utilities expand from thousands to hundreds of millions of signals, metadata query response time determines whether discovery is usable or impossible. PredictiveGrid is architected for sub-100 millisecond metadata query response at signal counts up to 100 million.
Topology, geospatial, and artifacts
The platform represents the physical relationships between sensors through a dedicated topology model. Analytics that span physical structure (fault propagation analysis, asset health correlation, geographic event detection) depend on this representation being a queryable part of the platform rather than an afterthought stored in a separate system.
Geospatial context is similarly first-class. Signals carry location metadata, the platform supports geographic queries, and visualizations can render signals on maps with map-tile integration. The geospatial layer is what makes "show me every PMU within fifty miles of this fault" a single query instead of a data engineering project.
Analytics outputs, models, and processed signals are stored back into the platform alongside raw data as user-created artifacts. All metadata is interoperable with CIM, IEC 61850, and PI AF.
Compression and tiered storage
PredictiveGrid uses an intelligent tiered storage system that blends storage tiers with different price and performance characteristics. Time-series access patterns are typically a function of age, so the platform places data on storage tiers that match its expected access pattern.
Critically, the compression is lossless. Legacy time-series vendors in this space rely on lossy compression schemes that discard measurements or reduce timestamp resolution to keep storage costs manageable. PredictiveGrid retains every measurement at its original precision, using a combination of best-in-class lossless compression, adjustable bit resolution where appropriate, and low-cost storage tiers for older data.
APIs
Time-series data is only useful if applications can access it. PredictiveGrid exposes its data through two API surfaces optimized for different access patterns, with first-class language bindings for the languages utility engineers and data scientists actually use.
High-performance query API
The high-performance binary query API offers users direct access to data at millions of measurements per second for time-series queries against datasets that span trillions of points. The API uses Apache Arrow as its wire format, enabling zero-copy reads and parallelizable batch operations on modern hardware. Downstream consumers in Python, R, Rust, Java, or any language with Arrow support can operate on returned data directly in memory, without serialization or deserialization overhead.
This API is designed for interactive visualization, real-time dashboards, ML training and inference, and any workload where time-series data needs to remain instantaneous to human or programmatic interaction.
REST and integration APIs
For everything else (automation, ETL pipelines, bulk operations, third-party system integration) PredictiveGrid provides RESTful APIs that follow standard patterns. These APIs handle ingest, metadata management, signal discovery, user and permission management, and platform administration. They are designed for accessibility across a wide range of systems and language environments.
Language bindings
PredictiveGrid ships with a first-class Python client library, tightly coupled with the core API and optimized for JupyterLab environments. The Python client supports time-series queries, metadata search, and artifact management with full error handling and performance tuning. Production-ready bindings have also been delivered for C# and Julia. The underlying API design supports rapid generation of bindings in additional languages as deployments require.
Access control and rate limiting
API access is governed by the platform's authentication and authorization subsystem. Every API request is authenticated, every operation is checked against the requester's permissions, every access is logged. Rate limiting is applied per-tenant and per-key. Sensitive operations (bulk deletion, schema modification, permission changes) require elevated permissions and produce audit trail entries that are queryable for compliance purposes.
Analytics and Applications
The analytics layer is where time-series data becomes operational intelligence. PredictiveGrid does not impose a single analytics paradigm. Instead, it provides a set of interconnected tools that serve users across the spectrum from operational engineers to research data scientists, all unified by the same underlying Python environment and the same data substrate.
Visualization and dashboards
PredictiveGrid integrates Grafana natively as the dashboarding layer. Grafana connects to the underlying time-series engine through low-latency connectors that respect the platform's metadata-aware query patterns. Users build dashboards visualizing individual signals, geographic regions, asset groups, or event conditions, with the platform's full-resolution data available at sub-second response times.
Interactive data exploration
The platform's Multi-Resolution Plotter is purpose-built for exploratory analysis of arbitrarily large time-series datasets. It implements the principle of "overview first, zoom and filter, then details-on-demand" by exploiting BTrDB's variable-resolution query capabilities. The visualizer requests the ideal amount of data for the current display (one data point per pixel) at consistently low latency, regardless of the underlying dataset size.
The plotter has been refined to handle environments with more than 10,000 streams and supports interactive geospatial stream selection for natural exploration of fleet-scale telemetry.
Notebooks and ad-hoc analysis
PredictiveGrid embeds a browser-accessible JupyterLab environment directly within the platform. The notebook environment is co-located with the storage layer, so engineers have instantaneous access to real-time and historical telemetry without data extraction, duplication, or external ETL. A pre-configured analytics stack includes NumPy, Pandas, SciPy, Matplotlib, and Scikit-learn. Additional packages can be installed from PyPI or private sources subject to approval workflows.
Operationalization is frictionless. Any notebook or script developed in the Jupyter environment can be scheduled, containerized, and deployed as a persistent service within the same infrastructure. This shrinks the traditional handoff between research and production from months to minutes, with the deployed service backed by the same data environment it was developed in.
Integrated Git support via JupyterLab-Git extensions provides version control, collaborative development, and structured review workflows.
DISTIL: signal processing at scale
DISTIL is PredictiveGrid's framework for building scalable analytics pipelines on time-series data, originally published at SIGMOD '18. DISTIL consists of two components: distillers, which implement the functions or transformations applied to sensor data, and the distill processing framework, which handles performance optimization and bookkeeping for multiple interleaved streams arriving at different rates, possibly out of order.
Each distiller defines a precompute function (specifying what data is needed) and a compute function (operating on that data to produce results). Distillers emit derived time series, events, or other outputs that are stored back into the platform alongside raw measurements. Everything is versioned: the data, the distillers, and the intermediate streams. As data changes, the framework determines what needs to be recomputed to produce consistent results with precise provenance.
The architectural advantage is that months or years of analytical results can be queried in milliseconds with pre-built temporal aggregations, as if the data were ingested directly from a sensor.
Optional analytics frameworks
For organizations with existing investments in big-data frameworks, PredictiveGrid can optionally provide managed clusters within the platform, co-located with the data: Apache Spark for large-scale distributed processing, TensorFlow for deep learning workflows, and Ray for scalable Python applications and distributed ML.
Security and Operations
PredictiveGrid is built for critical infrastructure environments. Cybersecurity is foundational, not bolted on. The platform's security architecture is aligned with NERC CIP, NIST 800-53, and IEC 62443 standards. PredictiveGrid is SOC 2 Type 2 compliant, currently undergoing its second annual audit, and has completed its second annual penetration test. The first ISO 27001 audit is in progress, with certification expected in 2026.
Defense in depth
The platform implements a defense-in-depth security architecture across three layers: data security and encryption, application-level access control, and software development and operations security. Data is encrypted in transit and at rest using industry-standard cryptographic protocols. Encryption keys are managed through dedicated key management infrastructure with rotation, audit, and recovery procedures.
Identity and access control
The platform's IAM subsystem supports role-based access control with fine-grained, hierarchical permissions. Streams are identified by collection and name, structured hierarchically. Permissions can be assigned at any level of the hierarchy, similar to how filesystem permissions work in shared file systems. Users are organized into groups, and groups carry the capabilities (READ, INSERT, DELETE, OBLITERATE, API access, plotter access) that apply across prefix-defined regions of the data.
Custom roles can be defined per-deployment, external identity providers can be mapped to internal roles, and access policies can be integrated with infrastructure-as-code tools for version-controlled, reproducible security governance.
Tenant isolation
For enterprise deployments, PredictiveGrid is provisioned as a dedicated, single-tenant instance. Each deployment runs in its own segregated compute and storage environment. There is no shared infrastructure or runtime dependencies with other customers. This isolation enables custom tuning, tailored scaling policies, and stricter access controls aligned with utility-specific compliance requirements.
Audit and access reporting
Every access is logged. Administrators can view summaries of data usage filtered by user, role, organizational unit, or timeframe. Reports can be exported on demand or scheduled. Audit logs can be cryptographically signed to support forensic workflows.
Backup and disaster recovery
PredictiveGrid implements a multi-layered backup and disaster recovery strategy. Nightly snapshots are taken of all stateful disk volumes (JupyterHub, etcd, PostgreSQL metadata, tileserver, intermediate data layers) and retained for a configurable period aligned with regulatory and operational requirements. Object-store-based time-series data supports native versioning; previous versions of each blob are retained and recoverable for the duration of the retention policy.
Operational monitoring
The platform itself is monitored using the same observability discipline it provides to its customers. Internal metrics, health checks, and alerting feed an incident response process with 24x7x365 coverage. Container images are continuously scanned for CVEs and threats are ranked by potential impact and probability of occurrence.