Detecting ransomware in near real time, from block IO alone

Crypto ransomware has a tell. Before the ransom note ever appears, the malware has to read your data, encrypt it, and write it back — and that activity leaves a distinctive fingerprint at the block storage layer. Bursts of writes, reads immediately followed by rewrites of the same blocks, IO scattered across the address space as forked worker processes attack many files in parallel, and payloads whose byte distributions look statistically random. None of these signals requires a signature database or visibility into the file system. They are visible in the raw stream of block IO.

This is a real-time ransomware detection system built on exactly that observation. The detection landscape today spans several approaches: signature-based antivirus, block IO trace analysis (Zerto, IBM FlashCore), file system and system call tracing (IBM Defender, Windows Defender), network monitoring (CrowdStrike, Palo Alto), and backup image scanning (Rubrik, CyberSense). We focus on block IO traces because they are cheap to capture and hard for an attacker to fake: a lightweight block-level IO Interceptor sits in the IO path on the host, captures the trace of every read and write, and ships it to a cloud-based engine where the heavy lifting — feature extraction and machine learning — happens out of band. The host stays fast; the analysis happens elsewhere.

The whole point of the design is time. The longer a ransomware process runs undetected, the more files it encrypts and the wider the blast radius. So the goal is near-real-time monitoring: shrink the window between the first malicious write and a confident verdict to seconds, not minutes or hours, so that detection fires while the attack is still small and containment is still cheap. A sliding-window feature pipeline running continuously on the streamed traces is what makes that low-latency verdict possible.

The system has two halves: a training pipeline that turns accumulated IO traces into an incrementally trained classification model on IBM watsonx, and an inference pipeline that uses that model to classify live IO activity in near real time. Both halves rest on the same foundation — a carefully engineered feature set computed over sliding windows — so we start with what gets captured and how it is turned into features, then walk through training, and finally inference.

What we capture, and where

On each protected host, an IO Interceptor — a filter driver sitting in the block IO path — captures the IO stream for both reads and writes. For every IO it records a small metadata entry: timestamp, IO type (read or write), logical block address (LBA), transfer size in bytes, and the volume/disk ID. For writes it also captures the payload bytes, which are needed for entropy analysis. The Interceptor is intentionally thin: it captures and forwards, and does as little computation as possible in the data path so the host pays almost nothing. Bulk operations such as full-volume copies are flagged separately so they can be excluded from analysis, and read metadata can be encoded compactly if trace volume becomes a concern.

A gateway on the host side batches these trace records and write buffers and streams them to the cloud engine. There, an ingestion gateway hands the stream to the detection module that runs the feature pipeline and the model. To keep latency low, the module is co-located with the ingestion path — ideally on the same node to avoid buffer copies, otherwise within the same cluster or VPC. All traffic for a given volume is routed to a single detection-module instance so that windowed analysis sees a complete, ordered view of that volume’s activity.

Feature engineering: turning a stream of IOs into window-level signals

Block IO carries no semantic information — no file names, no process IDs, no system calls. All the detector ever sees per operation is a timestamp, the IO type (read or write), a logical block address, a transfer size, and, for writes, the payload bytes. The job of feature engineering is to compress a window of these raw events into a single vector that makes the difference between mass encryption and ordinary work visible to a model. We keep the feature set deliberately simple and cheap to compute, organized around three families.

Throughput features. For each window we compute the average read throughput, average write throughput, and average rewrite throughput — the rate of writes landing on LBAs that were read shortly before. The combination is what matters. Encryption is read-modify-write at scale: data is read in bulk, transformed, and written straight back over (or near) the blocks it came from. Plenty of benign workloads are read-heavy or write-heavy, but very few sustain high read, high write, and high rewrite rates simultaneously, which makes the trio a strong joint signal even before entropy enters the picture.

Entropy features. For every write buffer we compute the Shannon entropy of its bytes, then summarize per window: the mean entropy across buffers and the maximum entropy observed. The mean captures the overall character of what is being written, but on its own it can be diluted — a strain that encrypts only parts of each file, or interleaves encrypted writes with normal activity, can keep the window average looking unremarkable. The maximum is the cheap counterweight: even a partially encrypting attacker still produces individual buffers at near-random entropy, and the max will catch them.

Histogram features. Averages and maxima describe a distribution with two numbers; histograms describe its shape, which is where the real discriminating power lives. We maintain two kinds. IO histograms bucket the IOs in the window by attributes such as transfer size and address region, exposing the dispersion pattern of ransomware — many parallel workers spraying similar-sized read-encrypt-rewrite operations across the volume — versus the concentrated, repetitive patterns of databases or log writers. Entropy histograms bucket the per-buffer entropy values, revealing, for example, a bimodal mix of normal writes and a dense mass near 8 bits/byte that an average would smear away. Histograms are also computationally friendly: they are simple counting operations, linear in the number of bytes or IOs, so they fit comfortably in a real-time streaming pipeline.

All of these features are computed out of band in the cloud-based Flink pipeline, so the production data path pays nothing for them. A few implementation notes: a read and its corresponding rewrite can straddle window boundaries, and in the first cut we only pair them within the same window — simpler at the cost of slight undercounting. An optional refinement compares the entropy of the originally read data against the data written back; a sharp jump is a strong encryption signal. Window size is the main tuning knob: larger windows approximate the underlying distributions better and improve accuracy, while smaller windows shrink detection latency. For training, each window vector is labeled 1 for ransomware and 0 for benign.

The entropy problem

Entropy deserves its own discussion, because it is simultaneously the most prominent feature and the easiest one to over-trust. The theory is clean: well-encrypted data approaches 8 bits per byte of Shannon entropy with a near-uniform byte distribution, while plaintext sits far lower — English text around 2.0 to 4.5 bits/byte, executable code around 4.0 to 6.5. The complication is compression. Lossless-compressed data (ZIP, GZIP, PNG) lands at roughly 7.0 to 7.99 bits/byte and lossy formats (JPEG, MP3) at 7.90 to 7.999 — territory that overlaps heavily with encrypted output. A backup job writing compressed archives can look statistically similar to ransomware writing ciphertext. This overlap between compressed and encrypted data is well documented in the literature: the limits of entropy as a standalone discriminator are analyzed in detail by Reategui, Pletka, and Diamantopoulos in On the Generalizability of Machine Learning-based Ransomware Detection in Block Storage (arXiv:2412.21084, 2024), whose feature work motivates pairing entropy with distribution-shape and access-pattern features rather than relying on a single entropy threshold.

The table below summarizes typical Shannon entropy ranges across common data types. Notice how the compressed categories crowd into the same 7.0 to 8.0 band as encrypted data: a detector that thresholds on entropy alone cannot tell a GZIP backup or a folder of JPEGs from ciphertext, and will flood operators with false positives on perfectly ordinary workloads.

Our empirical study confirmed the concern, with an extra wrinkle: AES-128/AES-256-encrypted test buffers in our experiments did not always show the textbook near-8.0 entropy, while a .tar.gz archive measured 7.9983 bits/byte. Shannon entropy alone is simply not sufficient to separate encrypted buffers from benign compressed ones. That is why the feature design blends Shannon entropy with chi-square and cumulative-sum statistics (with NIST SP-style tests under study), leans on entropy histograms rather than averages, and pairs entropy with behavioral features like the rewrite rate — a compressed backup does not first read low-entropy blocks and immediately overwrite them with high-entropy data, but ransomware does. We initially computed entropy at the IO Interceptor level, but that constrains how much compute can be spent; moving the computation off the host and into the cloud engine allows proper compute sizing for the heavier statistical tests.

The training pipeline

Training uses the feature machinery described above and adds labels. The benign half of the corpus comes from production and lab IO traces (sysbench and fio workloads, with blktrace-based capture during development) and is tagged 0.

Along with ransomware traces gathered in a sandbox environment, we also draw on a published trace corpus: Alibaba’s Disk IO for Ransomware Detection Open Dataset on the Tianchi platform. This is the block-level dataset released alongside Alibaba’s cloud ransomware-protection work, and it fits the pipeline almost exactly — it consists of disk-level IO records of ransomware activity captured in a cloud block-storage setting, which is the same vantage point our IO Interceptor occupies. Each ransomware trace carries the per-IO attributes our features need (timestamps, read/write type, addresses, sizes), so it can be replayed through the same sliding-window feature extraction we run on live traffic. The windows derived from these traces are tagged 1 (ransomware). Reusing a real-world cloud-storage trace set, rather than a handful of synthetic samples, gives the model exposure to genuine ransomware IO dynamics — the bursty, parallelized read-encrypt-rewrite behavior across many addresses that distinguishes an attack from ordinary load — and crucially lets us bootstrap the ransomware class without running live malware against our own volumes.

For the model to generalize, the combined corpus must eventually span multiple file systems and volume states, database and compression workloads as hard benign negatives, and copy-on-write and encrypted-device traces. Feature vectors deliberately carry no volume identifiers. The resulting CSV files are uploaded to an IBM Cloud Object Storage (COS) bucket as they hit a size threshold.

The training phase starts after an initial data-gathering period of roughly two weeks. Because an AutoAI experiment consumes a single dataset as its source, the accumulated CSV objects are first merged into one combined file — using pandas or IBM Data Refinery — and uploaded back to COS, which comfortably handles this since multipart uploads support objects up to 10 TB.

An AutoAI experiment then runs against the combined dataset. AutoAI automates the parts of the workflow that are otherwise tedious and error-prone: data pre-processing, model selection, feature engineering, hyperparameter optimization, and ensembling. Our preference among candidate algorithms is XGBoost, which is widely accepted on both performance and accuracy. In practice, the diversity of the training data matters more for accuracy than the choice among tree-based algorithms — and XGBoost’s exhaustive parameter list is precisely why we lean on AutoML for hyperparameter search rather than tuning by hand.

The key capability we depend on is incremental learning, native in watsonx 1.1.x. It matters here for two reasons. First, resource efficiency: training happens on small, newly arrived data batches rather than the full historical corpus, which keeps training fast and cheap. Second, adaptability: as benign workload patterns and ransomware behaviors evolve, the model absorbs the new patterns without a from-scratch retrain. Under the hood, AutoAI pairs the selected algorithm with a complementary BatchedTreeEnsemble method to enable batch-wise training; completing training over the entire dataset involves saving the winning pipeline as a notebook and continuing incremental fits from there. Once training completes, the model is promoted to a deployment space and deployed in batch mode with appropriately sized compute, and the ransomware module is flagged to begin inference.

The inference pipeline

With a trained model deployed, the live path comes alive. Inside the cloud detection module, a Kafka + Flink data pipeline does the stream processing. The module maintains a separate buffer (queue) per volume; the Flink job computes the sliding-window feature vector for each window — exactly the same computation used to build the training data — and prunes processed entries from the buffer. Because the pipeline runs continuously on the streamed traces, a verdict for a window is available within seconds of the IOs being issued, which is what keeps the detection window short.

The Flink job emits these vectors in batches, and the module makes a REST call to the classification model — the XGBoost model hosted on IBM watsonx, deployed in batch mode. Each window is classified as ransomware (1) or benign (0). To suppress noise from a single anomalous window, an alert is raised only when at least 70% of the predictions in a batch say ransomware; the alert then feeds the broader alerting and recovery workflow. This batch-level voting is a cheap post-processing step that trades a little detection latency for a large reduction in false positives — essential, since even a low per-window false-positive rate multiplies into an unmanageable alert volume across a large fleet of volumes.

Every computed feature vector is also appended to a file and uploaded to COS once it reaches a size threshold. This historical record exists so the model can be retrained later if drift is detected or new anomalies need to be learned, with lifecycle policies archiving or trimming aged files.

Operational lessons and design choices

A few practical findings from development are worth sharing.

Training is throughput-driven; inference is latency-sensitive. These pull deployment in different directions. Training runs occasionally on accumulated data — one early AutoAI run took four hours to process 1.2 GB, prompting library-level tuning — while inference runs continuously and sits in the alerting path. Inference scales horizontally: in a self-managed Kubernetes deployment, a stateless service endpoint behind a load balancer scales with load; on watsonx SaaS, scaling is achieved through multiple deployments or larger hardware specs, fronted by an application load balancer and monitored via IBM Cloud Monitoring or watsonx.governance.

Model freshness needs active management. Drift is detected via drops in precision score. We also plan a shadow model continuously trained on recent IO data; significant divergence between the shadow model’s predictions and the deployed model’s indicates either a shift in benign application behavior or an attack — both worth knowing about.

Capture overhead is real. Recording and analyzing every IO has a cost, and sampling techniques are the planned mitigation. In-band alternatives that compute entropy inline in the kernel IO path are known to tax both latency and throughput measurably, with the entropy calculation as the dominant cost — which reinforces the appeal of the out-of-band design, where the host only captures and ships traces while feature computation happens in the cloud engine.

Storage format. We store the feature datasets in Parquet rather than CSV. Parquet’s columnar layout and compression yield roughly 30 to 100% storage savings over CSV depending on the feature set, and its typed schema and column pruning make the data cheaper to load for training and analysis. Datasets can live either in a dedicated managed bucket or in the customer’s own COS account; the latter keeps data in customer hands at the cost of credential-management overhead.

What’s next

Block IO traces lack behavioral context, so a sufficiently novel attack pattern could slip through as a false negative. Phase 2 addresses this with an ensemble: eBPF-based tracking of file system activity — opens, creates, deletes, reads, writes — to catch the activity spikes characteristic of ransomware, paired with unsupervised learning (Isolation Forest) on watsonx alongside the supervised block-IO model. Phase 3 extends the system to Windows, adds backup-scan-based detection for complex scenarios, and integrates with the broader ecosystem — CyberSense, IBM AIX, IBM i, OpenShift AI, and IBM Defender among them.

The broader takeaway: the block IO stream already carries the raw material for ransomware detection. A thin host-side interceptor, a cloud-based sliding-window feature pipeline, and an incrementally trained gradient-boosted model turn that stream into a near-real-time early-warning system — catching an attack while it is still small, and without adding meaningful load to the hosts you are trying to protect.