Delta Encoding With Feather and Pandas

This is a short article on the use of delta encoding in Feather and Pandas. Imagine you have some data you acquired by an embedded system, and you want to store it for later postprocessing. How should you store it?

A Reasonable Sample Data Set

Here’s some sample data, a 10Hz sine wave, amplitude-modulated at 0.02Hz, with noise, sampled at close-to-regular intervals, and quantized as if sampled by a 12-bit analog-to-digital converter.

import os
import json
import gzip
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import datetime as DT
%matplotlib inline

with gzip.open('capture_timeticks_data.json.gz') as f:
    tick_data = json.load(f)

ticks = np.array(tick_data['ticks'])
t = ticks*1e-6
r = np.random.RandomState(seed=123)
xreal = 0.5 + (0.25 + 0.1 * np.sin(2*np.pi*0.02*t)) * np.sin(2*np.pi*10*t) + 0.01*r.randn(len(t))
qrange = 2**12
xq = np.floor(xreal*qrange).astype(np.int16)
x = xq*1.0/qrange

fig,axes = plt.subplots(ncols=2, figsize=(10.8,6.2),
                        width_ratios=[2,1],
                        gridspec_kw=dict(wspace=0.1))

for i,ax in enumerate(axes):
    ax.plot(t,x,'-' if i == 0 else '.-',
            linewidth=0.9)
    xlim = (t.min(), t.max() if i == 0 else t.min()+1)
    ax.set_xlim(xlim)
    ax.set_xlabel('Time (s)')
    ax.set_ylim(0.1,0.9)
    ax.set_yticks(np.arange(0.1,0.91,0.1))
    
isodate = DT.datetime.utcfromtimestamp(np.floor(tick_data['origin']/1000)).isoformat()
fig.suptitle('Sample data, time relative to origin %d (%s)' % 
              (tick_data['origin'],  isodate),
             y=0.92)
fig.savefig('capture_data.png', bbox_inches='tight')

In this data set, both the time and the value are quantized: values xq vary between 0 and 4095, and values ticks are integers measuring elapsed time in microseconds from a reference point (the “origin”).

df = pd.DataFrame(dict(ticks=ticks, values=xq)).set_index('ticks')
df

Hey, great! Let’s save that in CSV format.

df.to_csv('capture_data.csv')
os.path.getsize('capture_data.csv')

2960234

Hmmm… about three megabytes. Surely we can do better?

Compression, for the Win

Let’s compress it:

with open('capture_data.csv') as f:
    csvraw = f.read()
with gzip.open('capture_data.csv.gz', 'wt', encoding='utf8') as f:
    f.write(csvraw)
    
os.path.getsize('capture_data.csv.gz')

947355

Yeah, well, not bad, under a megabyte.

Pandas can do better, using the Apache Feather format used in the Arrow library:

df.to_feather('capture_data.1.arrow', 
              compression='zstd',
              compression_level=15)
os.path.getsize('capture_data.1.arrow')

742338

Wait a minute, you say, you didn’t use the same compression method…

Okay, we’ll use zstd to compress the CSV file too:

!zstd -15 -i capture_data.csv -o capture_data.csv.zstd
!ls -l capture_data.csv*

capture_data.csv     : 30.77%   (  2.82 MiB =>    890 KiB, capture_data.csv.zstd) 
-rw-r--r--@ 1 jmsachs  staff  2960234 Jul 19 22:41 capture_data.csv
-rw-r--r--  1 jmsachs  staff   947355 Jul 19 22:43 capture_data.csv.gz
-rw-r--r--  1 jmsachs  staff   910865 Jul 19 22:41 capture_data.csv.zstd

Zstandard does a little better than gzip on this CSV-format data, but not as good as Pandas in a native-binary format using compression.

Delta Encoding

But we can do better! How? Let’s look at the time-tick data

fig,ax = plt.subplots(figsize=(10,6))
ax.plot(ticks)

[<matplotlib.lines.Line2D at 0x150e4e250>]

There’s 200,000 samples, increasing regularly, with a little glitch after about two minutes. So let’s look at the time between samples:

fig,axes = plt.subplots(figsize=(10,6), nrows=2)

for i,ax in enumerate(axes):
    ax.plot(np.diff(ticks))
    if i == 1:
        ax.set_ylim(0,10000)
    ax.set_xlim(0,len(ticks))
axes[0].set_title('$\\Delta$tick = time between successive ticks');

On average, the time between samples was about 4 milliseconds, with one glitch of a delay of about one second, and a few minor glitches in the 5 – 9 millisecond range. (The tick values were captured by Javascript running in a browser using performance.now() — not bad for a browser running setInterval!)

The thing is, most of these Δtick differences are repeated frequently:

values, counts = np.unique(np.diff(ticks), return_counts=True)
dfdiff = pd.DataFrame(dict(delta=values, counts=counts))
dfdiff.sort_values('counts',ascending=False).iloc[:20]

cumulative_fraction = dfdiff.sort_values('counts',ascending=False)['counts'].cumsum()
cumulative_fraction = pd.Series(cumulative_fraction.values / cumulative_fraction.iloc[-1])
cumulative_fraction.index += 1        # N is one-based, not zero
fig,ax = plt.subplots(figsize=(10,6))
cumulative_fraction.plot(ax=ax)
ax.grid(True)
nx = len(cumulative_fraction)
ax.set_xlim(0,nx)
ax.set_xticks(np.arange(0,nx+1,5))
ax.set_xlabel('N')
ax.set_yticks(np.arange(0,1.01,0.05))
ax.set_title('Fraction of difference values $\\Delta$tick'
            +' covered by most frequent N values (max N=%d)' % nx);

The five most frequent difference values Δtick in this dataset (3500, 4500, 4599, 4601, and 3399 microseconds) make up 40% of all the difference values.

Which suggests that the Δtick values ought to be a much better candidate for compression than the tick values themselves:

def to_delta(df):
    """ Transform a DataFrame to its delta-encoded equivalent
    (assumes integer values)
    """
    df2 = df.reset_index()
    return df2.apply(lambda column: np.diff(column, prepend=0))

df2 = to_delta(df)
df2

Here the first row is the difference between zero and the first input; the remaining differences are small.

Okay, how large is the saved data?

df2.to_feather('capture_data.2.arrow', 
               compression='zstd',
               compression_level=15)
os.path.getsize('capture_data.2.arrow')

489562

Hey, it’s significantly smaller!

In fact, we ought to be able to compute the information-theoretical content of 200102 time-tick datapoints by running an Shannon entropy calculation \( H = - \sum p(x) \log_2 p(x) \), normalized to some number of bits per sample.

If each Δtick value were unique and therefore equally likely, then \( H_0 = - \sum \frac{1}{N} \log_2 \frac{1}{N} = - \log_2 \frac{1}{N} = \log_2 N \).

The actual value of \( H \) is less, because there are duplicates and some of them are more common than others:

def calc_entropy(delta_values):
    N = len(delta_values)
    values, counts = np.unique(delta_values, return_counts=True)
    p = counts/N
    return -np.sum(p * np.log2(p))

H = calc_entropy(df2.ticks)
H0 = np.log2(len(df2.ticks))
print('H=%.3f, H0=%.3f, H/H0=%.3f' % (H,H0,H/H0))

H=5.228, H0=17.610, H/H0=0.297

The ratio of these values \( H/H_0 \) for the same \( N \) gives us a theoretical compression ratio of over 3:1 for the information required to reconstruct the time tick values.

We can do even better if we look at the data types involved:

df2.dtypes

ticks     int64
values    int64
dtype: object

Oh, these are 64-bit data types; what if we converted to 32-bit integers, since we know the differences are smaller?

df2.astype(np.int32).to_feather('capture_data.3.arrow', 
               compression='zstd',
               compression_level=15)
os.path.getsize('capture_data.3.arrow')

455354

And some of this is because the tick resolution is so high. If we captured tick times with 1-millisecond resolution, we do even better because there is less information in the inter-tick intervals.

fig,axes = plt.subplots(figsize=(10,6), nrows=2)

tickqms = (ticks // 1000)

for i,ax in enumerate(axes):
    ax.plot(np.diff(tickqms))
    if i == 1:
        ax.set_ylim(0,10)
    ax.set_xlim(0,len(ticks))
axes[0].set_title('$\\Delta$tick = time between successive ticks (quantized to 1ms)')

Text(0.5, 1.0, '$\\Delta$tick = time between successive ticks (quantized to 1ms)')

df1ms = pd.DataFrame(dict(tickqms=tickqms, values=xq)).set_index('tickqms')
df4 = to_delta(df1ms)
df4.astype(np.int32).to_feather('capture_data.4.arrow', 
               compression='zstd',
               compression_level=15)
os.path.getsize('capture_data.4.arrow')

344546

This last step is a lossy one, but all the preceding steps are lossless, assuming the delta values fit in a 32-bit integer.

Part of the reason I wanted to show this, is when the timestep variation is relatively small (which is what happens if you capture it at low resolution), it significantly reduces the space required. We can see this more clearly if we set the values field to zero, in which case it compresses to near-zero, and most of the information we’re storing is the time ticks — here you can see how much less space the 1ms-quantized time requires:

reference_size = None
for tick_vals, file_part, use_delta, use_32bit in [
      (ticks,   '1z', False, False),  # control: full timestamp, no delta encoding 
      (ticks,   '2z', True,  False),  # control: delta encoding but at its full size
      (ticks,   '3z', True, True),    # 1 microsecond resolution, 32-bit delta-encoding
      (tickqms, '4z', True, True),    # 1 millisecond resolution
      (ticks*0, '0z', True, True),    # All zeros, as a control
      ]: 
    dfz = pd.DataFrame(dict(tick_vals=tick_vals, values=xq*0)).set_index('tick_vals')
    if use_delta:
        dfzout = to_delta(dfz)
    else:
        dfzout = dfz
    if use_32bit:
        dfzout = dfzout.astype(np.int32)
    filename = 'capture_data.%s.arrow' % file_part
    dfzout.to_feather(filename, 
               compression='zstd',
               compression_level=15)
    filesize = os.path.getsize(filename)
    if reference_size is None:
        reference_size = filesize
    print("%s %6d (%6.2f%%)" % (
          filename, filesize, filesize/reference_size*100))

capture_data.1z.arrow 417026 (100.00%)
capture_data.2z.arrow 178882 ( 42.89%)
capture_data.3z.arrow 151106 ( 36.23%)
capture_data.4z.arrow  40290 (  9.66%)
capture_data.0z.arrow   2898 (  0.69%)

The ratio of microsecond resolution, delta-encoded timestamp data (.3z), to raw timestamp data (.1z) is 36%. This is pretty close to the information-theoretical compression of 29.7% we calculated earlier, and it’s probably due to the fact that the raw timestamp data can still be compressed a little bit.

It takes about 2.9K just to encode a DataFrame of this size and data type with all-zeros content. The 1ms-quantized ticks require only a quarter of the size (in compressed form) needed to represent the original microsecond-quantized tick counts.

Anyway, delta-encoding time data is a significant reduction, especially for this sort of almost-regularly sampled timeseries data — and we can handle the lossless decoding and get our data back to its original form.

The only gotcha with all these transformations is that Pandas won’t know how to undo our delta encoding automatically; we have to help:

def from_delta(df2):
    return df2.apply(lambda column: np.cumsum(column)).set_index(df2.columns[0])

# we have to tell it what data type the values field was
df_decoded = from_delta(pd.read_feather('capture_data.3.arrow')).astype(np.int16)    
df_decoded

df_decoded.equals(df)

True

We get back exactly what we had originally: the same timeseries data. The two operations are duals:

• delta encoding: compute successive differences, starting from zero
• delta decoding: compute cumulative sums, starting from zero

These are both extremely fast operations on integers.

Scaling

Delta encoding is lossless with integer types, but can introduce errors with floating-point, so it’s probably not appropriate.

We can, however, represent floating-point values as integers with scale and offset.

A DataFrame like this could have a “raw” integer value representation underneath the hood, that represents its scaled values, with scale and offset or raw_offset for each column, where value = scale * raw + offset or value = scale * (raw + raw_offset) depending on the need.

This works naturally with values digitized by an analog-to-digital converter, where there is a linear relationship between raw count and real-world value. (Well, technically, there are departures from linearity that are captured by integral nonlinearity INL and differential nonlinearity DNL specs, but these tend to be very small for any good ADC, and in practice we often ignore them and treat the ADC as linear.)

A Plea for Native Library Support

I have a plea for the Feather and Pandas library maintainers: please add these features, so that serialized data has appropriate metadata to note that it should be deserialized accordingly.

It would allow significantly-reduced storage space for time-series data that is close to regularly sampled, or is otherwise amenable to delta-encoding.

Wrapup

Today we talked about delta encoding for time-series data, which stores the differences between timestamps (and data, if desired). This is more amenable to compression when the timestamps occur at intervals that are mostly regular, because it tends to yield reoccuring values — whereas a linearly-increasing or nearly-linearly-increasing set of raw timestamps is “invisible” to compression.

The compression is lossless, and is very quick to execute in both directions: taking successive differences to encode, and taking cumulative sums to decode.

We gave an example data set of 200,102 samples which had the storage sizes listed below. (If you’d like to examine the data yourself, I’ve uploaded it to Github as a Gist — have fun!)

Essentially we can reduce the total file space by almost 40% (and the size of the timestamp storage by 64%) if we use delta encoding on the timestamps and store the differences as 32-bit values.

(And use Feather or other well-known binary format for time-series data, rather than compressed CSV when you can. CSV is just ugly. Bleah! But good libraries like Pandas can handle either.)

Timestamps that have coarser granularity (millisecond instead of microsecond) have even higher compression ratios with delta encoding.

Hope you found this useful!

© 2026 Jason M. Sachs, all rights reserved.