Streaming large files

ecv.load reads a whole recording into memory. For multi-gigabyte files, ecv.open returns a lazy EventReader that slices the file on demand — for HDF5 it binary-searches the on-disk timestamps, so a slice costs a handful of reads and the file is never fully materialised (rosbag and txt have their own in-place backends).

We demonstrate the API on the small bundled fixture; the same calls work unchanged on a 707-million-event file.

import eventcv as ecv
import numpy as np
from pathlib import Path

# Point PATH at your own recording (.npz / .hdf5 / .bag / .txt / .aedat / .dat).
# Here we locate the small test fixture bundled with the EventCV source.
def _fixture(rel="data/test/example.npz"):
    for base in (Path.cwd(), *Path.cwd().parents):
        if (base / rel).exists():
            return str(base / rel)
    raise FileNotFoundError(rel)

PATH = _fixture()

reader = ecv.open(PATH, dt_ms=20)     # treat the recording as 20 ms frames
print("frames:", reader.n_slices)
print("span (ms):", reader.time_span_ms, "| duration:", reader.duration_ms)

frames: 3
span (ms): (0.0, 49.718) | duration: 49.718

Frame indexing

slice(n) returns the n-th dt_ms frame from the recording start, as an EventStream.

frame0 = reader.slice(0)
print(type(frame0).__name__, "with", len(frame0), "events")

EventStream with 54231 events

Walk every window

windows() is a lazy iterator; it streams a huge file one window at a time.

for i, w in enumerate(reader.windows()):
    print(f"window {i}: {len(w)} events")

window 0: 54231 events
window 1: 37127 events
window 2: 14937 events

Slices as representations

Pass repr= and the reader becomes a representation source: slice, slice_count, and every windows() item come back as the rendered EventFrame instead of a raw stream — so open(path, repr="count").slice(0) is exactly open(path).slice(0).count(). (reader[i] still gives the dense [C, H, W] array for the PyTorch dataset path — see the transforms tutorial.)

# Give the reader a representation and every slice comes back rendered:
frames = ecv.open(PATH, dt_ms=20, repr="count")
frame = frames.slice(0)                       # an EventFrame — no extra .count() call
print(type(frame).__name__, "| kind:", frame.kind, "| shape:", frame.numpy().shape)

# open(repr=…).slice(n) is exactly open().slice(n).count():
manual = ecv.open(PATH, dt_ms=20).slice(0).count()
print("matches manual .count():", np.array_equal(frame.numpy(), manual.numpy()))

EventFrame | kind: count | shape: (1, 480, 640)
matches manual .count(): True

Slice by time or count

Open without dt_ms to take arbitrary windows, relative to the recording start.

r = ecv.open(PATH)
print("first 10 ms:", len(r.slice(t0_ms=0.0, t1_ms=10.0)), "events")
print("events [0, 1000):", len(r.slice_count(0, 1000)))

first 10 ms: 27044 events
events [0, 1000): 1000

Remove hot pixels across the recording

Stuck (“hot”) pixels fire far more than the scene warrants and speckle every frame. Pass hot_pixel_filter=True and open flags them once, globally — it scans the recording up front and drops those pixels from every slice it returns, so the cleanup is identical across the whole file.

This has to be global: running stream.hot_pixel_filter() on each slice re-computes its threshold per window, so at long dt_ms a genuinely hot pixel stops standing out from the busy scene and survives. The pre-scan stays lightweight — it reads only the event coordinates (not timestamps/polarity) and, on a large file, samples windows spread across it rather than every event — so the overhead is small even on multi-gigabyte recordings. hot_pixel_std (default 3.0) sets how aggressive the cut is.

plain    = ecv.open(PATH, dt_ms=20)
filtered = ecv.open(PATH, dt_ms=20, hot_pixel_filter=True)

before = sum(len(w) for w in plain.windows())
after  = sum(len(w) for w in filtered.windows())
print(f"{before} events -> {after} after hot-pixel removal ({before - after} dropped)")

# One global mask cleans every slice, so hot pixels never reappear window to window:
print("frame 0:", len(plain.slice(0)), "->", len(filtered.slice(0)), "events")

106295 events -> 96323 after hot-pixel removal (9972 dropped)
frame 0: 54231 -> 50251 events

On a real gigabyte file

The identical API scales to huge recordings without loading them:

reader = ecv.open("brisbane_707M.hdf5", dt_ms=30)  # opens instantly — nothing read yet
reader.n_slices                                     # frame count from the timestamp range
frame = reader.slice(5000)                          # ~32 ms: one binary search + read

# Strip stuck pixels across the whole recording; the only up-front read is a light,
# coordinates-only sampled pre-scan, so it stays cheap even at this scale.
clean = ecv.open("brisbane_707M.hdf5", dt_ms=30, hot_pixel_filter=True)

Only the events in the requested window are decoded, so memory stays flat regardless of file size. To build a training loop or render a video, combine windows() with a representation and export_png — see the transforms tutorial and the open reference.