Representations¶

A representation turns a sparse EventStream into a dense tensor (most are [C, H, W], ready for NumPy / PyTorch) or, for pset(), a sparse point array. Most are reachable three ways:

stream.voxel(bins=5)                       # method
ecv.voxel(stream, bins=5)                  # free function (OpenCV-style)
stream.flatten("voxel")                    # by name (defaults; see "By name" below)

The method and free-function forms take the representation’s parameters directly. The name form (a string like "voxel") selects a representation for flatten(), view(), open(..., repr=...), and reader.with_repr(...) — but only reader.with_repr(name, **opts) also forwards the representation’s parameters; the other three use its defaults. A few representations don’t fit all three forms; each is flagged in its section below.

Summary¶

Name	Method	Channels	Dtype	What it encodes
`polarity`	(none — default view, or by name)	2	`uint64`/`uint8`	Per-pixel event count, split into positive/negative planes. The default view when no representation is requested.
`binary`	(none — `flatten(binary=True)`, or by name)	1	`uint8`	Pixel fired at all (`1`) or not (`0`), both polarities merged.
`count`	`count()`	1	`uint64`/`uint8`	Total events per pixel, both polarities summed.
`voxel`	`voxel()`	`bins`	`float32`	Signed, time-interpolated polarity across `bins` temporal bins over the last `window_ms`.
`tsurf`	`tsurf()`	2	`float32`	Time surface: `exp(-age/tau_ms)` of the latest event per pixel/polarity.
`atsurf`	`atsurf()`	2	`float32`	Averaged time surface: mean of `exp(-age/tau_ms)` over all events per pixel/polarity.
`tencode`	`tencode()`	3	`uint8`	Latest polarity + normalized age within `window_ms`, as an RGB-like image.
`redblue`	`redblue()`	3	`uint8`	White-background red/blue counts, independently normalized per polarity; the stronger normalized polarity wins.
`countmask`	`countmask()`	3	`uint8`	Positive/negative event counts in red/blue, jointly normalized by a percentile of the non-zero counts, plus a binary activity mask in green. Timestamps unused.
`mcts`	`mcts()`	`2·N` (10)	`float32`	Multi-channel time surface: 5 log-spaced windows up to `max_window_ms` (or the `N` windows in `windows_ms`), per polarity.
`flow`	`optical_flow()`	2	`float32`	Dense Lucas-Kanade optical flow `(flow_x, flow_y)` on the time surface, pixels/ms.
`pset`	`pset()`	— (`[N, 4]`)	`float32`	Sparse, normalized `(x, y, t, p)` point array — not a dense frame.
`labels`	`connected_components()`	1	`uint64`	Connected-component labels of an existing frame (post-processing, not stream-level).

Two rows are special cases rather than string-selectable representations: pset returns a sparse point array (not a dense frame), and labels is computed from an existing frame — so neither is a valid repr= / flatten / view name. See their sections below.

The time-span parameters (window_ms, tau_ms, max_window_ms) default to 30.0 on a plain stream, but an unset span follows the window the events came from: on a live capture it defaults to stream(dt_ms=…), and on a file it defaults to the slice duration — open(path, dt_ms=50).slice(0).mcts() covers the full 50 ms rather than clipping to 30. An explicit span always wins. Count-based windows (max_events) have no fixed duration, so the 30 ms default stands there.

The dtype column lists uint64/uint8 for the two representations with a normalize option (count and polarity): raw uint64 counts, or uint8 rescaled so the busiest pixel maps to downsample=k bins k×k sensor pixels into one cell (x // k, y // k) as it counts — the same integers as stream.resize(width // k, height // k) followed by the histogram, in one pass; on a reader (open(..., repr="polarity", downsample=k) or with_repr("polarity", downsample=k)) the RAW decoder feeds the accumulator directly, so no event stream is built per window. 255 when normalize=True. The default differs by representation: polarity normalizes by default (True), while count returns raw counts by default (False) so the frame preserves exact event totals. The rest have no normalization option. view always auto-contrasts for display regardless of the frame’s dtype.

Polarity¶

stream.view()                  # implicit — polarity is the default when nothing is stored/named
stream.flatten("polarity")
ecv.flatten(stream, "polarity", normalize=False)

Two channels, [positive, negative]: raw event counts per pixel per polarity (uint64), or uint8 rescaled to the busiest pixel when normalize=True (the default). This is the frame rendered whenever no representation is requested or stored on the stream — there is no dedicated .polarity() method, only the name form and the implicit default. (A reader opened with open(repr=...) renders that representation on slice() directly instead of falling back to polarity.)

Binary¶

stream.flatten(binary=True)    # dedicated flag
stream.flatten("binary")       # equivalent, by name

Single channel, uint8, 1 where a pixel fired at all (either polarity) and 0 elsewhere. No parameters, and — like polarity — no dedicated method; only reachable via flatten(binary=True) or the "binary" name.

Count¶

stream.count()                    # raw uint64 counts (default)
ecv.count(stream, normalize=True) # uint8, busiest pixel → 255

normalize (bool, default False) — when True, rescale to uint8 (busiest pixel → 255) instead of the default raw uint64 counts.

Single channel: total events per pixel, both polarities summed. The plainest “how much happened here” frame — unlike polarity, it collapses polarity into one intensity map. The raw uint64 default keeps exact totals (e.g. count().numpy().sum() equals the event count), which is what you want for analysis; pass normalize=True for a display-scaled image.

Voxel grid¶

stream.voxel(bins=9, window_ms=30.0)
ecv.voxel(stream, bins=5, window_ms=50.0)

bins (int, default 9) — number of temporal bins; must be at least 1.
window_ms (float, default 30.0) — trailing time window before the stream’s last event.

bins channels, float32, signed. Each event contributes +1/-1 (by polarity), linearly interpolated between its two nearest bins by how far into window_ms it falls — events outside the window are dropped.

Time surface (`tsurf`)¶

stream.tsurf(tau_ms=30.0)
ecv.tsurf(stream, tau_ms=10.0)

tau_ms (float, default 30.0) — exponential decay constant; must be finite and positive.

Two channels, [positive, negative], float32. Each pixel/polarity holds exp(-age_ms / tau_ms) for its most recent event only (0 if that polarity never fired there) — a snapshot of “how recently did this fire.”

Averaged time surface (`atsurf`)¶

stream.atsurf(tau_ms=30.0)
ecv.atsurf(stream, tau_ms=30.0)

tau_ms (float, default 30.0) — exponential decay constant; must be finite and positive.

Two channels, [positive, negative], float32. Same exponential response as tsurf, but averaged over every event that hit the pixel rather than keeping only the latest — recurring activity reads brighter than a single stale hit.

Related in spirit to HATS (Sironi et al., CVPR 2018), but not that paper’s scheme: HATS partitions the sensor into cells, keeps a local memory per cell and polarity, and builds a local neighbourhood surface for every event before averaging into per-cell histograms. atsurf is a per-pixel mean in a single pass, with a different output shape.

Tencode¶

stream.tencode(window_ms=30.0)
ecv.tencode(stream, window_ms=30.0)

window_ms (float, default 30.0) — trailing time window; must be finite and positive.

Three channels, [positive, age, negative], uint8. Per pixel: whichever polarity fired most recently sets its channel to 255 (the other stays 0), and the age channel holds that event’s age rescaled to 0-255 across window_ms. Reads like an RGB image (red/green-style polarity planes plus a brightness-coded age plane).

Count mask¶

stream.countmask(pct=99.0)
ecv.countmask(stream, pct=99.0, white_frame=False)

pct (float, default 99.0) — normalization percentile; must be between 0 and 100.
white_frame (bool, default False) — invert to a white background.

Three channels, [positive, activity, negative], uint8 — the event-accumulation image of the GEPT paper (Sec. 3.2, Eq. 2). Red holds the per-pixel count of positive events and blue the count of negative ones; green is a binary mask, 255 wherever an event of either polarity landed and 0 elsewhere. Timestamps are not used at all, which is what separates it from tencode (latest polarity plus event age) and from count (one plane of raw totals).

Both count planes are clipped and rescaled by a single scale: the pct-th percentile of the non-zero counts of the two planes pooled together. Pooling matters — a per-channel percentile would let a quiet polarity saturate against a busy one, so the joint scale is what keeps red and blue comparable. Counts above the percentile clamp to full scale.

Expect a dark red/blue image over a bright green mask; on typical data the green plane averages several times brighter than the other two. white_frame=True inverts the whole image, background included, and exists for parity with the reference renderer — descriptor models trained on these frames expect the black-background default.

Red/blue counts¶

stream.redblue(pct=99.0)
ecv.redblue(stream, pct=99.0, white_frame=True, device="cpu")

Three RGB channels [red, green, blue], shape [3, H, W], uint8. Positive activity shades white toward red and negative activity shades it toward blue. Each polarity is clipped and normalized independently by the pct-th percentile of its nonzero counts, using linear interpolation. At each pixel only the stronger normalized polarity contributes; ties favor positive. Green is the inverse winning intensity, not a binary activity mask.

pct defaults to 99.0 and must be finite and between 0 and 100, inclusive. white_frame defaults to True and is accepted for compatibility: either value produces the same white background. Empty streams produce white frames. Timestamps are unused. device follows the session default; GPU counting uses the existing polarity kernel and normalization stays on the CPU. Colors pass through unchanged when displayed or exported.

MCTS (multi-channel time surface)¶

stream.mcts(max_window_ms=30.0)
ecv.mcts(stream, max_window_ms=16.0)
stream.mcts(windows_ms=[1, 5, 20])   # explicit windows: 6 channels

max_window_ms (float, default: the slice/capture window, else 30.0) — largest of the 5 log-spaced windows; must be at least 1.0.
windows_ms (list[float]) — explicit windows instead of the log-spaced ladder: N finite, positive spans, used in the order given, producing 2·N channels. Mutually exclusive with max_window_ms (passing both is a ValueError); duplicates are allowed (and yield duplicate channel names). Takes the usual unit siblings (windows_s/windows_us/windows_ns).

2·N channels, float32 (ten by default): one channel per window for each of negative/positive (channel names carry the window, like negative_1.000ms … positive_30.000ms). By default the windows are 5 log-spaced spans from 1 ms up to max_window_ms. Each channel holds max(1 - age/window) over events within that window — a multi-timescale generalization of tsurf.

Optical flow¶

stream.optical_flow(window=3)
ecv.optical_flow(stream, window=5)
stream.view("flow")             # by name; also valid for open(repr="flow")

window (int, default 3) — half-width of the least-squares neighbourhood (a (2·window+1)² patch); must be at least 1.

Two channels, [flow_x, flow_y], float32, in pixels/ms. Dense Lucas-Kanade on the time surface (the Surface of Active Events): fits one velocity per pixel by least squares, falling back to normal flow along the gradient in the aperture-problem case, and leaving flat, event-free windows at zero. Flow is emitted only at pixels that saw an event, and assumes the stream is sorted ascending by time.

Sparse point set¶

points = stream.pset()
ecv.pset(stream).numpy()        # [N, 4] float32 array: columns (x, y, t, p)

No parameters. Unlike the frame representations above, pset returns an EventPointSet, not an EventFrame: an [N, 4] array with x/y normalized to [0, 1] by sensor size, t normalized to [0, 1] over the stream’s span, and p as +1/-1. Because it’s sparse rather than a dense [C, H, W] tensor, it isn’t a valid choice for open(repr=...) / reader.with_repr(...) (those require a dense frame per slice).

Connected-component labels¶

frame = stream.count()
labels = frame.connected_components(connectivity=8)

connectivity (int, default 8) — 4 or 8-connectivity for the flood fill.

Single channel, uint64. This is a post-processing step on an existing EventFrame (typically a binary or count image), not a representation generated directly from a stream: any non-zero pixel (across all channels) is foreground, and each maximal connected blob gets a distinct label 1..k, with background 0.

By name¶

A representation can be picked by string anywhere a name is accepted: flatten(), view(), open(..., repr=...), and reader.with_repr(...). The name is one of "polarity", "binary", "count", "countmask", "redblue", "voxel", "tsurf", "atsurf", "tencode", "mcts", or "flow".

flatten, view, and open(repr=...) render the representation with its default parameters — they don’t accept a representation’s own parameters (bins, tau_ms, window_ms, max_window_ms, windows_ms, window, pct, white_frame), so stream.flatten("voxel", bins=5) is a TypeError. The one exception is normalize, which is an argument of flatten/view themselves, so stream.flatten("count", normalize=True) does work (and each representation keeps its own normalize default when the argument is omitted — False for count, True for polarity). To override the other parameters by name, use reader.with_repr(name, **opts), which returns a new dataset reader:

# defaults, by name:
stream.flatten("voxel")                                  # == stream.voxel()  (bins=9)
reader = ecv.open("huge.hdf5", dt_ms=30, repr="mcts")    # mcts, max_window_ms follows dt (30)

# parameters by name — only with_repr forwards them:
reader = ecv.open("huge.hdf5", dt_ms=30).with_repr("voxel", bins=5)
array = reader[0]                                        # [5, H, W] voxel for frame 0
frame = reader.slice(7)                                  # EventFrame — voxel(bins=5) applied
reader = ecv.open("huge.hdf5", dt_ms=30).with_repr("countmask", pct=95.0)
reader = ecv.open("huge.hdf5", dt_ms=30).with_repr("mcts", windows_ms=[1, 5, 20])

# a repr-less reader yields a stream; name the representation on the slice:
frame = ecv.open("huge.hdf5", dt_ms=30).slice(7).flatten("mcts")   # → EventFrame

"pset" and "labels" are not valid names here — pset is sparse (see above), and labels is generated from a frame, not a stream.

Running on a GPU¶

Every representation runs on the CPU by default, and those implementations are the reference: the GPU kernels are checked against them, not the other way round. device="gpu" moves one call, eventcv.set_device("gpu") moves the session, and EVENTCV_DEVICE=gpu moves a whole process without touching any code.

frame = stream.voxel(bins=5, device="gpu")

ecv.set_device("gpu")        # everything from here on, including reader slices
print(ecv.get_device())      # "gpu"
print(ecv.gpu_available())   # False on a build or a machine without one

The backend is wgpu, so the same shaders run on Metal (macOS), Vulkan (Linux) and DX12 (Windows) — there is no CUDA path to install and no separate Apple path to maintain. Asking for a GPU that is not there raises rather than quietly falling back, because “my GPU is not being used” should not be something you discover by timing a benchmark.

Whether it is worth it¶

Usually not, and the numbers say so. Measured on an RTX 2080 against a 12-core CPU, release build, 8 M events from a 346×260 recording:

kernel	CPU	GPU
`count`	10.5 ms	19.1 ms	CPU wins
`countmask`	16.7 ms	20.3 ms	CPU wins
`voxel(bins=5)`	20.8 ms	28.8 ms	CPU wins
`tsurf`	34.2 ms	29.8 ms	about even
`atsurf`	83.9 ms	29.2 ms	2.9× faster

The pattern is not about the GPU being slow. These kernels are scatter-adds that the CPU already does at memory speed, so the cost is dominated by getting the events across the bus — and only atsurf, which evaluates an exponential per event, has enough arithmetic to pay for the trip. The place the GPU wins clearly is the simulator, where every sub-step is a full pass of real arithmetic over every pixel; see Simulation.

So: leave it on the CPU unless you have measured otherwise on your own data. cargo bench --features gpu -- representations_gpu runs the comparison above on your machine.

What changes, numerically¶

Nothing you would notice, and the tests pin exactly how much:

count, polarity, countmask, redblue and tsurf are bit-identical to the CPU. They accumulate integers, and integer addition commutes exactly where float addition does not.
voxel and atsurf accumulate in Q16.16 fixed point, which is what makes them independent of the order the GPU happened to run in. They agree with the CPU to about 1e-4 on a cell — well below float32’s own resolution for those values — and are bit-reproducible run to run.
A representation with no kernel (tencode, mcts, binary, flow) simply runs on the CPU when you ask for the GPU. A pipeline that sets device="gpu" once should not have to know which steps were ported.