Evaluators

Evaluation happens at four levels — sample, sensor, timestep, and scenario — matching the benchmark protocol. Each evaluator is a streaming accumulator: call update() once per batch, then compute() at the end for the aggregated metrics. See Evaluating your model for a guided tour with code examples.

Every evaluator identifies data by scenario_id (the filesystem path to the scenario directory) and — below the scenario level — timestep_id (an integer timestep number). The path must follow the on-disk layout, e.g. .../test/anomaly/{town}/{anomaly_type}/... or .../test/normal/{town}/...: evaluators read it both to locate label files and to parse the anomaly type and town for per-type breakdowns.

Sample level

Per-pixel (camera) and per-point (LiDAR) scores. Every evaluated sample in the split — from anomalous timesteps, anomaly-free timesteps, and normal scenarios alike — is pooled into a single binary problem, instead of averaging per-timestep scores over anomaly-containing timesteps. This removes the selection bias that lets a detector score a whole class (e.g. traffic lights) high without separating anomalous instances from normal ones. Metrics use bounded-memory streaming histograms; compute() returns global pooled AUROC/AUPR/AP/FPR95 plus a per-anomaly-type breakdown and a per-scenario macro-average diagnostic.

class carlanomaly.PixelEvaluator(sensor, num_workers=32, max_inflight=32, *, n_bins=65536, scenario_bins=4096, score_transform=None, score_range=None)[source]

Bases: _SampleLevelEvaluator

Sample-level metrics over per-pixel scores of one camera, pooled over the split.

Loads ground-truth masks from {scenario_id}/anomaly-{sensor}/{timestep_id:06d}.png (any non-zero pixel is anomalous); a missing mask counts as all-negative.

Parameters:

  • sensor – Camera direction ("front", "left", "right", "rear").

  • num_workers – Worker threads that decode masks and bin scores off the main loop.

  • max_inflight – Maximum timesteps queued at once; bounds retained host and GPU memory.

  • n_bins – Bins for the global and per-type score histograms.

  • scenario_bins – Bins for the per-scenario histograms; must divide n_bins.

  • score_transform – Strictly monotonic map applied to scores before binning; defaults to asinh, which handles unbounded scores without tuning.

  • score_range – (lo, hi) covered by the histogram after score_transform.

Parameters:

  • sensor (str)

  • num_workers (int)

  • max_inflight (int)

  • n_bins (int)

  • scenario_bins (int)

  • score_transform (Optional[ScoreTransform])

  • score_range (Optional[Tuple[float, float]])

update(pixel_scores, scenario_ids, timestep_ids)[source]

Accumulate one batch of per-pixel scores.

Scores stay on their source device (the model’s GPU); each timestep’s mask decode + GPU metric computation is handed to a worker thread. Each timestep’s score slice is cloned so the full batch tensor can be freed immediately, bounding retained GPU memory to max_inflight timesteps.

Parameters:

  • pixel_scores – FloatTensor (B, H, W) of per-pixel anomaly scores.

  • scenario_ids – length-B sequence of scenario path strings (see module docstring).

  • timestep_ids – length-B sequence of integer timestep numbers.

Return type:

None

Parameters:
compute()
Return type:

Dict[str, Any]

reset()
Return type:

None

class carlanomaly.PointEvaluator(num_workers=32, max_inflight=32, *, n_bins=65536, scenario_bins=4096, score_transform=None, score_range=None)[source]

Bases: _SampleLevelEvaluator

Sample-level metrics over per-point LiDAR scores, pooled over the split.

Loads ground-truth labels from {scenario_id}/anomaly-lidar/{timestep_id:06d}.feather (anomaly column) and handles variable-length point clouds. Constructor parameters (num_workers, max_inflight, n_bins, scenario_bins, score_transform, score_range) match PixelEvaluator; this evaluator takes no sensor argument.

Parameters:

  • num_workers (int)

  • max_inflight (int)

  • n_bins (int)

  • scenario_bins (int)

  • score_transform (Optional[ScoreTransform])

  • score_range (Optional[Tuple[float, float]])

update(point_scores, scenario_ids, timestep_ids)[source]

Accumulate one batch of per-point scores.

Parameters:

  • point_scores – length-B sequence of FloatTensor (N_i,) per-point scores.

  • scenario_ids – length-B sequence of scenario path strings (see module docstring).

  • timestep_ids – length-B sequence of integer timestep numbers.

Return type:

None

Parameters:
compute()
Return type:

Dict[str, Any]

reset()
Return type:

None

Sensor level

One score per sensor reading (one sensor at one timestep), evaluated against labels that mark whether the anomaly is visible in that specific sensor.

class carlanomaly.SensorEvaluator(sensor)[source]

Bases: _PerTimestepEvaluator

Sensor-level AUROC: one score per sensor reading, against sensor-specific labels.

A sensor reading (this sensor at one timestep) is positive only if the anomaly is visible in this specific sensor. Labels come from {scenario_id}/anomaly-{sensor}/sensor.feather.

Parameters:

sensor – Any sensor name ("front", "left", "right", "rear", "lidar").

Parameters:

sensor (str)

compute()
Return type:

Dict[str, Any]

max_per_scenario()

Reduce accumulated timestep scores to one max score per scenario.

Return type:

Dict[str, float]

reset()
Return type:

None

to_dataframe()
Return type:

DataFrame

update(scores, scenario_ids, timestep_ids)

Accumulate one batch of scalar per-timestep scores.

Parameters:

  • scores – FloatTensor (B,) or length-B sequence of scalar scores.

  • scenario_ids – length-B sequence of scenario path strings.

  • timestep_ids – length-B sequence of timestep numbers.

Return type:

None

Parameters:

Timestep level

One score per timestep, treating all synchronized sensor readings as one multimodal observation. This is where fused multi-sensor scores are evaluated.

class carlanomaly.TimestepEvaluator[source]

Bases: _PerTimestepEvaluator

Timestep-level AUROC against timestep-level labels.

The timestep label marks whether the scene is anomalous at a given timestep — all synchronized sensor readings treated as one multimodal observation, independent of any single sensor. It can fire when an anomaly is visible to one or more sensors, when the sensors disagree, or for purely semantic anomalies that no sensor localises (e.g. anomalous weather). Labels are stored on disk as {scenario_id}/anomaly-observation.feather. Accepts fused multi-sensor scores; a scenario with 30 timesteps contributes 30 scores.

compute()
Return type:

Dict[str, Any]

max_per_scenario()

Reduce accumulated timestep scores to one max score per scenario.

Return type:

Dict[str, float]

reset()
Return type:

None

to_dataframe()
Return type:

DataFrame

update(scores, scenario_ids, timestep_ids)

Accumulate one batch of scalar per-timestep scores.

Parameters:

  • scores – FloatTensor (B,) or length-B sequence of scalar scores.

  • scenario_ids – length-B sequence of scenario path strings.

  • timestep_ids – length-B sequence of timestep numbers.

Return type:

None

Parameters:

Scenario level

One score per scenario; the label is inferred from the scenario path.

class carlanomaly.ScenarioEvaluator[source]

Bases: object

Scenario-level AUROC: one score per scenario.

The label is inferred from the scenario path (/test/anomaly/ is positive, /test/normal/ is negative). Each update() takes one scalar score for the whole scenario; how that score is produced is up to the caller and need not involve timestep-level evaluation (e.g. a max over timestep scores, or a model that scores whole scenarios directly).

update(score, scenario_id)[source]

Accumulate one scenario’s score; label inferred from path.

Return type:

None

Parameters:
compute()[source]
Return type:

Dict[str, Any]

to_dataframe()[source]
Return type:

DataFrame

reset()[source]
Return type:

None