SLLCP가 VLM 안전 판단을 보정해 자율주행 충돌 미탐지율을 크게 낮췄다
미국 미시간대 연구진이 공개한 프리프린트에서, 냉동 비전언어모델(VLM)의 안전 예측을 보정하는 SLLCP(Split Label-Localized Conformal Prediction)를 제안했다. 이 방법은 CARLA 시뮬레이터의 미등장 시나리오 15,321개 프레임에서 Qwen 백본은 충돌 유발 궤적의 89.6%, Cosmos 백본은 88.4%를 탐지해 원시 VLM의 4.6%와 39.1%를 크게 끌어올렸다. VLM을 미세 조정하지 않고 계획 모듈과 독립적으로 쓸 수 있는 사후 보정 층이라는 점이 특징이다.
주행 궤적을 실행하기 전에 VLM 기반 안전 모니터가 충돌을 일으킬 궤적을 놓칠 확률을 사용자 지정 수준 이하로 보장하면서 불필요한 경보를 줄일 수 있는가?
자율주행 차량이 계획한 궤적을 실행하기 전에 충돌 여부를 판단해야 하는데, 보행자나 주변 차량이 자차 움직임에 반응하고 가려진 행위자가 있어 기존 기하 충돌 검사는 예측 모델 품질에 의존한다. VLM은 상식 추론에 강하지만 원시 안전 확률이 과신할 수 있어 안전 필수 응용에 직접 쓰기 어렵다.
기존 안전 모니터는 인접 행위자 궤적을 예측한 뒤 기하 충돌 검사를 수행하거나, VLM의 원시 safe/unsafe 판단을 사용했다. 전자는 장면 지식과 예측 정확도에 의존하고 후자는 확률적 보증이 없다. 일반 분할 순응 예측은 클래스 불균형을 반영하지 못하고, 입력 조건부 보장은 유한 데이터에서 불가능해 SLCP도 주변적 커버리지에 그친다.
SLLCP는 교정 데이터를 local 집합과 CP 집합으로 나눈 뒤, 동결된 VLM의 마지막 RMSNorm 이후 은닉 상태에서 로지스틱 프로브를 학습해 비적합 점수를 만들고, UMAP 2차원 투영 공간에서 라플라스 커널 가중치로 라벨별 국소 분위수를 계산한다. 이후 별도 교정 집합에서 라벨별 잔차 분위수를 더해 예측 집합을 구성하고, unsafe가 포함되면 경보를 울린다. 이 구조는 라벨 조건부 유한표본 분포 무관 커버리지를 목표로 하며 VLM 가중치를 바꾸지 않는다.
평가 집합은 Fail2Drive의 HardBrake, PedestriansOnRoad, Wall 시나리오에서 얻은 15,321개 프레임이며, 이 중 2,489개가 unsafe, 12,832개가 safe다. CARLA 시뮬레이터에서 SimLingo 플래너가 생성한 궤적을 2.5초 동안 PID 추적해 충돌 여부를 판정했다. α=0.1일 때 SLLCP는 Qwen 백본에서 unsafe 커버리지 89.6%, 경보 FPR 51.8%를 기록했고, Cosmos 백본에서는 88.4%와 44.5%를 기록했다. 원시 VLM 이진 판단은 Qwen 4.6%, Cosmos 39.1%에 그쳤고, α 인지형 원시 규칙도 Qwen 20.1%, Cosmos 78.1%였다. 로지스틱 프로브를 뺀 ablation은 커버리지가 Qwen 99.2%, Cosmos 99.6%까지 오르지만 FPR이 95.6%, 99.0%로 비실용적이었다. UMAP 투영을 빼면 커버리지가 84.3%, 82.4%로 하락했고, 라벨 조건부 교정을 빼면 70.9%, 66.7%로 떨어졌다. SLLCP의 커버리지는 명목 90%의 90% 베타-이항 신뢰구간 안에 들었다.
저자들은 결과가 UMAP 투영과 커널 하이퍼파라미터에 민감하고, 에피소드 전체 프레임을 모아 교정해 데이터가 상관되어 교환 가능성 가정이 깨진다고 밝혔다. 시나리오 이동도 경험적 성능에 영향을 준다. 또한 경보 FPR이 44.5%에서 51.8%로 높고, 양쪽 라벨이 동시에 포함되는 모호성도 33.6%에서 37.6%에 달해 실제 차량에 바로 적용하기 어렵다. 평가는 CARLA 시뮬레이터에 한정됐고 실물 주행 검증은 논문에 보고되지 않았다. 코드 공개 여부도 논문에 보고되지 않았다.
자율주행 스택 개발사가 계획 모듈과 독립적인 사후 안전 모니터로 활용할 수 있다. 동결 VLM을 쓰므로 추가 미세 조정 없이 Qwen, Cosmos 같은 개방 가중치 모델에 붙일 수 있다. 다만 FPR과 모호성이 높아 당장 실차에 쓰기는 어렵고, 실제 주행 데이터에서 순차적 상관관계를 다루는 온라인 보정, 하이퍼파라미터 강건성, 경보 지연 개선이 선행돼야 한다. 실도로 적용 시점은 1~3년 뒤로 보인다.
논문 전문
Localized Conformal Safety Monitoring with Vision-Language Models for Autonomous Driving
CC BY 4.0 라이선스로 공개된 논문입니다. 출처를 밝혀 전재하며, 원문은 arXiv:2610.02765(PDF)에서 볼 수 있습니다.
Abstract
Monitoring planned driving trajectories requires accurately estimating the collision likelihood with actors whose motion is itself impacted by the ego motion. Existing classical approaches are often limited by the quality of their forecasting model. Vision-language models (VLMs) have shown promise in reasoning about the consequences of high-level actions, yet their approximate predictions are unsuitable for safety-critical applications such as autonomous driving. Conformal prediction (CP) has emerged as a data-driven framework for quantifying the uncertainty of black-box model predictions. We propose Split Label-Localized Conformal Prediction (SLLCP), a post-hoc calibration layer over frozen VLMs that transforms their unreliable predictions into probabilistically calibrated safety prediction sets. We consider how the ability to estimate safety can depend on the observed driving scene and introduce a localized procedure that upweights relevant past experience when calculating uncertainty thresholds. We provide label-conditional finite-sample distribution-free coverage under exchangeability. Evaluated over 15k CARLA trajectories from unseen scenarios, SLLCP correctly flags 89.6\% of collision-causing trajectories with a Qwen backbone and 88.4\% with a Cosmos backbone, while the base VLMs only flagged 4.6\% and 39.1\% of the collision-causing trajectories, respectively. These results indicate that local, label-conditional calibration can reduce missed unsafe trajectories.
Index Terms:
Conformal prediction, safety monitoring, uncertainty quantification, vision-language models, autonomous driving
I Introduction and Related Work
Before self-driving cars execute planned trajectories, they must assess whether the intended motion can be safely executed, considering the surrounding scene. This is nontrivial: pedestrians and other vehicles move in response to the ego vehicle. For example, a trajectory passing a pedestrian near the curb may be collision-free if the pedestrian waits, but unsafe if they step into the vehicle’s path. Further, the information available at planning time might miss occluded actors and not provide sufficient signal to enable accurate forecasting of the scene’s evolution. Even seemingly robust driving policies can demonstrate degraded performance when operating in unfamiliar scenarios — Fail2Drive reports substantial drops in closed-loop success under controlled scenario shifts [5].
We study the problem of designing a safety monitor that examines each candidate trajectory before execution, independently of the quality of the underlying planning module. That monitor should flag trajectories that would collide within a fixed time horizon, while avoiding unnecessary alarms hindering task progress. In general, a safety monitor might need to reason about the candidate motion and the scene jointly, while considering uncertainty in the true unknown outcome. While vision-language driving systems, such as SimLingo, incorporate scene understanding with action-taking [17], we aim to construct a monitor that can be applied post-hoc over any base planner. We seek a theoretically-informed way to probabilistically limit the number of missed unsafe trajectories.
Safety monitors. Driving pipelines often forecast the motion of nearby actors before checking a planned trajectory for collision [16]. Model-based safety checks can account for uncertain actor motion and dynamics [9], but their assurances depend on assumed scene knowledge. When an actor is missed entirely or its response to the ego vehicle is incorrectly predicted, using a collision check may fail. We thus investigate the complementary problem of assessing candidate trajectories using front-facing images, so that a vision-language model (VLM) can assess hard to model risks. A VLM’s raw safe/unsafe prediction provides no assurances of low-rate missed collisions, since VLM spatial reasoning can be unreliable [18]. Hence, we aim to augment VLMs’ common-sense reasoning with a post-hoc calibration layer providing probabilistic guarantees on their prediction performance.
0.5\,s intervals.Conformal prediction in robotics. Conformal prediction (CP) transforms approximate predictions into prediction sets with finite-sample guarantees under exchangeability [19, 1]. In robotics, CP has calibrated: forecasts in dynamic scenes for social navigation [9, 3]; how dynamics uncertainty varies over a system’s state–action space [11]; action uncertainty for Lie-algebraic systems [12]; and motion uncertainty across contact modes [13].
In this extended abstract, we propose a monitor calibrating a base VLM safety prediction in a scene-adaptive manner. Our key contributions are:
- We develop Split Label-Localized Conformal Prediction (SLLCP), a post-hoc method over a planner-agnostic generated trajectory image and a frozen VLM backbone, that provides probabilistic guarantees on a low probability of missing unsafe trajectories.
- We empirically validate the tradeoff between missed unsafe trajectories and false alarms on edge-case CARLA scenarios, demonstrating significant improvements in the calibration of two open-weight VLMs, using the same hyperparameter set.
II Problem Statement
Let x_{t} denote the state of an autonomous driving agent at time t\in\mathbb{N}, whose planner generates candidate H-step-long trajectories \tau:=\{x_{t},x_{t+1},\ldots,x_{t+H-1}\}. We consider the problem of provably monitoring the safety of said finite-horizon trajectories given a black-box planner of unknown quality. We assume that executing \tau without replanning can produce one of two, initially unknown, outcomes Y: either a collision occurs within H steps (Y=\textsc{unsafe}), or it does not (Y=\textsc{safe}). While conventional collision-based monitors could be implemented by projecting \tau into the environment observed so far and performing geometric checks, the validity of their predictions is dependent on an accurate forecast of the motion of other dynamic entities in the scene, such as pedestrians and other vehicles. However, these forecasts are notoriously difficult to estimate accurately, even more so given that the ego motion can influence the behaviours of nearby actors. In this work, we consider vision-language models (VLMs) with fixed parameters as the base of our safety monitor, due to their common-sense reasoning capabilities. However, VLM-estimated safety likelihood \hat{p}(Y\mid X) is not directly suitable for safety-critical applications, since it might be overconfident. Our VLM-based monitor receives as input X, consisting of a front-facing RGB camera image of the scene overlaid with the proposed trajectory, as well as a prompt describing the safety monitoring task.
Problem. Given a user-specified acceptable failure rate \upalpha\in(0,1), we seek a set-valued monitor m:\mathcal{X}\to 2^{\{\textsc{safe},\textsc{unsafe}\}} whose probability of missing an unsafe trajectory is at most \upalpha. Formally, we require
\mathbb{P}(\textsc{unsafe}\notin\widehat{C}(X)\mid Y=\textsc{unsafe})\leq\upalpha,and further want to minimize the false positive rate (FPR)
\operatorname{FPR}=\mathbb{P}(\textsc{unsafe}\in\widehat{C}(X)\mid Y=\textsc{safe}),since high FPR implies unnecessary alarms which can hinder the agent’s task performance. To make this problem tractable, we make the standard assumption in the conformal prediction literature of having access to a held-out calibration set of labeled examples that are iid with the test data, i.e.:
Assumption 1
We have VLM inputs-outcome pairs D_{\mathrm{cal}}=\{(X_{i},Y_{i})\}_{i=1}^{n} that are iid with the test pair (X_{n+1},Y_{n+1}).
Following literature, D_{\mathrm{cal}} is uncorrupted.
III Background: Conformal Prediction
We briefly introduce relevant conformal prediction (CP) concepts below and refer to [19, 1] for deeper insight. Let X\in\mathcal{X} denote the available information at the time of prediction and Y\in\mathcal{Y}=\{\textsc{safe},\textsc{unsafe}\} the unknown true label. Given a fixed classifier, the estimated unsafe probability is \widehat{Y}(x)=\hat{p}(\textsc{unsafe}\mid x). The nonconformity score function r:[0,1]\times\mathcal{Y}\to\mathbb{R} measures disagreement between a model prediction and the true label, with higher values indicating higher prediction uncertainty. For binary classification, a conventional choice is r(\widehat{Y}(x),y)=1-\hat{p}(y\mid x).
Split conformal prediction (SplitCP). SplitCP is a computationally efficient CP variant that uses an exchangeable calibration set of labeled examples (Ass. 1) to provide probabilistic guarantees on test-time prediction uncertainty. Given D_{\mathrm{cal}}, we can evaluate the nonconformity score on each example to obtain a list of scores \{R_{i}\}_{i=1}^{n}, where R_{i}=r(\widehat{Y}(X_{i}),Y_{i}). Then the k=\lceil(n+1)(1-\upalpha)\rceilth smallest score in \{R_{i}\}_{i=1}^{n}\cup\{\infty\}, serves as a probabilistic threshold \widehat{q} for the test-time nonconformity score r(\widehat{Y}(X_{n+1}),Y_{n+1}). Using exchangeability [19], the following prediction set
\widehat{C}(x)=\{y\in\mathcal{Y}:r(\widehat{Y}(x),y)\leq\widehat{q}\},is guaranteed to provide marginal coverage, i.e., satisfy \mathbb{P}\{Y\in\widehat{C}(X)\}\geq 1-\upalpha on average.
Label-conditional prediction. Marginal coverage alone does not account for imbalanced datasets. Since the guarantees are based on average uncertainty, underrepresented labels in D_{\mathrm{cal}} that present higher predictive uncertainty can be severely undercovered. For instance, D_{\mathrm{cal}} may contain many more safe than unsafe examples, and unsafe might be more difficult for the VLM to estimate. To address this issue, Label-conditional CP instead calibrates each class label separately [19]. Let I_{y}=\{i:Y_{i}=y\} be the calibration indices for class y, n_{y}=|I_{y}| the number of calibration examples y, and let \widehat{q}_{y} be the \lceil(n_{y}+1)(1-\upalpha)\rceilth smallest score in \{R_{i}:i\in I_{y}\}\cup\{\infty\}. Replacing the SplitCP label-shared threshold \widehat{q} in Eq. (3) by the per-label threshold \widehat{q}_{y} gives rise to the prediction set
\displaystyle\widehat{C}(x) \\ \displaystyle=\{y\in\mathcal{Y}:r(\widehat{Y}(x),y)\leq\widehat{q}_{y}\},which satisfies label-conditional coverage, i.e., implies Eq. (1). This guarantee holds for finite-sized D_{\mathrm{cal}} in a distribution-free manner, i.e., without making any assumptions on the underlying score distribution. The prediction region \widehat{C} may now contain safe, unsafe, both labels, or the empty set. If \widehat{C} includes unsafe, the monitor should raise a safety alarm.
Split localized conformal prediction (SLCP). While each label-dependent threshold \widehat{q}_{y} is input-independent, prediction difficulty may vary across inputs x. For example, some driving scenes may be simpler to classify as unsafe than others, depending on the size, color, and proximity of nearby obstacles. Hence, we desire a threshold that can adapt to the model uncertainty at each input. Unfortunately, input-conditional coverage \mathbb{P}(Y\in\widehat{C}(X)\mid X=x)\geq 1-\upalpha has been shown to be impossible with finite data, without making additional distributional assumptions [8]. SLCP aims to approximate input-conditional coverage by fitting a local score quantile \widehat{q}_{\mathrm{loc}}(x) which up-weights calibration examples with inputs “near” the test-time input x. The key idea is that nearby inputs might share similar prediction difficulty, and thus their scores should be considered as more informative than the scores of faraway calibration examples. Since on its own \widehat{q}_{\mathrm{loc}}(x) does not provide any finite-sample guarantees, SLCP then adds a held-out correction \widehat{q}_{\mathrm{corr}} to obtain a final composed threshold \widehat{q}_{\mathrm{SLCP}}(x) [6, 7]. For exchangeability to hold, D_{\mathrm{cal}} must be initially split into two disjoint sets D_{\mathrm{cal}}^{\mathrm{local}} and D_{\mathrm{cal}}^{\mathrm{CP}} used for local fitting and held-out correction, respectively. Let I_{\mathrm{local}} and I_{\mathrm{CP}} index D_{\mathrm{cal}}^{\mathrm{local}} and D_{\mathrm{cal}}^{\mathrm{CP}}, respectively, and let the local distance between a calibration input X_{i} and a test input x be d_{i}(x)=d(\phi(X_{i}),\phi(x)) where \phi is a feature mapping. A kernel \kappa and bandwidth h>0 define the local weights
w_{i}(x)=\frac{\kappa(d_{i}(x)/h)}{\sum_{j\in I_{\mathrm{local}}}\kappa(d_{j}(x)/h)},and \widehat{q}_{\mathrm{loc}}(x) is the weighted (1-\upalpha) quantile of D_{\mathrm{cal}}^{\mathrm{local}} scores. The held-out correction \widehat{q}_{\mathrm{corr}} is the SplitCP quantile of \{R_{j}-\widehat{q}_{\mathrm{loc}}(X_{j}):j\in I_{\mathrm{CP}}\}, giving the SLCP composite threshold of
\widehat{q}_{\mathrm{SLCP}}(x)=\widehat{q}_{\mathrm{loc}}(x)+\widehat{q}_{\mathrm{corr}}.Despite possibly better empirical performance, SLCP only provides marginal coverage due to \widehat{q}_{\mathrm{corr}}’s construction [7].
IV Method: SLLCP
We propose Split Label-Localized Conformal Prediction (SLLCP), which builds on SLCP by combining it with label-conditional conformal prediction. Instead of using the raw VLM nonconformity score, we learn a nonconformity score to better inform the local quantile fit. Figure 2 summarizes our method. Following SLCP, we split D_{\mathrm{cal}}=D_{\mathrm{cal}}^{\mathrm{local}}\sqcup D_{\mathrm{cal}}^{\mathrm{CP}}, using D_{\mathrm{cal}}^{\mathrm{local}} to both learn r and fit \widehat{q}_{\mathrm{loc}}, while D_{\mathrm{cal}}^{\mathrm{CP}} now informs per-label conformal corrections \widehat{q}_{\mathrm{corr},y} for each label y\in\mathcal{Y}.
D_{\mathrm{cal}}^{\mathrm{local}}, we fit a logistic probe to obtain safety scores and a UMAP projection for localization, then define per-label local score quantiles. The disjoint D_{\mathrm{cal}}^{\mathrm{CP}} provides per-label corrections from score residuals. Online: given a test-time image–prompt pair, we combine its local quantiles with the fitted corrections to construct \widehat{C}(x). The monitor raises an alarm when \textsc{unsafe}\in\widehat{C}(x).Probe score and local representation with D_{\mathrm{cal}}^{\mathrm{local}}. Let \mathbf{e}_{\mathrm{VLM}}(x) be the VLM’s hidden state before the Boolean answer token (safe, unsafe) and after the frozen language model’s final RMSNorm. A raw VLM-based score could be computed from the VLM’s Boolean-token probabilities. However, we have empirically observed that this score may not expose all safety-relevant information contained in the VLM hidden state. Therefore, we fit a linear logistic probe \widehat{Y}(x)=\sigma(\mathbf{w}^{\top}\mathbf{e}_{\mathrm{VLM}}(x)+b) to \mathbf{e}_{\mathrm{VLM}}(x) using the ground-truth labels in D_{\mathrm{cal}}^{\mathrm{local}}. This simple classifier provides an alternative way to estimate the unsafe probability \widehat{Y}(x), which we then use instead of Boolean token probabilities to compute the standard binary nonconformity score of Sec. III. This is cheap and does not require updating any VLM weights. For localization, since the raw VLM embedding is very high-dimensional, we first fit a two-dimensional UMAP [14] projection g to \{\mathbf{e}_{\mathrm{VLM}}(X_{i}):i\in I_{\mathrm{local}}\}, yielding a lower-dimensional latent representation of the input \mathbf{z}(x)=g(\mathbf{e}_{\mathrm{VLM}}(x)). This \mathbf{z}(x) is then used as the feature mapping \phi(x) in Eq. (5), leading to the local distance d(u,v)=\|u-v\|_{2} for projected vectors u:=\mathbf{z}(X_{i}),v:=\mathbf{z}(x). Both the classifier and UMAP fits are fixed independently of D_{\mathrm{cal}}^{\mathrm{CP}}.
Label-conditional local quantiles and correction. For each label y\in\mathcal{Y}, let I_{y}^{\mathrm{local}}=\{i\in I_{\mathrm{local}}:Y_{i}=y\}. We obtain the label-conditional thresholds \widehat{q}_{\mathrm{loc},y}(x) by computing the weighted (1-\upalpha) quantile of the scores R_{i}=r(\widehat{Y}(X_{i}),y) for i\in I_{y}^{\mathrm{local}}, using Eq. (5). Given correction examples j\in I_{\mathrm{CP}} with Y_{j}=y, their residual quantile gives \widehat{q}_{\mathrm{corr},y} similarly to SLCP. The \widehat{C} then is
\widehat{C}(x)=\{y\in\mathcal{Y}:r(\widehat{Y}(x),y)\leq\widehat{q}_{\mathrm{loc},y}(x)+\widehat{q}_{\mathrm{corr},y}\}.Our safety monitor raises an alarm when \textsc{unsafe}\in\widehat{C}(x), to match the objective in Sec. II. Assuming within-label exchangeability, the proposed method provides label-conditional marginal coverage. While this assumption might not hold in our experiments, we demonstrate that our theory-driven method still provides adequate empirical coverage in practice.
V Experimental Results
We evaluate if the proposed safety monitor can detect collision-causing trajectories in the CARLA simulator [4]. We report unsafe-label coverage, the monitor’s false-positive rate on safe trajectories, and the fraction of produced prediction sets containing both labels. The latter indicates “ambiguity” in the monitor’s decision, and could be used to trigger a different recovery policy than unsafe-alarms. We use \upalpha=0.1 for all experiments. Coverage is a random variable that, under exchangeability, follows a Beta-binomial distribution given a fixed calibration set and a finite test set [10]. We report if the observed empirical coverage falls within the 90\% confidence interval of the nominal coverage 1-\upalpha, which would indicate if the applied procedure mitigates the nonexchangeable nature of the data.
We compare SLLCP with several relevant baselines: two uncalibrated raw-VLM rules and four component ablations. Raw VLM (binary) emits one safe/unsafe decision: the higher-scoring candidate label. Raw VLM (0.1/0.9) instead uses a \upalpha-aware scoring and a two-label prediction set. This would be the natural monitor if the raw VLM score were well-calibrated. The ablations remove, one at a time: locality (global, input-independent label-conditional CP); the linear probe (using original VLM score); the UMAP projection (raw embedding distances); the per-label correction (use label-agnostic uncertainty threshold). This analysis aims to isolate the impact of our choices of score, latent space representation, and label-aware correction.
We collect trajectories and front-facing images using the SimLingo planner [17]. A trajectory is deemed unsafe if tracking it for 2.5\,s with a PID controller causes a collision with a vehicle, pedestrian, or other object. In total, we obtain 40,455 frames from 90 episodes across 30 underlying scenario IDs in three Fail2Drive families [5]: HardBrake, PedestriansOnRoad, and Wall (seen in Figure 2). As base VLMs, we evaluated Qwen3-VL-8B-Instruct [2] and Cosmos-Reason2-8B [15]. We relied on open-weight VLMs so that we could access the hidden embeddings needed for the linear probe and for doing localized CP in embedding-space. Neither VLM was fine-tuned and both received the same fixed binary safety prompt and trajectory image.
We split the entire data into D_{\mathrm{cal}} and an evaluation set. All seeds of each scenario ID stay on the same side of this split. This prevents leakage of information from the evaluation set into D_{\mathrm{cal}} — each test scenario was not seen during the offline calibration phase. For some methods, D_{\mathrm{cal}} is further divided into D_{\mathrm{cal}}^{\mathrm{local}} and D_{\mathrm{cal}}^{\mathrm{CP}}. This results in |D_{\mathrm{cal}}^{\mathrm{local}}|= 19,528 frames (2,615 unsafe), |D_{\mathrm{cal}}^{\mathrm{CP}}|= 5,606 frames (856 unsafe), and an evaluation set with 15,321 frames (2,489 unsafe; 12,832 safe). SLLCP projected VLM embeddings to UMAP-2 (100 neighbors; Euclidean distance), then used a Laplace kernel \kappa(u)=\exp(-u), with per-label bandwidth h_{y}=0.4 times the label-specific median distance in D_{\mathrm{cal}}^{\mathrm{local}}. Hyperparameters were shared across the two VLMs, yet this results in model-specific conformal thresholds. Table I reports the three evaluation metrics over the 15,321 frames in the evaluation set.
- Raw VLM (binary)
- Qwen
- 4.6
- 0.3
- 0.0
- —
- Cosmos
- 39.1
- 1.9
- 0.0
- Raw VLM (0.1/0.9)
- Qwen
- 20.1
- 2.1
- 4.9
- —
- Cosmos
- 78.1
- 37.3
- 43.9
- Ablation w/o locality
- Qwen
- 77.5
- 27.7
- 18.6
- —
- Cosmos
- 80.5
- 30.4
- 21.3
- Ablation w/o logistic probe
- Qwen
- 99.2
- 95.6
- 89.4
- —
- Cosmos
- 99.6
- 99.0
- 89.3
- Ablation w/o projection
- Qwen
- 84.3
- 41.6
- 28.3
- —
- Cosmos
- 82.4
- 37.9
- 26.8
- Ablation w/o label-conditional CP
- Qwen
- 70.9
- 15.4
- 6.1
- —
- Cosmos
- 66.7
- 11.9
- 5.2
- SLLCP (ours)
- Qwen
- 89.6
- 51.8
- 37.6
- —
- Cosmos
- 88.4
- 44.5
- 33.6
Raw VLM (binary) alarms included unsafe on only 4.6\% of Qwen’s and 39.1\% of Cosmos’s unsafe frames, failing to meet the user-set safety requirements. Similarly, the safety-biased raw rule also remained well below the nominal 90\% target. Our approach, SLLCP, increased unsafe-label coverage to 89.6\% and 88.4\% staying within the finite-sample Beta-binomial confidence interval of the user-set requirement. However, the alarm FPRs rose to 51.8\% and 44.5\% and both-label ambiguity to 37.6\% and 33.6\%, respectively. This indicates an efficiency–coverage tradeoff. In safety-critical applications, where consequences can be severe, one may prefer to reduce missed unsafe trajectories at the cost of more false alarms.
The ablations motivate our design decisions. Without the logistic probe, the local method includes nearly all unsafe frames but raises false alarms on at least 95.6\% of safe frames, making it impractical for real-world applications. Without UMAP, unsafe coverage falls to 84.3\% and 82.4\%. Using a label-independent threshold lowers coverage further to 70.9\% and 66.7\%. These comparisons suggest that the score, local representation, and label-specific correction each impact the coverage–false-alarm tradeoff.
Limitations. Results are sensitive to the UMAP projection and kernel hyperparameters. The original dataset includes video frames collected along full-episode SimLingo trajectories. Hence, frames are correlated, breaking exchangeability. The scenario shift from D_{\mathrm{cal}} also impacts empirical performance.
VI Conclusion
On unseen Fail2Drive scenarios in CARLA, SLLCP achieved 89.6\% and 88.4\% unsafe-label coverage for Qwen and Cosmos, improvements of 85.0 and 49.3 percentage points, respectively, over the raw VLM performance. In this extended abstract, we considered trajectory monitoring as a two-label prediction-set problem. We combined a learned score, from frozen VLM representations, with local calibration in a projected embedding space and label-conditional calibration. Compared with the no-probe ablation, our approach reduces the alarm FPR by 43.8 and 54.5 percentage points for Qwen and Cosmos, respectively, while keeping unsafe-label coverage near 90\%. SLLCP provides a planner-agnostic way to balance safety monitoring and false alarms without requiring any VLM fine-tuning. Future work could consider applications to semantic safety violations, such as failing to yield or obey traffic signals. Yet, obtaining reliable ground-truth labels in these settings is nontrivial.