In this paper, we investigate the impact of adversarial attacks on identity encoders within a realistic de-identification framework. Our experiments show that the transferability of attacks transfers from an external surrogate model to the system model (e.g., CosFace to ArcFace) allows the adversary to cause identity information to leak in a sufficiently sensitive face recognition system. We present experimental evidence and propose strategies to mitigate this vulnerability. Specifically, we show how fine-tuning on adversarial examples helps to mitigate this effect for distortion-based attacks (i.e., snow, fog, etc.), while a simple low-pass filter can attenuate the effect of adversarial noise without affecting the de-identified images. Our mitigation results in a de-identification system that preserves its functionality while being significantly more robust to adversarial noise. © 2025 IEEE.

Sensor degradation poses a significant challenge in autonomous driving. During heavy rainfall, interference from raindrops can adversely affect the quality of LiDAR point clouds, resulting in, for instance, inaccurate point measurements. This, in turn, can potentially lead to safety concerns if autonomous driving systems are not weather-aware, i.e., if they are unable to discern such changes. In this study, we release a new, large-scale, multi-modal emulated rain dataset, REHEARSE-3D, to promote research advancements in 3D point cloud de-raining. Distinct from the most relevant competitors, our dataset is unique in several respects. First, it is the largest point-wise annotated dataset (9.2 billion annotated points), and second, it is the only one with high-resolution LiDAR data (LiDAR-256) enriched with 4D RADAR point clouds logged in both daytime and nighttime conditions in a controlled weather environment. Furthermore, REHEARSE-3D involves rain-characteristic information, which is of significant value not only for sensor noise modeling but also for analyzing the impact of weather at the point level. Leveraging REHEARSE-3D, we benchmark raindrop detection and removal in fused LiDAR and 4D RADAR point clouds. Our comprehensive study further evaluates the performance of various statistical and deep learning models, where SalsaNext and 3D-OutDet achieve above 94% IoU for raindrop detection. © 2026 by the authors.

In this work, we propose a novel, fast, and memory-efficient unsupervised statistical method, combined with an unsupervised deep learning (DL) model, for de-snowing 3D LiDAR point clouds in a fully unsupervised fashion. The results obtained on the real-scanned Winter Adverse Driving dataSet (WADS) show that our DL model achieves a 6.3% improvement in mIoU over the current state-of-the-art unsupervised DL methods and performs comparable to supervised counterparts, substantially narrowing the performance gap between supervised and unsupervised approaches. In addition to that, our model also outperforms its closest competitor by 12.8% mIoU when tested on our Canadian Adverse Driving Conditions (CADC) dataset annotations. Additionally, our de-snowing algorithm enhances downstream semantic segmentation and object detection tasks without even requiring any modifications to the base segmentation and detection models. The source code, trained models, and the online supplementary information are available at the following URL: https://sporsho.github.io/3DUnOutDet. © 2025 IEEE.

The testing and safety cases of Assisted and Automated Driving functions require considerations for non ideal environmental conditions, such as adverse and extreme weather. In these extreme conditions, perception sensors (e.g. camera, LiDAR, RADAR), which build the situational awareness of the vehicle, might produce noisy and degraded data. Therefore it is key to consider: (i) how to reliably and robustly measure data degradation; (ii) how to evaluate de-noising techniques; (iii) linking perception performance to denoising quality. This paper focuses on de-snowing of LiDAR data, as falling snow is one of the most variable and dangerous conditions encountered while driving - and LiDAR can provide essential 3D information to still enable safe vehicle navigation. Using the WADS dataset, which contains segmented pointclouds including falling and deposited snow points, state-of-the-art de-snowing techniques are compared using an array of adapted pointcloud quality metrics, combined with downstream segmentation (perception) performance after de-snowing evaluated using perception based metrics. The different metrics are able to capture different aspects/effects of the data degradation, and hereby the novel De-Snow Score (DSS) is proposed and applied to have a holistic evaluation of the de-noising techniques considering both data quality and expected perception performance. Based on DSS, the most promising de-noising algorithms are identified. The proposed methodology and De-Snow Score can pave the way for a standardised approach when evaluating perception sensor data degradation and de-noising techniques. © 2001-2012 IEEE.

Recognition of human manipulation actions in real-time is essential for safe and effective human-robot interaction and collaboration. The challenge lies in developing a model that is both lightweight enough for real-time execution and capable of generalization. While some existing methods in the literature can run in real-time, they struggle with temporal scalability, i.e., they fail to adapt to long-duration manipulations effectively. To address this, leveraging the generalizable scene graph representations, we propose a new Factorized Graph Sequence Encoder network that not only runs in real-time but also scales effectively in the temporal dimension, thanks to its factorized encoder architecture. Additionally, we introduce Hand Pooling operation, a simple pooling operation for more focused extraction of the graph-level embeddings. Our model outperforms the previous state-of-the-art real-time approach, achieving a 14.3% and 5.6% improvement in F1-macro score on the KIT Bimanual Action (Bimacs) Dataset and Collaborative Action (CoAx) Dataset, respectively. Moreover, we conduct an extensive ablation study to validate our network design choices. Finally, we compare our model with its architecturally similar RGB-based model on the Bimacs dataset and show the limitations of this model in contrast to ours on such an objectcentric manipulation dataset. Our code and trained models are available at https://github.com/eneserdo/FGSE. ©2025 by IEEE.

Adverse weather conditions such as snow, rain, and fog are natural phenomena that can impair the performance of the perception algorithms in autonomous vehicles. Although LiDARs provide accurate and reliable scans of the surroundings, its output can be substantially degraded by precipitation (e.g., snow particles) leading to an undesired effect on the downstream perception tasks. Several studies have been performed to battle this undesired effect by filtering out precipitation outliers, however, these works have large memory consumption and long execution times which are not desired for onboard applications. To that end, we introduce a novel outlier detector for 3D LiDAR point clouds captured under adverse weather conditions. Our proposed detector 3D-OutDet is based on a novel convolution operation that processes nearest neighbors only, allowing the model to capture the most relevant points. This reduces the number of layers, resulting in a model with a low memory footprint and fast execution time, while producing a competitive performance compared to state-of-the-art models. We conduct extensive experiments on three different datasets (WADS, SnowyKITTI, and SemanticSpray) and show that with a sacrifice of 0.16% mIOU performance, our model reduces the memory consumption by 99.92%, number of operations by 96.87%, and execution time by 82.84% per point cloud on the real-scanned WADS dataset. Our experimental evaluations also showed that the mIOU performance of the downstream semantic segmentation task on WADS can be improved up to 5.08% after applying our proposed outlier detector. We release our source code, supplementary material and videos in https://sporsho.github.io/3DOutDet. Upon clicking the link you will have to option to go to source code, see supplementary information and view videos generated with our 3D-OutDet. © 2024 IEEE.

This work presents a new depth-and semantics-aware conditional generative model, named TITAN-Next, for cross-domain image-to-image translation in a multi-modal setup between LiDAR and camera sensors. The proposed model leverages scene semantics as a mid-level representation and is able to translate raw LiDAR point clouds to RGB-D camera images by solely relying on semantic scene segments. We claim that this is the first framework of its kind and it has practical applications in autonomous vehicles such as providing a fail-safe mechanism and augmenting available data in the target image domain. The proposed model is evaluated on the large-scale and challenging Semantic-KITTI dataset, and experimental findings show that it considerably outperforms the original TITAN-Net and other strong baselines by 23.7% margin in terms of IoU. © 2023 The Author(s). 

An autonomous service robot should be able to interact with its environment safely and robustly without requiring human assistance. Unstructured environments are challenging for robots since the exact prediction of outcomes is not always possible. Even when the robot behaviors are well-designed, the unpredictable nature of the physical robot-object interaction may lead to failures in object manipulation. In this letter, we focus on detecting and classifying both manipulation and post-manipulation phase failures using the same exteroception setup. We cover a diverse set of failure types for primary tabletop manipulation actions. In order to detect these failures, we propose FINO-Net (Inceoglu et al., 2021), a deep multimodal sensor fusion-based classifier network architecture. FINO-Net accurately detects and classifies failures from raw sensory data without any additional information on task description and scene state. In this work, we use our extended FAILURE dataset (Inceoglu et al., 2021) with 99 new multimodal manipulation recordings and annotate them with their corresponding failure types. FINO-Net achieves 0.87 failure detection and 0.80 failure classification F1 scores. Experimental results show that FINO-Net is also appropriate for real-time use. © 2016 IEEE.

Deformable object manipulations, such as those involving textiles, present a significant challenge due to their high dimensionality and complexity. In this paper, we propose a solution for estimating semantic states in cloth manipulation tasks. To this end, we introduce a new, large-scale, fully-annotated RGB image dataset of semantic states featuring a diverse range of human demonstrations of various complex cloth manipulations. This effectively transforms the problem of action recognition into a classification task. We then evaluate the generalizability of our approach by employing domain adaptation techniques to transfer knowledge from human demonstrations to two distinct robotic platforms: Kinova and UR robots. Additionally, we further improve performance by utilizing a semantic state graph learned from human manipulation data. © 2024 by SCITEPRESS – Science and Technology Publications, Lda.

State estimation of deformable objects such as textiles is notoriously difficult due to its extreme high dimensionality and complexity. Lack of data and benchmarks is another challenge impeding progress in robotic cloth manipulation. In this paper, we make a first attempt to solve the problem of semantic state estimation through RGB-D data only in an end-to-end manner with the help of deep neural networks. Since neural networks require large amounts of labeled data, we introduce a novel Mujoco simulator to generate a large-scale fully annotated robotic textile manipulation dataset including bimanual actions. Finally, we provide a set of baseline deep neural networks and benchmark them on the problem of semantic state prediction on our proposed dataset. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.