hh.sePublications
Change search
Link to record
Permanent link

Direct link
Bouguelia, Mohamed-RafikORCID iD iconorcid.org/0000-0002-2859-6155
Publications (10 of 31) Show all publications
Vettoruzzo, A., Bouguelia, M.-R., Vanschoren, J., Rögnvaldsson, T. & Santosh, K. (2024). Advances and Challenges in Meta-Learning: A Technical Review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(7), 4763-4779
Open this publication in new window or tab >>Advances and Challenges in Meta-Learning: A Technical Review
Show others...
2024 (English)In: IEEE Transactions on Pattern Analysis and Machine Intelligence, ISSN 0162-8828, E-ISSN 1939-3539, Vol. 46, no 7, p. 4763-4779Article, review/survey (Refereed) Published
Abstract [en]

Meta-learning empowers learning systems with the ability to acquire knowledge from multiple tasks, enabling faster adaptation and generalization to new tasks. This review provides a comprehensive technical overview of meta-learning, emphasizing its importance in real-world applications where data may be scarce or expensive to obtain. The paper covers the state-of-the-art meta-learning approaches and explores the relationship between meta-learning and multi-task learning, transfer learning, domain adaptation and generalization, selfsupervised learning, personalized federated learning, and continual learning. By highlighting the synergies between these topics and the field of meta-learning, the paper demonstrates how advancements in one area can benefit the field as a whole, while avoiding unnecessary duplication of efforts. Additionally, the paper delves into advanced meta-learning topics such as learning from complex multi-modal task distributions, unsupervised metalearning, learning to efficiently adapt to data distribution shifts, and continual meta-learning. Lastly, the paper highlights open problems and challenges for future research in the field. By synthesizing the latest research developments, this paper provides a thorough understanding of meta-learning and its potential impact on various machine learning applications. We believe that this technical overview will contribute to the advancement of meta-learning and its practical implications in addressing realworld problems.

Place, publisher, year, edition, pages
Piscataway, NJ: IEEE, 2024
Keywords
Adaptation models, Data models, deep neural networks, few-shot learning, Meta-learning, Metalearning, representation learning, Surveys, Task analysis, Training, transfer learning, Transfer learning
National Category
Robotics and automation
Identifiers
urn:nbn:se:hh:diva-52730 (URN)10.1109/TPAMI.2024.3357847 (DOI)2-s2.0-85183973598 (Scopus ID)
Funder
Knowledge Foundation
Available from: 2024-02-23 Created: 2024-02-23 Last updated: 2025-02-09Bibliographically approved
Vettoruzzo, A., Joaquin, V., Bouguelia, M.-R. & Rögnvaldsson, T. (2024). Learning to Learn without Forgetting using Attention. In: : . Paper presented at 3rd Conference on Lifelong Learning Agents (CoLLAs), Pisa, Italy, July 29 - 1 August, 2024 (pp. 1-16).
Open this publication in new window or tab >>Learning to Learn without Forgetting using Attention
2024 (English)Conference paper, Published paper (Refereed)
Abstract [en]

Continual learning (CL) refers to the ability to continually learn over time by accommodating new knowledge while retaining previously learned experience. While this concept is inherent in human learning, current machine learning methods are highly prone to overwrite previously learned patterns and thus forget past experience. Instead, model parameters should be updated selectively and carefully, avoiding unnecessary forgetting while optimally leveraging previously learned patterns to accelerate future learning. Since hand-crafting effective update mechanisms is difficult, we propose meta-learning a transformer-based optimizer to enhance CL. This meta-learned optimizer uses attention to learn the complex relationships between model parameters across a stream of tasks, and is designed to generate effective weight updates for the current task while preventing catastrophic forgetting on previously encountered tasks. Evaluations on benchmark datasets like SplitMNIST, RotatedMNIST, and SplitCIFAR-100 affirm the efficacy of the proposed approach in terms of both forward and backward transfer, even on small sets of labeled data, highlighting the advantages of integrating a meta-learned optimizer within the continual learning framework. 

Keywords
Continual learning, Meta-learning, Few-shot learning, Catastrophic forgetting
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-55120 (URN)
Conference
3rd Conference on Lifelong Learning Agents (CoLLAs), Pisa, Italy, July 29 - 1 August, 2024
Available from: 2024-12-19 Created: 2024-12-19 Last updated: 2025-01-07Bibliographically approved
Vettoruzzo, A., Bouguelia, M.-R. & Rögnvaldsson, T. (2024). Meta-learning for efficient unsupervised domain adaptation. Neurocomputing, 574, Article ID 127264.
Open this publication in new window or tab >>Meta-learning for efficient unsupervised domain adaptation
2024 (English)In: Neurocomputing, ISSN 0925-2312, E-ISSN 1872-8286, Vol. 574, article id 127264Article in journal (Refereed) Published
Abstract [en]

The standard machine learning assumption that training and test data are drawn from the same probability distribution does not hold in many real-world applications due to the inability to reproduce testing conditions at training time. Existing unsupervised domain adaption (UDA) methods address this problem by learning a domain-invariant feature space that performs well on available source domain(s) (labeled training data) and the specific target domain (unlabeled test data). In contrast, instead of simply adapting to domains, this paper aims for an approach that learns to adapt effectively to new unlabeled domains. To do so, we leverage meta-learning to optimize a neural network such that an unlabeled adaptation of its parameters to any domain would yield a good generalization on this latter. The experimental evaluation shows that the proposed approach outperforms standard approaches even when a small amount of unlabeled test data is used for adaptation, demonstrating the benefit of meta-learning prior knowledge from various domains to solve UDA problems.

Place, publisher, year, edition, pages
Amsterdam: Elsevier, 2024
Keywords
Domain adaptation, Meta-learning, Unsupervised learning, Distribution shift
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-52450 (URN)10.1016/j.neucom.2024.127264 (DOI)001170864800001 ()2-s2.0-85184141702 (Scopus ID)
Funder
Knowledge Foundation
Available from: 2024-01-22 Created: 2024-01-22 Last updated: 2025-01-07Bibliographically approved
Vettoruzzo, A., Bouguelia, M.-R. & Rögnvaldsson, T. (2024). Multimodal meta-learning through meta-learned task representations. Neural Computing & Applications, 36(15), 8519-8529
Open this publication in new window or tab >>Multimodal meta-learning through meta-learned task representations
2024 (English)In: Neural Computing & Applications, ISSN 0941-0643, E-ISSN 1433-3058, Vol. 36, no 15, p. 8519-8529Article in journal (Refereed) Published
Abstract [en]

Few-shot meta-learning involves training a model on multiple tasks to enable it to efficiently adapt to new, previously unseen tasks with only a limited number of samples. However, current meta-learning methods assume that all tasks are closely related and belong to a common domain, whereas in practice, tasks can be highly diverse and originate from multiple domains, resulting in a multimodal task distribution. This poses a challenge for existing methods as they struggle to learn a shared representation that can be easily adapted to all tasks within the distribution. To address this challenge, we propose a meta-learning framework that can handle multimodal task distributions by conditioning the model on the current task, resulting in a faster adaptation. Our proposed method learns to encode each task and generate task embeddings that modulate the model’s activations. The resulting modulated model becomes specialized for the current task and leads to more effective adaptation. Our framework is designed to work in a realistic setting where the mode from which a task is sampled is unknown. Nonetheless, we also explore the possibility of incorporating auxiliary information, such as the task-mode-label, to further enhance the performance of our method if such information is available. We evaluate our proposed framework on various few-shot regression and image classification tasks, demonstrating its superiority over other state-of-the-art meta-learning methods. The results highlight the benefits of learning to embed task-specific information in the model to guide the adaptation when tasks are sampled from a multimodal distribution. © The Author(s) 2024.

Place, publisher, year, edition, pages
London: Springer, 2024
Keywords
Meta-learning, Few-shot learning, Transfer learning, Task representation, Multimodal distribution
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-52924 (URN)10.1007/s00521-024-09540-1 (DOI)001167510900006 ()2-s2.0-85187241643 (Scopus ID)
Funder
Halmstad UniversityKnowledge Foundation
Available from: 2024-03-21 Created: 2024-03-21 Last updated: 2025-01-07Bibliographically approved
Vettoruzzo, A., Bouguelia, M.-R. & Rögnvaldsson, T. (2024). Personalized Federated Learning with Contextual Modulation and Meta-Learning. In: Shashi Shekhar; Vagelis Papalexakis; Jing Gao; Zhe Jiang; Matteo Riondato (Ed.), Proceedings of the 2024 SIAM International Conference on Data Mining (SDM): . Paper presented at SIAM International Conference on Data Mining (SDM), Houston, USA, April 18-20, 2024 (pp. 842-850). Philadelphia, PA: Society for Industrial and Applied Mathematics
Open this publication in new window or tab >>Personalized Federated Learning with Contextual Modulation and Meta-Learning
2024 (English)In: Proceedings of the 2024 SIAM International Conference on Data Mining (SDM) / [ed] Shashi Shekhar; Vagelis Papalexakis; Jing Gao; Zhe Jiang; Matteo Riondato, Philadelphia, PA: Society for Industrial and Applied Mathematics, 2024, p. 842-850Conference paper, Published paper (Refereed)
Abstract [en]

Federated learning has emerged as a promising approach for training machine learning models on decentralized data sources while preserving data privacy. However, challenges such as communication bottlenecks, heterogeneity of client devices, and non-i.i.d. data distribution pose significant obstacles to achieving optimal model performance. We propose a novel framework that combines federated learning with meta-learning techniques to enhance both efficiency and generalization capabilities. Our approach introduces a federated modulator that learns contextual information from data batches and uses this knowledge to generate modulation parameters. These parameters dynamically adjust the activations of a base model, which operates using a MAML-based approach for model personalization. Experimental results across diverse datasets highlight the improvements in convergence speed and model performance compared to existing federated learning approaches. These findings highlight the potential of incorporating contextual information and meta-learning techniques into federated learning, paving the way for advancements in distributed machine learning paradigms. Copyright © 2024 by SIAM.

Place, publisher, year, edition, pages
Philadelphia, PA: Society for Industrial and Applied Mathematics, 2024
Series
Proceedings of the ... SIAM International Conference on Data Mining, ISSN 2167-0102, E-ISSN 2167-0099
Keywords
Personalized federated learning, Meta-learning, Federated learning, Context learning
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-53426 (URN)10.1137/1.9781611978032.96 (DOI)2-s2.0-85193511974 (Scopus ID)978-1-61197-803-2 (ISBN)
Conference
SIAM International Conference on Data Mining (SDM), Houston, USA, April 18-20, 2024
Funder
Knowledge Foundation
Available from: 2024-05-28 Created: 2024-05-28 Last updated: 2025-01-07Bibliographically approved
Vettoruzzo, A., Bouguelia, M.-R. & Rögnvaldsson, T. (2023). Meta-Learning from Multimodal Task Distributions Using Multiple Sets of Meta-Parameters. In: 2023 International Joint Conference on Neural Networks (IJCNN): . Paper presented at International Joint Conference on Neural Networks (IJCNN 2023), Gold Coast, Australia, 18-23 June, 2023 (pp. 1-8). Piscataway, NJ: IEEE
Open this publication in new window or tab >>Meta-Learning from Multimodal Task Distributions Using Multiple Sets of Meta-Parameters
2023 (English)In: 2023 International Joint Conference on Neural Networks (IJCNN), Piscataway, NJ: IEEE, 2023, p. 1-8Conference paper, Published paper (Refereed)
Abstract [en]

Meta-learning or learning to learn involves training a model on various learning tasks in a way that allows it to quickly learn new tasks from the same distribution using only a small amount of training data (i.e., few-shot learning). Current meta-learning methods implicitly assume that the distribution over tasks is unimodal and consists of tasks belonging to a common domain, which significantly reduces the variety of task distributions they can handle. However, in real-world applications, tasks are often very diverse and come from multiple different domains, making it challenging to meta-learn common knowledge shared across the entire task distribution. In this paper, we propose a method for meta-learning from a multimodal task distribution. The proposed method learns multiple sets of meta-parameters (acting as different initializations of a neural network model) and uses a task encoder to select the best initialization to fine-tune for a new task. More specifically, with a few training examples from a task sampled from an unknown mode, the proposed method predicts which set of meta-parameters (i.e., model’s initialization) would lead to a fast adaptation and a good post-adaptation performance on that task. We evaluate the proposed method on a diverse set of few-shot regression and image classification tasks. The results demonstrate the superiority of the proposed method compared to other state of-the-art meta-learning methods and the benefit of learning multiple model initializations when tasks are sampled from a multimodal task distribution. © 2023 IEEE.

Place, publisher, year, edition, pages
Piscataway, NJ: IEEE, 2023
Keywords
Meta-Learning, Few-Shot Learning, Transfer Learning, Task Representation, Multimodal Distribution
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-51352 (URN)10.1109/IJCNN54540.2023.10191944 (DOI)001046198707013 ()2-s2.0-85169561819 (Scopus ID)978-1-6654-8867-9 (ISBN)
Conference
International Joint Conference on Neural Networks (IJCNN 2023), Gold Coast, Australia, 18-23 June, 2023
Available from: 2023-08-10 Created: 2023-08-10 Last updated: 2025-01-07Bibliographically approved
Taghiyarrenani, Z., Nowaczyk, S., Pashami, S. & Bouguelia, M.-R. (2023). Multi-Domain Adaptation for Regression under Conditional Distribution Shift. Expert systems with applications, 224, Article ID 119907.
Open this publication in new window or tab >>Multi-Domain Adaptation for Regression under Conditional Distribution Shift
2023 (English)In: Expert systems with applications, ISSN 0957-4174, E-ISSN 1873-6793, Vol. 224, article id 119907Article in journal (Refereed) Published
Abstract [en]

Domain adaptation (DA) methods facilitate cross-domain learning by minimizing the marginal or conditional distribution shift between domains. However, the conditional distribution shift is not well addressed by existing DA techniques for the cross-domain regression learning task. In this paper, we propose Multi-Domain Adaptation for Regression under Conditional shift (DARC) method. DARC constructs a shared feature space such that linear regression on top of that space generalizes to all domains. In other words, DARC aligns different domains and makes explicit the task-related information encoded in the values of the dependent variable. It is achieved using a novel Pairwise Similarity Preserver (PSP) loss function. PSP incentivizes the differences between the outcomes of any two samples, regardless of their domain(s), to match the distance between these samples in the constructed space.

We perform experiments in both two-domain and multi-domain settings. The two-domain setting is helpful, especially when one domain contains few available labeled samples and can benefit from adaptation to a domain with many labeled samples. The multi-domain setting allows several domains, each with limited data, to be adapted collectively; thus, multiple domains compensate for each other’s lack of data. The results from all the experiments conducted both on synthetic and real-world datasets confirm the effectiveness of DARC. © 2023 The Authors

Place, publisher, year, edition, pages
Oxford: Elsevier, 2023
Keywords
Regression, Multi-Domain Adaptation, Conditional Shift, Concept Shift, Neural Networks, Siamese neural networks
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-47894 (URN)10.1016/j.eswa.2023.119907 (DOI)000966508000001 ()2-s2.0-85151474329 (Scopus ID)
Funder
VinnovaKnowledge Foundation
Available from: 2022-08-17 Created: 2022-08-17 Last updated: 2024-01-31Bibliographically approved
Nilsson, F., Bouguelia, M.-R. & Rögnvaldsson, T. (2023). Practical Joint Human-Machine Exploration of Industrial Time Series Using the Matrix Profile. Data mining and knowledge discovery, 37, 1-38
Open this publication in new window or tab >>Practical Joint Human-Machine Exploration of Industrial Time Series Using the Matrix Profile
2023 (English)In: Data mining and knowledge discovery, ISSN 1384-5810, E-ISSN 1573-756X, Vol. 37, p. 1-38Article in journal (Refereed) Published
Abstract [en]

Technological advancements and widespread adaptation of new technology in industry have made industrial time series data more available than ever before. With this development grows the need for versatile methods for mining industrial time series data. This paper introduces a practical approach for joint human-machine exploration of industrial time series data using the Matrix Profile (MP), and presents some challenges involved. The approach is demonstrated on three real-life industrial data sets to show how it enables the user to quickly extract semantic information, detect cycles, find deviating patterns, and gain a deeper understanding of the time series. A benchmark test is also presented on ECG (electrocardiogram) data, showing that the approach works well in comparison to previously suggested methods for extracting relevant time series motifs. © 2022, The Author(s).

Place, publisher, year, edition, pages
New York, NY: Springer, 2023
Keywords
Time Series, Matrix Profile, Motif Discovery, Industry 4.0
National Category
Computer Sciences
Research subject
Smart Cities and Communities
Identifiers
urn:nbn:se:hh:diva-48164 (URN)10.1007/s10618-022-00871-y (DOI)000864041100001 ()2-s2.0-85139448908 (Scopus ID)
Note

Funding: Open access funding provided by Halmstad University.

Available from: 2022-09-28 Created: 2022-09-28 Last updated: 2023-01-12Bibliographically approved
Farouq, S., Byttner, S., Bouguelia, M.-R. & Gadd, H. (2022). A conformal anomaly detection based industrial fleet monitoring framework: A case study in district heating. Expert systems with applications, 201, Article ID 116864.
Open this publication in new window or tab >>A conformal anomaly detection based industrial fleet monitoring framework: A case study in district heating
2022 (English)In: Expert systems with applications, ISSN 0957-4174, E-ISSN 1873-6793, Vol. 201, article id 116864Article in journal (Refereed) Published
Abstract [en]

The monitoring infrastructure of an industrial fleet can rely on the so-called unit-level and subfleet-level models to observe the behavior of a target unit. However, such infrastructure has to confront several challenges. First, from an anomaly detection perspective of monitoring a target unit, unit-level and subfleet-level models can give different information about the nature of an anomaly, and which approach or level model is appropriate is not always clear. Second, in the absence of well-understood prior models of unit and subfleet behavior, the choice of a base model at their respective levels, especially in an online/streaming setting, may not be clear. Third, managing false alarms is a major problem. To deal with these challenges, we proposed to rely on the conformal anomaly detection framework. In addition, an ensemble approach was deployed to mitigate the knowledge gap in understanding the underlying data-generating process at the unit and subfleet levels. Therefore, to monitor the behavior of a target unit, a unit-level ensemble model (ULEM) and a subfleet-level ensemble model (SLEM) were constructed, where each member of the respective ensemble is based on a conformal anomaly detector (CAD). However, since the information obtained by these two ensemble models through their p-values may not always agree, a combined ensemble model (CEM) was proposed. The results are based on real-world operational data obtained from district heating (DH) substations. Here, it was observed that CEM reduces the overall false alarms compared to ULEM or SLEM, albeit at the cost of some detection delay. The analysis demonstrated the advantages and limitations of ULEM, SLEM, and CEM. Furthermore, discords obtained from the state-of-the-art matrix-profile (MP) method and the combined calibration scores obtained from ULEM and SLEM were compared in an offline setting. Here, it was observed that SLEM achieved a better overall precision and detection delay. Finally, the different components related to ULEM, SLEM, and CEM were put together into what we refer to as TRANTOR: a conformal anomaly detection based industrial fleet monitoring framework. The proposed framework is expected to enable fleet operators in various domains to improve their monitoring infrastructure by efficiently detecting anomalous behavior and controlling false alarms at the target units. © 2022

Place, publisher, year, edition, pages
Oxford: Elsevier, 2022
Keywords
Conformal anomaly detection, Fleet monitoring, Unit-level model, Subfleet-level model, Ensemble model, District heating substations
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-46273 (URN)10.1016/j.eswa.2022.116864 (DOI)000798741200007 ()2-s2.0-85129522080 (Scopus ID)
Funder
Knowledge Foundation, 20160103
Note

Som manuskript i avhandling / As manuscript in thesis

Available from: 2022-02-01 Created: 2022-02-01 Last updated: 2022-09-01Bibliographically approved
Taghiyarrenani, Z., Nowaczyk, S., Pashami, S. & Bouguelia, M.-R. (2022). Adversarial Contrastive Semi-Supervised Domain Adaptation.
Open this publication in new window or tab >>Adversarial Contrastive Semi-Supervised Domain Adaptation
2022 (English)Manuscript (preprint) (Other academic)
Abstract [en]

Domain Adaptation (DA) aims to transfer knowledge from a source to a target domain by aligning their respective data distributions. In the unsupervised setting, however, this may cause the source and target samples of different classes to align to each other, consequently leading to negative transfer. Semi-Supervised Domain Adaptation (SSDA) tries to solve such class misalignment problem by exploiting a few sample labels in the target domain. This paper proposes a new SSDA method called Adversarial Contrastive Semi-Supervised Domain Adaptation (ACSSDA) which combines two objectives, optimized for the case where very few target sample labels are available, to learn a shared feature representation for both source and target domains. ACSSDA uses a domain classifier to ensure that the resulting feature space is domain agnostic. Simultaneously, Contrastive loss aims to pull together samples of the same class and push apart samples of different classes. This is shown to reduce class misalignment and negative transfer even with as little as a single labeled sample per class. We demonstrate the effectiveness of ACSSDA with experiments on several benchmark data sets. The results show the superiority of our method over state-of-the-art approaches.

Keywords
Semi-Supervised Learning, Domain Adaptation, Contrastive Loss, Neural Networks
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-47895 (URN)
Note

As manuscript in Thesis.

Available from: 2022-08-17 Created: 2022-08-17 Last updated: 2024-11-15Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0002-2859-6155

Search in DiVA

Show all publications