hh.sePublications
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
From Domain Adaptation to Federated Learning
Halmstad University, School of Information Technology.ORCID iD: 0000-0002-1759-8593
2024 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Data-driven methods have been gaining increasing attention; however, along with the benefits they offer, they also present several challenges, particularly concerning data availability, accessibility, and heterogeneity, the three factors that have shaped the development of this thesis. Data availability is the primary consideration when employing data-driven methodologies. Suppose we consider a system for which we aim to develop a Machine Learning (ML) model. Gathering labeled samples, particularly in the context of real-world problem-solving, consistently poses challenges. While collecting raw data may be feasible in certain situations, the process of labeling them is often difficult, leading to a shortage of labeled data. However, historical (outdated) data or labeled data may occasionally be available from different yet related systems. A feasible approach would be to leverage data from different but related sources to assist in situations in which data is scarce. The challenge with this approach is that data collected from various sources may exhibit statistical differences even if they have the same features, i.e., data heterogeneity. Data heterogeneity impacts the performance of ML models. This issue arises because conventional machine learning algorithms assume what’s known as the IID (Independently and Identically Distributed) assumption; training and test data come from the same underlying distribution and are independent and identically sampled. The IID assumption may not hold when data comes from different sources and can result in a trained model performing less effectively when used in another system or context. In such situations, Domain Adaptation (DA) is a solution. DA enhances the performance of ML models by minimizing the distribution distance between samples originating from diverse resources. Several factors come into play within the DA context, each necessitating distinct DA methods. In this thesis, we conduct an investigation and propose DA methods while considering various factors, including the number of domains involved, the quantity of data available (both labeled and unlabeled) within these domains, the task at hand (classification or regression), and the nature of statistical heterogeneity among samples from different domains, such as covariate shift or concept shift. It is crucial to emphasize that DA techniques work by assuming that we access the data from different resources. Data may be owned by different data owners, and data owners are willing to share their data. This data accessibility enables us to adapt data and optimize models accordingly. However, privacy concerns become a significant issue when addressing real-world problems, for example, where the data owners are from industry sectors. These privacy considerations necessitate the development of privacy-preserving techniques, such as Federated Learning (FL). FL is a privacy-preserving machine learning technique that enables different data owners to collaborate without sharing raw data samples. Instead, they share their ML models or model updates. Through this collaborative process, a global machine learning model is constructed, which can generalize and perform well across all participating domains. This approach addresses privacy concerns by keeping individual data localized while benefiting from collective knowledge to improve the global model. Among the most widely accepted FL methods is Federated Averaging (FedAvg). In this method, all clients connect with a central server. The server then computes the global model by aggregating the local models from each client, typically by calculating their average. Similar to DA, FL encounters issues when data from different domains exhibit statistical differences, i.e., heterogeneity, that can negatively affect the performance of the global model. A specialized branch known as Heterogeneous FL has emerged to tackle this situation. This thesis, alongside DA, considers the heterogeneous FL problem. This thesis examines FL scenarios where all clients possess labeled data. We begin by conducting experimental investigations to illustrate the impact of various types of heterogeneity on the outcomes of FL. Afterward, we perform a theoretical analysis and establish an upper bound for the risk of the global model for each client. Accordingly, we see that minimizing heterogeneity between the clients minimizes this upper bound. Building upon this insight, we develop a method aimed at minimizing this heterogeneity to personalize the global model for the clients, thereby enhancing the performance of the federated system. This thesis focuses on two practical applications that highlight the relevant challenges: Predictive Maintenance and Network Security. In predictive maintenance, the focus is on fault identification using both DA and FL. Additionally, the thesis investigates predicting the state of health of electric bus batteries using DA. Regarding network security applications, the thesis addresses network traffic classification and intrusion detection, employing DA. ©Zahra Taghiyarrenani.

Place, publisher, year, edition, pages
Halmstad: Halmstad University Press, 2024. , p. 37
Series
Halmstad University Dissertations ; 107
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:hh:diva-52510ISBN: 978-91-89587-28-1 (print)ISBN: 978-91-89587-27-4 (electronic)OAI: oai:DiVA.org:hh-52510DiVA, id: diva2:1833004
Public defence
2024-02-22, Wigforss, Kristian IV:s väg 3, Halmstad, 10:00 (English)
Opponent
Supervisors
Available from: 2024-02-01 Created: 2024-01-31 Last updated: 2024-02-01Bibliographically approved
List of papers
1. Noise-robust representation for fault identification with limited data via data augmentation
Open this publication in new window or tab >>Noise-robust representation for fault identification with limited data via data augmentation
2022 (English)In: Proceedings of the European Conference of the Prognostics and Health Management Society 2022 / [ed] Phuc Do; Gabriel Michau; Cordelia Ezhilarasu, State College, PA: Prognostics and Health Management Society , 2022, Vol. 7 (1), p. 473-479Conference paper, Published paper (Refereed)
Abstract [en]

Noise will be unavoidably present in the data collected from physical environments, regardless of how sophisticated the measurement equipment is. Furthermore, collecting enough faulty data is a challenge since operating industrial machines in faulty modes not only has severe consequences to the machine health, but also may affect collateral machinery critically, from health state point of view. In this paper, we propose a method of denoising with limited data for the purpose of fault identification. In addition, our method is capable of removing multiple levels of noise simultaneously. For this purpose, inspired by unsupervised contrastive learning, we first augment the data with multiple levels of noise. Later, we construct a new feature representation using Contrastive Loss. The last step is building a classifier on top of the learned representation; this classifier can detect various faults in noisy environments. The experiments on the SOUTHEAST UNIVERSITY (SEU) dataset of bearings confirm that our method can simultaneously remove multiple noise levels.

Place, publisher, year, edition, pages
State College, PA: Prognostics and Health Management Society, 2022
Series
Proceedings of the European Conference of the Prognostics and Health Management Society (PHME), E-ISSN 2325-016X
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-47893 (URN)10.36001/phme.2022.v7i1.3334 (DOI)978-1-936263-36-3 (ISBN)
Conference
The 7th European Conference of the Prognostics and Health Management Society, Turin, Italy, July 6-8, 2022
Funder
Vinnova
Available from: 2022-08-17 Created: 2022-08-17 Last updated: 2024-01-31Bibliographically approved
2. Facilitating Semi-Supervised Domain Adaptation through Few-shot and Self-supervised Learning
Open this publication in new window or tab >>Facilitating Semi-Supervised Domain Adaptation through Few-shot and Self-supervised Learning
(English)Manuscript (preprint) (Other academic)
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-52509 (URN)
Note

Som manuskript i avhandling / As manuscript in thesis

Available from: 2024-01-31 Created: 2024-01-31 Last updated: 2024-01-31Bibliographically approved
3. Multi-Domain Adaptation for Regression under Conditional Distribution Shift
Open this publication in new window or tab >>Multi-Domain Adaptation for Regression under Conditional Distribution Shift
2023 (English)In: Expert systems with applications, ISSN 0957-4174, E-ISSN 1873-6793, Vol. 224, article id 119907Article in journal (Refereed) Published
Abstract [en]

Domain adaptation (DA) methods facilitate cross-domain learning by minimizing the marginal or conditional distribution shift between domains. However, the conditional distribution shift is not well addressed by existing DA techniques for the cross-domain regression learning task. In this paper, we propose Multi-Domain Adaptation for Regression under Conditional shift (DARC) method. DARC constructs a shared feature space such that linear regression on top of that space generalizes to all domains. In other words, DARC aligns different domains and makes explicit the task-related information encoded in the values of the dependent variable. It is achieved using a novel Pairwise Similarity Preserver (PSP) loss function. PSP incentivizes the differences between the outcomes of any two samples, regardless of their domain(s), to match the distance between these samples in the constructed space.

We perform experiments in both two-domain and multi-domain settings. The two-domain setting is helpful, especially when one domain contains few available labeled samples and can benefit from adaptation to a domain with many labeled samples. The multi-domain setting allows several domains, each with limited data, to be adapted collectively; thus, multiple domains compensate for each other’s lack of data. The results from all the experiments conducted both on synthetic and real-world datasets confirm the effectiveness of DARC. © 2023 The Authors

Place, publisher, year, edition, pages
Oxford: Elsevier, 2023
Keywords
Regression, Multi-Domain Adaptation, Conditional Shift, Concept Shift, Neural Networks, Siamese neural networks
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-47894 (URN)10.1016/j.eswa.2023.119907 (DOI)000966508000001 ()2-s2.0-85151474329 (Scopus ID)
Funder
VinnovaKnowledge Foundation
Available from: 2022-08-17 Created: 2022-08-17 Last updated: 2024-01-31Bibliographically approved
4. ITL-IDS: Incremental Transfer Learning for Intrusion Detection Systems
Open this publication in new window or tab >>ITL-IDS: Incremental Transfer Learning for Intrusion Detection Systems
2022 (English)In: Knowledge-Based Systems, ISSN 0950-7051, E-ISSN 1872-7409, Vol. 253, article id 109542Article in journal (Refereed) Published
Abstract [en]

Utilizing machine learning methods to detect intrusion into computer networks is a trending topic in information security research. The limitation of labeled samples is one of the challenges in this area. This challenge makes it difficult to build accurate learning models for intrusion detection. Transfer learning is one of the methods to counter such a challenge in machine learning topics. On the other hand, the emergence of new technologies and applications might bring new vulnerabilities to computer networks. Therefore, the learning process cannot occur all at once. Incremental learning is a practical standpoint to confront this challenge. This research presents a new framework for intrusion detection systems called ITL-IDS that can potentially start learning in a network without prior knowledge. It begins with an incremental clustering algorithm to detect clusters’ numbers and shape without prior assumptions about the attacks. The outcomes are candidates to transfer knowledge between other instances of ITL-IDS. In each iteration, transfer learning provides target environments with incremental knowledge. Our evaluation shows that this method can combine incremental and transfer learning to identify new attacks. © 2022 Published by Elsevier B.V.

Place, publisher, year, edition, pages
Amsterdam: Elsevier, 2022
Keywords
Network security, Intrusion detection system, NIDS, Transfer learning, Incremental learning
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-47891 (URN)10.1016/j.knosys.2022.109542 (DOI)000861208200008 ()2-s2.0-85135717752 (Scopus ID)
Available from: 2022-08-17 Created: 2022-08-17 Last updated: 2024-01-31Bibliographically approved
5. Domain Adaptation with Maximum Margin Criterion with application to network traffic classification
Open this publication in new window or tab >>Domain Adaptation with Maximum Margin Criterion with application to network traffic classification
2023 (English)In: Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part II / [ed] Irena Koprinska et al., Cham: Springer, 2023, p. 159-169Conference paper, Published paper (Refereed)
Abstract [en]

A fundamental assumption in machine learning is that training and test samples follow the same distribution. Therefore, for training a machine learning-based network traffic classifier, it is necessary to use samples obtained from the desired network. Collecting enough training data, however, can be challenging in many cases. Domain adaptation allows samples from other networks to be utilized. In order to satisfy the aforementioned assumption, domain adaptation reduces the distance between the distribution of the samples in the desired network and that of the available samples in other networks. However, it is important to note that the applications in two different networks can differ considerably. Taking this into account, in this paper, we present a new domain adaptation method for classifying network traffic. Thus, we use the labeled samples from a network and adapt them to the few labeled samples from the desired network; In other words, we adapt shared applications while preserving the information about non-shared applications. In order to demonstrate the efficacy of our method, we construct five different cross-network datasets using the Brazil dataset. These results indicate the effectiveness of adapting samples between different domains using the proposed method.

Place, publisher, year, edition, pages
Cham: Springer, 2023
Series
Communications in Computer and Information Science, ISSN 1865-0929, E-ISSN 1865-0937 ; 1753
Keywords
Traffic classification, Domain Adaptation, Transfer Learning
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-47897 (URN)10.1007/978-3-031-23633-4_12 (DOI)000967761200012 ()2-s2.0-85149944135 (Scopus ID)978-3-031-23632-7 (ISBN)978-3-031-23633-4 (ISBN)
Conference
ECML/PKDD 2022 Workshop on Machine Learning for Cyber Security, Grenoble, France, 19-23 September, 2022
Available from: 2022-08-17 Created: 2022-08-17 Last updated: 2024-01-31Bibliographically approved
6. Towards Geometry-Preserving Domain Adaptation for Fault Identification
Open this publication in new window or tab >>Towards Geometry-Preserving Domain Adaptation for Fault Identification
2022 (English)In: Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part II / [ed] Irena Koprinska, Paolo Mignone, Riccardo Guidotti, Szymon Jaroszewicz, Holger Fröning, Springer Nature, 2022Conference paper, Published paper (Refereed)
Abstract [en]

In most industries, the working conditions of equipment vary significantly from one site to another, from one time of a year to another, and so on. This variation poses a severe challenge for data-driven fault identification methods: it introduces a change in the data distribution. This contradicts the underlying assumption of most machine learning methods, namely that training and test samples follow the same distribution. Domain Adaptation (DA) methods aim to address this problem by minimizing the distribution distance between training (source) and test (target) samples.

However, in the area of predictive maintenance, this idea is complicated by the fact that different classes – fault categories – also vary across domains. Most of the state-of-the-art DA methods assume that the data in the target domain is complete, i.e., that we have access to examples from all the possible classes or faulty categories during adaptation. In reality, this is often very difficult to guarantee.

Therefore, there is a need for a domain adaptation method that is able to align the source and target domains even in cases of having access to an incomplete set of test data. This paper presents our work in progress as we propose an approach for such a setting based on maintaining the geometry information of source samples during the adaptation. This way, the model can capture the relationships between different fault categories and preserve them in the constructed domain-invariant feature space, even in situations where some classes are entirely missing. This paper examines this idea using artificial data sets to demonstrate the effectiveness of geometry-preserving transformation. We have also started investigations on real-world predictive maintenance datasets, such as CWRU.

Place, publisher, year, edition, pages
Springer Nature, 2022
Series
Communications in Computer and Information Science ; 1753
Keywords
Predictive Maintenance, Fault identification, Domain Adaptation, Geometry
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-47896 (URN)10.1007/978-3-031-23633-4_30 (DOI)000967761200030 ()2-s2.0-85149926744 (Scopus ID)978-3-031-23632-7 (ISBN)978-3-031-23633-4 (ISBN)
Conference
ECML/PKDD 2022 Workshop on IoT Streams for Predictive Maintenance, Grenoble, France, 19-23 September, 2022
Funder
Knowledge FoundationVinnova
Available from: 2022-08-17 Created: 2022-08-17 Last updated: 2024-01-31Bibliographically approved
7. Analysis of Statistical Data Heterogeneity in Federated Fault Identification
Open this publication in new window or tab >>Analysis of Statistical Data Heterogeneity in Federated Fault Identification
2023 (English)In: Proceedings of the Asia Pacific Conference of the PHM Society 2023 / [ed] Takehisa Yairi; Samir Khan; Seiji Tsutsumi, New York: The Prognostics and Health Management Society , 2023, Vol. 4Conference paper, Published paper (Refereed)
Abstract [en]

Federated Learning (FL) is a setting where different clients collaboratively train a Machine Learning model in a privacy-preserving manner, i.e., without the requirement to share data. Given the importance of security and privacy in real-world applications, FL is gaining popularity in many areas, including predictive maintenance. For example, it allows independent companies to construct a model collaboratively. However, since different companies operate in different environments, their working conditions may differ, resulting in heterogeneity among their data distributions. This paper considers the fault identification problem and simulates different scenarios of data heterogeneity. Such a setting remains challenging for popular FL algorithms, and thus we demonstrate the considerations to be taken into account when designing federated predictive maintenance solutions.  

Place, publisher, year, edition, pages
New York: The Prognostics and Health Management Society, 2023
Series
Proceedings of the Asia Pacific Conference of the PHM Society, E-ISSN 2994-7219
Keywords
Predictive Maintenance, Federated Learning, Predictive Maintenance Federated Learning Statistical Heterogeneity
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-52478 (URN)10.36001/phmap.2023.v4i1.3708 (DOI)
Conference
4th Asia Pacific Conference of the Prognostics and Health Management, Tokyo, Japan, September 11-14, 2023
Funder
Vinnova
Available from: 2024-01-25 Created: 2024-01-25 Last updated: 2024-01-31Bibliographically approved
8. Heterogeneous Federated Learning via Personalized Generative Networks
Open this publication in new window or tab >>Heterogeneous Federated Learning via Personalized Generative Networks
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Federated Learning (FL) allows several clients to construct a common global machine-learning model without having to share their data. FL, however, faces the challenge of statistical heterogeneity between the client's data, which degrades performance and slows down the convergence toward the global model. In this paper, we provide theoretical proof that minimizing heterogeneity between clients facilitates the convergence of a global model for every single client. This becomes particularly important under empirical concept shifts among clients, rather than merely considering imbalanced classes, which have been studied until now. Therefore, we propose a method for knowledge transfer between clients where the server trains client-specific generators. Each generator generates samples for the corresponding client to remove the conflict with other clients' models. Experiments conducted on synthetic and real data, along with a theoretical study, support the effectiveness of our method in constructing a well-generalizable global model by reducing the conflict between local models.

National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-52480 (URN)10.48550/arXiv.2308.13265 (DOI)
Funder
Knowledge FoundationVinnova
Note

As manuscript in thesis

Available from: 2024-01-25 Created: 2024-01-25 Last updated: 2024-01-31Bibliographically approved

Open Access in DiVA

fulltext(2269 kB)22 downloads
File information
File name FULLTEXT02.pdfFile size 2269 kBChecksum SHA-512
5e7fdce2ccccfb14268b4ab4e4953e33b2ed6ae2036d5a707e844b464a18a674d977b990075d7877df9016a1adbe7ae40bfd2a9415374cdae44a6002bc99ec2e
Type fulltextMimetype application/pdf

Authority records

Taghiyarrenani, Zahra

Search in DiVA

By author/editor
Taghiyarrenani, Zahra
By organisation
School of Information Technology
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 22 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 136 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf