hh.sePublications
Planned maintenance
A system upgrade is planned for 10/12-2024, at 12:00-13:00. During this time DiVA will be unavailable.
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Deep Learning on the Edge: A Flexible Multi-level Optimization Approach
Halmstad University, School of Information Technology.ORCID iD: 0000-0002-4674-3809
2022 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Recent advances in Deep Learning (DL) research have been adopted in a wide variety of applications, including autonomous driving, AI in health care, and smart homes. In parallel, research in high-performance embedded computing has resulted in advanced hardware platforms that offer enhanced performance and energy efficiency for demanding computations. However, the high demands of DL models for computational and memory resources are still a challenge for embedded computing. Algorithmic optimizations can be used to reduce the computational and memory requirements of DL models. Hardware implementations and architectures can also be tuned to support DL applications’ requirements. This thesis identifies that insufficient coordination between hardware implementations and models’ optimizations limits the efficiency of the resulting implementations. In addition, the implementation methods themselves suffer from poor flexibility in adapting to changes in the model and application constraints. The overarching theme of this thesis is to study and propose methods for the efficient and flexible implementation of DL models on embedded platforms. The work in this thesis bridges the gap between DL models’ algorithmic optimizations and embedded platforms’ hardware-specific optimizations, and investigates the features that need support from DL domain-specific architectures. In addition, a method for multi-objective quantization of DL models is proposed to address both the model error and platform performance metrics. Post-training optimization techniques are employed to facilitate the multiobjective optimization of the models because they do not require retraining after model optimization. This thesis also reviews the optimization methods that are known to have been applied to improve the implementation efficiency of DL models. It highlights the most fruitful optimizations found in existing, highly efficient implementations, and applies them in the proposed methods. A method for mapping Convolution Neural Networks (CNN) on Epiphany, a manycore architecture, is proposed and evaluated. A method for quantization and approximation for RNN models in a post-training fashion is also proposed, and evaluated on four RNN models. The proposed quantization method is used in a hardware-aware multi-objective optimization for RNN models to be deployed on SiLago and Bit fusion architectures.

Place, publisher, year, edition, pages
Halmstad: Halmstad University Press, 2022. , p. 65
Series
Halmstad University Dissertations ; 95
National Category
Embedded Systems
Identifiers
URN: urn:nbn:se:hh:diva-48680ISBN: 978-91-89587-03-8 (print)ISBN: 978-91-89587-02-1 (electronic)OAI: oai:DiVA.org:hh-48680DiVA, id: diva2:1712943
Public defence
2022-12-15, Halda, Visionen, Kristian IV:s, väg 3, Halmstad, 11:58 (English)
Opponent
Supervisors
Note

Kompletteras med LibrisID när tillgängligt.

Available from: 2022-11-24 Created: 2022-11-23 Last updated: 2022-11-24Bibliographically approved
List of papers
1. Recurrent Neural Networks: An Embedded Computing Perspective
Open this publication in new window or tab >>Recurrent Neural Networks: An Embedded Computing Perspective
2020 (English)In: IEEE Access, E-ISSN 2169-3536, Vol. 8, p. 57967-57996Article in journal (Refereed) Published
Abstract [en]

Recurrent Neural Networks (RNNs) are a class of machine learning algorithms used for applications with time-series and sequential data. Recently, there has been a strong interest in executing RNNs on embedded devices. However, difficulties have arisen because RNN requires high computational capability and a large memory space. In this paper, we review existing implementations of RNN models on embedded platforms and discuss the methods adopted to overcome the limitations of embedded systems. We will define the objectives of mapping RNN algorithms on embedded platforms and the challenges facing their realization. Then, we explain the components of RNN models from an implementation perspective. We also discuss the optimizations applied to RNNs to run efficiently on embedded platforms. Finally, we compare the defined objectives with the implementations and highlight some open research questions and aspects currently not addressed for embedded RNNs. Overall, applying algorithmic optimizations to RNN models and decreasing the memory access overhead is vital to obtain high efficiency. To further increase the implementation efficiency, we point up the more promising optimizations that could be applied in future research. Additionally, this article observes that high performance has been targeted by many implementations, while flexibility has, as yet, been attempted less often. Thus, the article provides some guidelines for RNN hardware designers to support flexibility in a better manner. © 2020 IEEE.

Place, publisher, year, edition, pages
Piscataway: IEEE, 2020
Keywords
Compression, flexibility, efficiency, embedded computing, long short term memory (LSTM), quantization, recurrent neural networks (RNNs)
National Category
Computer Systems
Identifiers
urn:nbn:se:hh:diva-41981 (URN)10.1109/ACCESS.2020.2982416 (DOI)000527411700168 ()2-s2.0-85082939909 (Scopus ID)
Projects
NGES (Towards Next Generation Embedded Systems: Utilizing Parallelism and Reconfigurability)
Funder
Vinnova, INT/SWD/VINN/p-10/2015
Note

As manuscript in thesis.

Other funding: Government of India

Available from: 2020-04-30 Created: 2020-04-30 Last updated: 2022-11-23Bibliographically approved
2. Streaming Tiles: Flexible Implementation of Convolution Neural Networks Inference on Manycore Architectures
Open this publication in new window or tab >>Streaming Tiles: Flexible Implementation of Convolution Neural Networks Inference on Manycore Architectures
2018 (English)In: 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Los Alamitos: IEEE Computer Society, 2018, p. 867-876Conference paper, Published paper (Refereed)
Abstract [en]

Convolution neural networks (CNN) are extensively used for deep learning applications such as image recognition and computer vision. The convolution module of these networks is highly compute-intensive. Having an efficient implementation of the convolution module enables realizing the inference part of the neural network on embedded platforms. Low precision parameters require less memory, less computation time, and less power consumption while achieving high classification accuracy. Furthermore, streaming the data over parallelized processing units saves a considerable amount of memory, which is a key concern in memory constrained embedded platforms. In this paper, we explore the design space for streamed CNN on Epiphany manycore architecture using varying precisions for weights (ranging from binary to 32-bit). Both AlexNet and GoogleNet are explored for two different memory sizes of Epiphany cores. We are able to achieve competitive performance for both Alexnet and GoogleNet with respect to emerging manycores. Furthermore, the effects of different design choices in terms of precision, memory size, and the number of cores are evaluated by applying the proposed method.

Place, publisher, year, edition, pages
Los Alamitos: IEEE Computer Society, 2018
Keywords
manycores, CNN, stream processing, embedded systems
National Category
Embedded Systems
Identifiers
urn:nbn:se:hh:diva-36887 (URN)10.1109/IPDPSW.2018.00138 (DOI)000541051600099 ()2-s2.0-85052195969 (Scopus ID)978-1-5386-5555-9 (ISBN)978-1-5386-5556-6 (ISBN)
Conference
The 7th International Workshop on Parallel and Distributed Computing for Large Scale Machine Learning and Big Data Analytics, Vancouver, British Columbia, Canada, May 21, 2018
Projects
NGES (Towards Next Generation Embedded Systems: Utilizing Parallelism and Reconfigurability)
Funder
Vinnova
Note

As manuscript in thesis.

Other funding: Department of Science and Technology, Government of India.

Available from: 2018-06-01 Created: 2018-06-01 Last updated: 2023-10-05Bibliographically approved
3. MOHAQ: Multi-Objective Hardware-Aware Quantization of recurrent neural networks
Open this publication in new window or tab >>MOHAQ: Multi-Objective Hardware-Aware Quantization of recurrent neural networks
Show others...
2022 (English)In: Journal of systems architecture, ISSN 1383-7621, E-ISSN 1873-6165, Vol. 133, article id 102778Article in journal (Refereed) Published
Abstract [en]

The compression of deep learning models is of fundamental importance in deploying such models to edge devices. The selection of compression parameters can be automated to meet changes in the hardware platform and application. This article introduces a Multi-Objective Hardware-Aware Quantization (MOHAQ) method, which considers hardware performance and inference error as objectives for mixed-precision quantization. The proposed method feasibly evaluates candidate solutions in a large search space by relying on two steps. First, post-training quantization is applied for fast solution evaluation (inference-only search). Second, we propose the ”beacon-based search” to retrain selected solutions only and use them as beacons to estimate the effect of retraining on other solutions. We use speech recognition models on TIMIT dataset. Experimental evaluations show that Simple Recurrent Unit (SRU)-based models can be compressed up to 8x by post-training quantization without any significant error increase. On SiLago, we found solutions that achieve 97% and 86% of the maximum possible speedup and energy saving, with a minor increase in error on an SRU-based model. On Bitfusion, the beacon-based search reduced the error gain of the inference-only search on SRU-based models and Light Gated Recurrent Unit (LiGRU)-based model by up to 4.9 and 3.9 percentage points, respectively.

Place, publisher, year, edition, pages
Amsterdam: Elsevier, 2022
Keywords
Simple recurrent unit, Light gated recurrent unit, Quantization, Multi-objective optimization, Genetic algorithms
National Category
Embedded Systems
Identifiers
urn:nbn:se:hh:diva-48679 (URN)10.1016/j.sysarc.2022.102778 (DOI)000892114100006 ()2-s2.0-85141919627 (Scopus ID)
Available from: 2022-11-23 Created: 2022-11-23 Last updated: 2023-08-21Bibliographically approved
4. Shrink and Eliminate: A Study of Post-Training Quantization and Repeated Operations Elimination in RNN Models
Open this publication in new window or tab >>Shrink and Eliminate: A Study of Post-Training Quantization and Repeated Operations Elimination in RNN Models
2022 (English)In: Information, E-ISSN 2078-2489, Vol. 13, no 4, article id 176Article in journal (Refereed) Published
Abstract [en]

Recurrent neural networks (RNNs) are neural networks (NN) designed for time-series applications. There is a growing interest in running RNNs to support these applications on edge devices. However, RNNs have large memory and computational demands that make them challenging to implement on edge devices. Quantization is used to shrink the size and the computational needs of such models by decreasing weights and activation precision. Further, the delta networks method increases the sparsity in activation vectors by relying on the temporal relationship between successive input sequences to eliminate repeated computations and memory accesses. In this paper, we study the effect of quantization on LSTM-, GRU-, LiGRU-, and SRU-based RNN models for speech recognition on the TIMIT dataset. We show how to apply post-training quantization on these models with a minimal increase in the error by skipping quantization of selected paths. In addition, we show that the quantization of activation vectors in RNNs to integer precision leads to considerable sparsity if the delta networks method is applied. Then, we propose a method for increasing the sparsity in the activation vectors while minimizing the error and maximizing the percentage of eliminated computations. The proposed quantization method managed to com-press the four models more than 85%, with an error increase of 0.6, 0, 2.1, and 0.2 percentage points, respectively. By applying the delta networks method to the quantized models, more than 50% of the operations can be eliminated, in most cases with only a minor increase in the error. Comparing the four models to each other under the quantization and delta networks method, we found that compressed LSTM-based models are the most-optimum solutions at low-error-rates constraints. The compressed SRU-based models are the smallest in size, suitable when higher error rates are acceptable, and the compressed LiGRU-based models have the highest number of eliminated operations. © 2022 by the authors. Licensee MDPI, Basel, Switzerland.

Place, publisher, year, edition, pages
Basel: MDPI, 2022
Keywords
delta networks, edge devices, quantization, recurrent neural network
National Category
Computer Sciences
Identifiers
urn:nbn:se:hh:diva-46752 (URN)10.3390/info13040176 (DOI)000786262400001 ()34789458 (PubMedID)2-s2.0-85128393517 (Scopus ID)
Funder
ELLIIT - The Linköping‐Lund Initiative on IT and Mobile Communications
Available from: 2022-05-02 Created: 2022-05-02 Last updated: 2022-11-23Bibliographically approved

Open Access in DiVA

fulltext(2760 kB)272 downloads
File information
File name FULLTEXT01.pdfFile size 2760 kBChecksum SHA-512
112c0a822d1227d4c9f2c7f12a54bc7874a9692abfbd79a1f8a225f2b046c6a382620a0042bc56b7d0829112ccac925776c3202e743b991daf0346c85bb59c35
Type fulltextMimetype application/pdf

Authority records

Rezk, Nesma

Search in DiVA

By author/editor
Rezk, Nesma
By organisation
School of Information Technology
Embedded Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 273 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1049 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf