Machine activity recognition (MAR) is an essential and effective approach for equipment productivity monitoring. Developing MAR methods for forklift trucks, a vital piece of the industry, can benefit productivity efficiency, maintenance service, product design, and potential savings. With the growth of the Internet of Things, a large amount of sensory data has become accessible. Conventional MAR methods that have been developed primarily focus on data collected from external sensors, such as inertial measurement units (IMUs) and cameras. However, they are not effective for forklift applications: the IMU data does not reflect kinematic patterns due to a lack of large articulated parts, while the vision-based data collection requires many cameras to create sufficient coverage of an indoor environment, which, in result, risks the privacy and is less economical. Moreover, typical objectives in the existing MAR works are heavy equipment in construction sites where the working environment and tasks differ from the logistics sector. Therefore, it is necessary to develop intelligent and innovative approaches that are more suitable for forklift trucks.

This thesis demonstrates developing and utilizing representation learning methods to solve forklift MAR problems, based on the assumption that forklift activities are formed by a series of basic movements that can be detected from the onboard communication, i.e., signals in a Controller Area Network (CAN). Most of the methods proposed in this thesis incorporate semi-supervised techniques to deal with the limited amount of labeled data and to capitalize on a large amount of unlabeled data in our experiments. Deep neural networks are implemented to overcome different challenges of recognizing forklift activities and learn various representations of the data: i) learning invariant features to reconstruct input CAN signals by applying autoencoders, ii) learning discriminative features to recognize forklift activities by fine-tuning pre-training networks, and iii) learning temporal coherence to capture activity transitions by implementing gated recurrent units. Apart from achieving promising classification performance for forklift MAR problems, the representations obtained also support visualization and interpretability of the data as they are three-dimensional. Our ongoing works are new experiments about learning domain-invariant features, where domain adaptation methods are implemented to recognize activities performed by forklift trucks from different sites.

Machine Activity Recognition (MAR) is a research topic that focuses on the development of data-driven methods to improve productivity monitoring. The application and the perspective of MAR research jointly influence the diffi- culty of a MAR problem. Unlike previous MAR works, which have studied construction machinery from the viewpoint of the user, this project focuses on logistics equipment from the viewpoint of the original equipment manufac- turer. In terms of the application, forklift trucks have flexible functions and complex usage. The former is an intrinsic characteristic, as forklifts are me- chanically agile, and the latter is an extrinsic factor, as forklift usage can vary greatly with different drivers, loads, work shifts, and warehouse environments. As for the standpoint, manufacturers have customers who use their products all over the world. Studying a single machine or machines in a single site, i.e. the conventional MAR setting, cannot provide a general understanding of the equipment usage. Therefore, existing MAR methods with external sensory data and only supervised learning techniques are impractical in this case.

This thesis investigates learning representation-based methods for recog- nizing forklift routine activities using on-board sensory signals. Three methods are developed to capture important data features to overcome the challenges of forklift MAR. First, by pre-training autoencoders with unlabeled data and then fine-tuning them with pseudo-labeled data, discriminative features can be ex- tracted. Classifiers built on these features can outperform conventional MAR solutions that use only the labeled data. Second, training gated recurrent unit networks to recognize activities in different contexts can help to learn a repre- sentation that captures activities and their transitions, which further improves the MAR result. Third, implementing domain adversarial-training neural net- works with pseudo-labeled data can essentially compensate for the limited la- beled data from source domains, leading to representations that are informative for more than one domain. In addition, testing the full method on a real truck has demonstrated the applicability of the proposed method and the feasibility of an online MAR solution.