Title : Artificial intelligence based mouse beahviour recognition
Abstract:
The behavioural phenotype research can be of great importance as the first symptom of neurodegenerative disorders is often identifiable through subtle changes in day-to-day human behaviors. Because of the similarity and homology between humans and mice, modelling mouse behaviour provides a valuable platform to study neurodegenerative diseases such as Alzheimer’s disease and Parkinson’s disease. Comprehensive behavioural phenotypes of transgenic mice can be used to reveal the underlying functional role of genes, and can provide new insights into the pathophysiology and treatment of neurodegenerative diseases carried by the mice.
Historically, studying mouse behaviour can be a time-consuming and difficult task as the collected data requires experts to physically engage. On the other hand, the collected data requires experts to annotate and to analyse mice behaviors manually. It is a highly labor-intensive process which is error-prone and subjective to individual interpretation. Furthermore, human experts may fail to detect behavioral events that are very quick or too slow, and some behaviour events maybe missed because of dwindling attention span. Inspired by the advances in machine learning over the last decades, our works based on computer vision and pattern recognition aim to facilitate automated analysis of complex mouse behaviours. Currently our works mainly cover two problems: mouse pose estimation and mouse behaviour recognition.
As one of the fundamental problems in mouse behaviour analysis, mouse pose estimation is defined as the problem of measuring mouse posture which denotes the geometrical configuration of body parts in images or videos. This can provide useful information for relevant behaviour analysis. However, spatial contextual information between mouse body parts is weak due to highly deformable body structures. For this problem, a novel Graphical Model based Structured Context Enhancement Network (GM-SCENet) is introduced. The proposed architecture can adaptively learn and enhance the structured contextual information of each keypoint by exploring the global and keypoint-specific contextual information. Besides, a Multi-Level Keypoint Aggregation (MLKA) algorithm is designed to integrate multi-level localisation results (https://ieeexplore.ieee.org/abstract/document/9492104).
Regarding mouse behavior recognition, we propose four solutions from different perspectives:
- In order to effectively use contextual information in videos to recognize mouse behaviour, novel contextual features of interest points are exploited, which potentially describe spatial location and temporal changes without using an independent tracking or detecting algorithm. These features are then encoded as spatial-temporal stacked Fisher vectors which are used as the input to the neural network for mouse behaviour recognition(https://pureadmin.qub.ac.uk/ws/files/122523899/SciTePressPure_version_ICPRAM.pdf).
- Many mouse actions have pairwise relationships in the temporal domain. For example, it is very unlikely to have a hang or rest action immediately after a drink action. Based on this characteristic?a novel hybrid learning architecture for mouse behavior recognition is proposed in which a Hidden Markov Model is used to model the temporal transition relation of mouse behaviours (https://ieeexplore.ieee.org/abstract/document/8488486).
- Most vision-based techniques for mouse behavior recognition only rely on the single-view video recordings, which can be ambiguous when essential information of behaviours is occluded. In this work, a novel multi-view mouse behaviour recognition system based on trajectory-based motion and spatio-temporal features is introduced which aims to model: (a) the temporal relationship of image frames in each view, (b) the relationship between camera views, and (c) the correlations between the neighbouring labels (https://ieeexplore.ieee.org/abstract/document/9444134).
- Behavioral interaction information between mice is crucial for behavior recognition. Based on mouse skeleton key point information, a Cross-Skeleton Interaction Graph Aggregation Network (CS-IGANet) is proposed to effectively learn abundant interaction relationships between mice. Also, an auxiliary self-supervised learning strategy to enable the proposed model to focus on the similarity between pairwise nodes from different skeletons, so as to enhance the representation ability of the model (https://arxiv.org/abs/2208.03819).
Overall, these works have potential to contribute to conducting a wide range of behavioural experiments without human intervention. We believe these can provide another dimension to understand the relationship between neural activities and behavioural phenotypes in neurodegenerative diseases research
