Although Virtual Reality is considered by most people a great piece of technology, users commonly experience cybersickness symptoms. But there is hope! Here, I will explore how data science can help predict cybersickness, which might be useful in employing cybersickness mitigation mechanisms.
This is somewhat of an advanced reading. You might struggle to understand some concepts if you are not used to statistics and some basic data science.
All the analyses were conducted in Python and the complete project is available on my GitHub at the end of the post.
Introduction
High-Level Overview
Currently, virtual reality (VR) is one of the hottest topics in technology. Cybersickness, however, represents one of the largest concerns of the industry. This condition is characterized by the same symptoms as motion sickness, including nausea, dizziness, disorientation, headaches, sweating, fatigue, and eye strain. The prevalence of these symptoms varies between 20 and 45%, with females having a higher incidence. While the etiology of that discomfort has been extensively studied, no consensus has been reached regarding its causes. My belief is that we fail to stabilize our bodies, resulting in cybersickness. The postural instability theory is based on this concept. You can see more about the theory and studies related to it in the article we recently published HERE.
Consequently, analyzing how people control their posture may provide some insight into who is more likely to experience motion sickness. The purpose of this article is to examine the relationship between cybersickness and posture. Specifically, we will examine machine-learning models trained with real-world data to determine if those models can be used to predict cybersickness. This is relevant because if we are able to predict people who will become sick in a virtual environment, then we may be able to develop specific strategies to mitigate cybersickness before the user becomes sick.
Description of the data
The dataset used in this analysis comes from three different experiments. In all of these experiments, before wearing the VR-headset and playing a driving game, participants stood on a force plate for one minute to perform two tasks. The first task required them to stare at the wall (inspection task) while the second required them to read a text on a paper on the wall (search task). As a result, we examined how the body moved in the tasks along the medio-lateral and anterior-posterior axes. After completing these tasks, participants played a virtual reality driving game. The participants played for 15 minutes or discontinued the game if they became ill.
Cybersickness Incidence
In total, there were 138 participants, 83 females and 55 males. Regarding cybersickness, 28.92% of the females got sick, and 20% of the males.
Body Sway
The graphs below depict the mean of the body sway time series data across participants within that condition. A convex-hull method was used to calculate the area. The area of the average X and Y displacements of all participants is depicted in the figure title.
When analyzing body sway patterns for the two tasks, we can see that the shapes and area were somewhat similar. It seems that for the search task, participants swayed more laterally in relation to the other conditions.
Body Sway and Cybersickness
Females
Analyzing the body sway patterns for females, we can see some differences between people who got sick and those who did not. For instance, in the inspection task, the area is much larger. Similarly, for the search task, but in much smaller magnitude. Please note that this task was performed before playing the VR game. So we did not know who would get sick. Still, those differences could be correlated to cybersickness.
Males
For males, we found a similar pattern, with those participants who got sick, showing a larger area. That might indicate that the body sway area could be an indicator of cybersickness.
Using Machine Learning to Predict Cybersickness
As part of our effort to predict cybersickness, we will build two machine-learning models (actually four). In the first model, I will use the time-series data obtained from the force plates (X, and Y coordinates). An advantage of this method is that the model can capture the complexities of the data as they are, rather than using derivates such as area or standard deviation that may not adequately represent the pattern of the data. For the second model, I will use a more common model that uses area as the main feature to predict cybersickness and does not consider the time series data.
Time Series ML-Model
To predict cybersickness, we will build a BOSSVS model from the package pyts. The approach of this model involves breaking down the time series data into smaller segments and analyzing the specific pattern in that segment using Symbolic Fourier Approximation (See more here). Then, the transformed time series data, represented as fixed-length vectors through BOSSVS, is used to train a classification. This model learns the relationship between the BOSSVS-encoded vectors and the corresponding class labels, enabling predictions for new time series based on their encoded representations.
Results – Search Task
The model correctly classified sickness for 83% of the test samples. However, it is important to note that there were more 0s (non-sick) than 1s (sick). If we analyze the 1s, we can see that the model got 7/13 correct (~53%).
Results – Inspection Condition
The model correctly classified sickness for 73% of the test samples. However, it is important to note that there were more 0s (non-sick) than 1s (sick). Additionally, this model was weak at predicting the 1s (sick participants), getting only 2/13 right (~15%).
Discrete data ML-Model
For the following models, the time-series data will be removed from the dataset, leaving the following features: experiment, sex, and body sway area. To select the optimal model, again I selected the model with the best accuracy. The selection was done using the models: LogisticRegression, RandomForestClassifier, SVC, GradientBoostingClassifier, and DecisionTreeClassifier. These models are available in the scikit-learn (sklearn) library.
Results – Search Task
The model achieved a 71% accuracy in correctly labeling the test data. The model correctly classified 3/13 (23%) of 1s. Also, it misclassified 4/29 cases in the non-sick group. Certainly, the performance was much worse than the time series model, but potentially with some changes, this model could be improved. Maybe implementing an optimization algorithm, may help to strengthen its predictive capabilities.
Results – Inspection Task
This model exhibited lower efficiency compared to the previous ones, achieving an accuracy of 69%. However, upon examining the confusion matrix, it becomes evident that the model struggled in identifying individuals who were sick. Remarkably, it appears that the inspection task lacked the necessary sensitivity to promote variations in body sway that may contribute to cybersickness.
Comparing Models
In comparing the models used in this study, it can be seen that using time series data is fundamental to predicting cybersickness. A second interesting finding was that the results generated by the two tasks were very different. Cybersickness appears not to be related to the inspection task. The results indicate that more research should be conducted to determine the most optimal task for capturing the body sway patterns, which will result in more accurate predictions. Finally, in future studies, combining different algorithms could be explored to further improve the accuracy of the models.
Conclusion
In conclusion, our analysis of the different models used in this study reveals that the time series models outperformed the discrete models in terms of predictive accuracy. However, it is important to consider the limitations of the dataset, particularly its size and the variation in data sources due to different experiments.
The superiority of the time series models highlights the importance of capturing temporal patterns and dependencies in the data for accurate predictions. Leveraging the sequential nature of the time series data provided a more comprehensive understanding of the underlying patterns, resulting in improved model performance.
In light of the constraints posed by our analyses and datasets, we have successfully demonstrated the feasibility of utilizing motion data to predict cybersickness prior to individuals entering VR. This finding holds immense significance as it enables VR developers to design a preliminary application that identifies individuals susceptible to cybersickness before engaging in VR experiences. This proactive approach empowers developers to implement tailored strategies for cybersickness mitigation, thereby enhancing the overall user experience and ensuring user comfort and safety during VR interactions.
Improvements
This is a highly exploratory investigation and certainly, there is much room for improvement. Certainly, including a larger dataset might significantly increase the accuracy of the models. Predicting cybersickness is an ongoing process, and over time more data will be available to improve the models. Another improvement relates to how the models were designed. Potentially, trying out more models, and experimenting with different parameters could help to further improve the results. In addition, using a combination of models or algorithms could also be explored. Finally, in this analysis, we divided the models according to the tasks. However, maybe including the two tasks in the same model could feed the model with relevant information and improve its performance.
Acknowledgments
The data used in this project is the property of Affordance Perception-Action Laboratory. It is not to be published, shared, or distributed without explicit authorization. This data is provided solely for the purpose of transparency when generating the models. If you are interested in using this data, please contact me to request permission and obtain the necessary authorization. Any unauthorized use or distribution of this data is strictly prohibited and may result in legal action.
Extras – Contact Information
This blog post is an initial effort to develop predictive models for cybersickness. If you want to collaborate and help, please reach out to my email at daniloarruda13@gmail.com. Also, the complete analysis and scripts are publicly available in a GitHub repository here
Leave a Reply