Aleksandra Govorova, Department of Software Engineering and Computer Systems ITMO University, St. Petersburg, Russia, firstname.lastname@example.org
Abstract — The article is dedicated to the problem of evaluating the functional state of a person experiencing a stressful state while working with a virtual reality simulator. An approach for assessing stress levels using a heart rate sensor is proposed. The approach is based on the following calculations: the number of deviations from the optimal route, the number of errors made, and the simulation time.
Keywords: virtual reality, VR, safety, emergency, simulation, personnel training
© The Authors, published by CULTURAL-EDUCATIONAL CENTER, LLC, 2020
This work is licensed under Attribution-NonCommercial 4.0 International
To date, virtual reality applications have reached a high technological level and have become widespread in numerous areas of human activity.
First, in the modern world, virtual reality technologies are actively used in the gaming industry. Developers create interactive games that allow players to fully immerse themselves in the gameplay, become a part of the game world. In addition, various simulators are being developed, which make it possible to become, for example, an airplane pilot, or a firefighter entering a burning house. Such simulators are used not only for entertainment. Their main purpose is training because VR technology allows to create the most realistic conditions. This means that users can plunge into extreme conditions in which people must work in different professions without any danger to their own lives. Based on the obtained data on the behavior and emotional state of the test participants, it is possible to judge the degree of their readiness to work in these conditions .
Even though there are a great many different VR systems, all of them are united by the effect they produce — the effect of immersion. The essence of this effect is that the user does not perceive himself as a bystander but becomes a part of a virtual environment that feels “real” (or “almost real”).
Thus, the relevance of this work lies in the fact that the use of actively developing and widely used virtual reality technologies allows to prepare a person for actions that need to be performed in various emergency situations. Since virtual reality provides a sufficiently high level of immersion, this method can be considered more effective than simply conducting lectures on this topic and even than training evacuations and other simulations undertaken in real life. This happens because the circumstances of their conduct may not provide a sufficient level of plausibility of the event.
Now there are several different virtual reality simulators designed both for training personnel in professional duties and peculiarities of working in stressful conditions, which includes preparing for emergency situations. Such simulators are designed to replace the classical methods of training, such as lectures and seminars, video tutorials. However, all presented simulators have a noticeable drawback — they do not have the ability to assess the behavior of the student. Assessment methods remain the same as before: written tests, interviews with the examiner. In addition, such training methods do not consider the level of emotional tension.
Strong emotional tension associated with work in extreme conditions causes decrease in performance (the correct sequence of operations is violated, the pace of work slows down, the number of mistakes increases). Most often, such a decrease is observed at a low level of training. Conversely, people’s performance increases if they have studied in advance the place where the activity will take place, the features of the activity, tools and means of labor. Thus, it is known that ignorance of the situation causes, as a rule, much greater mental stress than a certain signal of danger .
Thus, there is interest in creating a new type of simulators that can automatically assess the knowledge of students. Considering the specifics of safety simulators, the user has the following requirements: he or she must remember the necessary procedure as well as successfully cope with stress. To determine the level of stress, it is necessary to assess the functional state of the person, which has never been done in such simulators before. The correct assessment of the functional state of the organism and its adaptive reserves to various kinds of impacts remains now an unsolved problem.
II. Methods and Materials
A. Project Goals and Objectives
The project is devoted to the development of a virtual reality simulator. The simulator includes a set of different scenarios related to safety and procedures performed during an emergency at a certain enterprise, as well as a module that allows to assess the level of human preparation for such situations. To make this assessment, a heart rate sensor is used. Based on the data obtained from it, the level of stress experienced by the user will be assessed. It is also planned to consider the number of deviations from the optimal route, the number of errors and the time of the simulation.
The purpose of such simulator is not only memorizing the required operations, but preparing a person for stress, which accompanies any emergency. To get an excellent assessment, the user needs not only to successfully cope with a set of actions in the allotted time, but also to remain calm throughout the simulation.
It is expected to receive a ready-made virtual simulator which can be used during safety briefings. Based on the data on the condition of the subjects, it will be established whether the simulator has a sufficient degree of immersion. This will determine how effective such a simulator is compared to the currently used training methods.
B. Literature Review
As part of this work, an analysis of virtual reality technologies was conducted, including an overview of modern ways to implement them. The possibility and relevance of creating VR simulators have been studied, and information about what has already been achieved in this area by other researchers has been collected.
Virtual reality is an artificial environment that is created with software and presented to the user in such a way that the user suspends belief and accepts it as a real environment. On a computer, virtual reality is primarily experienced through two of the five senses: sight and sound.
Objects modeled in virtual reality tend to behave close to the usual behavior of similar objects in material reality. The user can interact with them in accordance with the real laws of physics, such as, for example, gravity, the properties of liquids, collision with objects.
The technical basis for VR is computer modeling and simulation technologies, which, thanks to accelerated three-dimensional visualization, allow to realistically recreate movement on the screen. There is a specific set of hardware that is required to interact with the VR, including a screen that displays the image and pointing devices such as joysticks.
The main feature inherent in all VR models is the illusion of direct presence in a computer-simulated environment, which is called remote presence. The sense of remote presence depends less on how natural the images of the environment look than on how realistic the movement is reproduced and how convincingly the VR model reacts when interacting with the user .
The following set of basic properties inherent in VR models is defined:
• plausibility: the model creates a sense of reality for the user.
• Interactivity: it provides the ability to interact with the environment.
• Machine generation: it is based on powerful hardware.
• Accessibility for detailed study: it allows the user to explore the virtual world in detail.
• Presence effect: it creates a sense of the user’s direct presence in the simulated environment .
There are several studies that compare the process of storing information and navigating in space in virtual reality and when working with applications that do not use these technologies. Since both processes are important for the simulator being developed, the results of such studies have been studied.
It has been experimentally established that a certain procedure learned in the virtual world can be reproduced in the real world just as quickly and with the same level of accuracy. Thus, the knowledge obtained during training in the virtual world can be transferred to the real world, which is of great importance for the use of VR technologies in education, since knowledge that cannot be transferred is, in fact, absolutely useless .
The concept of immersion is key feature of VR systems, and its level is considered the main criterion for evaluating the quality of a particular application. In the case of habitual interaction of a person with a computer, the appearance of a presence effect is impossible . At the same time, it was found out that this effect positively affects both the process of memorization and the speed of recalling information when necessary .
The following experiment was performed: participants were asked to study two mock-ups of virtual buildings: one using a VR headset, and the other using only a personal computer. After familiarizing themselves with both buildings, each user was placed in the lobby of that particular building and told to go to each of the five named rooms and then return to the lobby. It was found that participants who worked with headsets reached the goal faster, made fewer irrational movements, and more accurately estimated the distance compared to those who used a computer .
The results of another experiment are also interesting for the development of a simulator related to teaching evacuation procedures. Its participants studied the complex layout of the building in the following ways: using 2D maps, physically passing through a real building, as well as through a virtual representation of the building in 3D, created on the Doom II game engine. The study showed that in the case of a single-story building, users who both studied in the real world and worked with a virtual environment showed comparable results. Differences were recorded in the case of working with two-story buildings. If participants were at the same point as during the training, they could navigate the environment in the same way as those who were trained in real-world conditions. However, participants were prone to disorientation if their initial position in the learning process and during testing was different . Therefore, it is necessary to take this effect into account and when developing the simulator, pay attention to the fact that each time the route starts from different possible points.
With all these advantages, it should be borne in mind that at the moment virtual reality systems have certain disadvantages. A significant problem is that being in a virtual environment can cause motion sickness. A possible explanation for this side effect, in turn, is that users suffer from sensory conflicts when they visually observe a convincing representation of movement in space while their body remains stationary. In the works of the end of the last century devoted to virtual reality technologies, it was noted that a significant number of users felt symptoms such as dizziness, nausea and headache while working with the virtual environment.  However, recent paper noted that only 5% of the study participants felt motion sickness. The authors attribute this to the fact that a high-quality system of moving in space was used . Thus, at the moment, the negative effect can be reduced by carefully considering the user’s movement system in the virtual world.
This section describes examples of development of simulators related to safety and emergency behavior.
S. Smith and E. Eriksson encouraged the participants to determine the potential fire hazard. After that, the participants were invited to participate in evacuation practice. The results of the work reflect that after passing the training, a significant increase in knowledge of fire safety was recorded .
L. Cittaro and F. Buttussi presented in a virtual environment the entire set of emergencies possible in the field of aviation, from situations involving high turbulence to evacuation to a life raft in case of forced landing at sea. Participants in the experiment worked with ten different scenarios describing the correct order of actions for each of the situations. The results of training using the VR application were compared with the standard method of using a special card, where information is presented graphically and using text (such materials are provided to each passenger on board the aircraft). It was found that those who studied in VR demonstrated a higher level of knowledge both immediately after training and a week later. This is an important result, because people should remember the order of necessary actions in case of an emergency for as long as possible, since it is impossible to predict at what point in time an emergency situation will occur — immediately after the next training was held or a few months later .
Lee suggested using a virtual reality application as a tool to teach how to behave during an earthquake. On average, participants who worked with this application performed better than those who used training videos or security guides .
As part of the experiment, J. Smith and B. Veitch compared the results of training oil platform personnel through lectures and using VR applications. Both training programs were used to familiarize new employees with basic operating procedures and evacuation rules. The effectiveness of training methods was measured by a combination of the time spent on training and the effectiveness achieved by each of the training groups. The data obtained showed that the use of the virtual environment was more effective .
In all the works reviewed, it is noted that virtual reality applications provide user with knowledge about the evacuation procedure and the necessary actions in case of an emergency.
C. Research Methods and Models
The following hypotheses are put forward:
• Personnel training with the help of virtual reality applications is effective. Knowledge is successfully assimilated and stored with the passage of time.
• The application provides a sufficient level of immersion. The participants of the experiment demonstrate involvement in the process and perceive the virtual world “almost as real”.
• The developed simulator successfully assesses the level of preparation of a person for a stressful situation.
An experimental research method is used to test the hypotheses. This allows to observe the phenomenon under study in specified conditions, which allows to follow the course of the phenomenon and reproduce it repeatedly when these conditions are repeated.
A significant stage of preparation for the experiment is the development of a virtual reality application, which will serve as a model of emergency. The participants of the experiment will have to work with it.
At the moment, virtual reality systems are presented in three variations: the first involve the use of personal computers to run the applications (while computers must meet high hardware requirements), the second require the use of smartphones, while the third work autonomously .
It was decided to use headsets that work with a PC, as they can offer higher quality and realistic level of graphics, as well as the ability to create more complex virtual worlds. This is of key importance for the simulator, as it significantly increases level of immersion.
The developed virtual model has a certain number of requirements related to the fact that it must provide knowledge about the object under study. This means that it must correspond to the characteristics of the studied activity in the real world. In addition, the model should provide the possibility of repetition of the experiment. For this purpose, the method of random event generation is used. After each new start of the simulation, the user finds himself at a random point in the virtual space (in a randomly selected room), after which the location of the threat and the level of its danger are also randomly selected.
The actions of the user working with the application are automatically evaluated using a special module. Assessment consists of two components: assessment of behavior and stress level.
The number of mistakes is taken into account to assess the behavior:
The total number of mistakes (M) is composed of: m1 — number of mistakes of the first type, m2 — number of mistakes of the second type, m3 — number of mistakes of the third type. The k coefficients reflect the severity of each type of error in each scene.
Errors of the first type relate to choosing the wrong direction. For example, during evacuation, a ladder located at one end of the corridor should be used to descend from the second floor to the first, but the user moves to the opposite end. To identify such errors, the simulator uses a system of control points that the user passes on the way to the goal. An error is counted if the user crosses a point that is not on the optimal escape route.
Errors of the second type are associated with an incorrect assessment of the level of danger. For example, the use of means capable of eliminating the emergency of smaller scale (a small fire can be eliminated with the help of fire extinguishers), in the situation when it is necessary to evacuate the building. Such an error is counted if the user interacts with an unnecessary object.
The error of the third type is associated with the incorrect use of necessary items (for example, with the already mentioned fire extinguisher).
To assess the level of stress in the simulator, heart rate data obtained from the connected heart rate monitor is used. In extreme conditions there are changes in heart rate. During a stressful situation, the heart rate may reach 150–180 beats per minute. When the application is started, the base pulse value is recorded within a minute. This allows to determine what value can be considered as normal for this particular person. The value is recorded while the user rests without movement. Later, when the user is working with the simulator, the user’s heart rate is compared with the previously saved base value. If the readings differ by more than 30% up, this is counted as a stress-related error.
The level of stress is taken into account due to the fact that a high level of tension negatively affects the results of user’s activities not only in a virtual reality application, but also in real life. Overcoming stress is possible if the situation becomes familiar.
If civilians who do not have special training find themselves in dangerous conditions, this usually causes psychological and emotional tension and stress.
Operational tension occurs if the motor apparatus and mental abilities of a person are subject to increased requirements and there is no danger that would cause a person to worry in advance. Operational tension is gradually increasing.
Emotional tension occurs under the influence of factors that cause such negative emotions as anxiety, fear, and others. This tension is often associated with inconsistency between the objective significance of the situation and its subjective assessment, with a large proportion of the body’s defense responses.
This division of types of tension is conditional, since any type of human activity is always somehow associated with emotions, and under certain conditions, operational tension can develop into emotional, so the most characteristic is mixed tension .
When there is a strong emotional tension associated with working in extreme conditions, there are fluctuations in performance (the correct sequence of operations is violated, the pace of work slows down, and the number of errors increases). Most often, such a decrease in performance is observed with a low level of training. Conversely, people’s performance increases if they have previously studied the environment in which the activity will take place, the features of the activity, tools and means of labor. Thus, it is known that ignorance of the situation causes, as a rule, much greater mental stress than a certain signal of danger .
Loss of flexibility, especially in performing complex, coordinated movements, is a distinctive feature of behavior in extreme conditions. At the same time, formulaic, stereotypical movements can flow better and even tend to be automated.
At the psychological level, negative phenomena observed in extreme conditions are expressed in general disorganization of behavior, inadequate responses to sudden unexpected stimuli, inhibition of previous skills (the more difficult the action is), reduced performance, difficulties in distributing and switching attention, narrowing its scope, errors of perception, memory lapses, the appearance of unnecessary, unjustified and impulsive actions, easy distraction, a sense of confusion and inability to focus on the performed activity, etc. .
Thus, it is advisable to use an evaluation module that would consider how the user follows the instructions and what level of stress he or she is experiencing.
The simulator operates in two modes. The first one is “exam”, which involves a single passage of the scenario with the assessment. The second one is training mode which the user can pass several times, receiving advice and information about what mistakes he made, as well as information about when the stress exceeded the allowable level.
After passing an exam, the user receives a behavioral assessment based on the number of errors of three types. The behavioral assessment is counted as “passed” if the parameter M does not exceed 3. Stress-related errors are not allowed — the simulation is considered passed only when no one is recorded. It is considered that the exam is passed if both parameters take the value “passed”.
In the ‘training” mode, the simulator is supposed to adapt to the user’s behavior. If the trainee demonstrates a high level of stress, and his condition does not normalize over time, the simulator first reduces the level of complexity, and then completely suspends the passage of the simulation until the heart rate comes to normal. In training mode, data on the number of errors are displayed on the screen.
When the heart rate increases, the current heart rate values are displayed. If the values do not normalize soon, the script execution stops until they return to normal. In the exam mode, no data is displayed in the process of passing the simulation. After completing the work, the user receives information about the number of mistakes and the corresponding verdict on whether the passage of the exam is counted.
The method used to prepare a person for an emergency is based on a model of overcoming stress, called the result model. The main essence of this model is that the quality of overcoming stress is estimated by its impact on the result, which has a certain value for human life and activity. In other words, overcoming stress can be considered successful if a person successfully copes with the task.
That is, it is assumed that the user is ready to counteract the emergency in real life if he copes with the simulation without errors.
It was decided to use the Unreal Engine in order to develop the simulator, as it has several advantages:
• It is more suitable for three-dimensional PC projects because it supports projects with complex mechanics.
• Due to its ability to work with materials and postprocessing, it provides the ability to achieve more realistic graphics, which is important for creating a presence effect.
• The Blueprints system makes it easy to implement basic mechanics, which simplifies the development process and allows the developer to focus on more complex elements .
Immediately after selecting the modeling object, it was necessary to create locations using the Unreal Engine.
To do this, a number of three-dimensional models of the necessary objects were developed in 3ds Max. Moreover, several models from free libraries were used.
The resulting models were imported to Unreal Engine, and then the necessary configuration of their properties was performed. The location’s geometry was created that corresponds to the pre-planned layout of the premises. Then the imported models were placed in the resulting virtual rooms, resulting in a realistic simulation of the hotel building.
The simulator under development has a fairly high level of graphics, comparable to that currently used in games developed for virtual reality systems. This makes the image on the screen look realistic.
A realistic level of graphics should have a positive impact on the level of immersion, help the user feel part of the virtual environment, perceive this world “almost as real”. This should lead to the fact that the occurrence of an emergency in virtual world will cause stress that is minimally different from what the user would have experienced in a similar situation in real life.
In order to avoid repeating the same scenario, the principle of generating a random point is used. Each time a “fire” occurs at a new point in the virtual world in accordance with the specified rules.
The following principle is applied: when starting the simulator, the user finds himself at a randomly selected location every time. After some time, at a random point located within a given radius from its location, a “fire” occurs, gradually growing.
The elements that enable the evaluation module to work were configured. A system of control points was added in each scene. The simulator considers the user’s intersection of points that do not lie on the correct route set during the development of the episode. If the user crosses more than two points in a row, this is counted as an error. The intersection of only one point is considered acceptable, since it may occur accidentally when working with virtual reality, especially if the person has little experience with such applications.
Objects of the scene that the user should not interact with have the appropriate label. Interaction with an object with this label is counted as an error.
Experiments were conducted to test the correctness of hypotheses. They can be divided into several stages:
• Recording background values using the device “Psychophysiologist”.
• Working with the virtual simulator.
• Recording of data after passing the simulation using the device “Psychophysiologist”.
The method of variational cardiointervalometry with a duration of 128 cardiocycles was chosen for recording the background values. It is used to assess the functional state of the autonomic nervous system of the subject according to the parameters of the rhythm of his heart activity, as well as to assess the overall functional state of the person. Vegetative maintenance of activity reflects the ability to maintain an optimal level of functioning of the ANS in various stress situations.
To do this, the heart rate variability (HRV) is determined — the variability in the duration of RR intervals of consecutive heart cycles over certain periods of time and the severity of heart rate fluctuations in relation to its average level.
5 test groups of 20 people of different ages participated in the experiment.
Each member of the group received a standard safety briefing to refresh their knowledge. They were introduced to the plan of the building modeled in the virtual simulator.
Users who have not previously worked with virtual reality applications were given the opportunity to work with a training scene, in which they were able to learn how to interact with the virtual world.
Background data were collected using the device “Psychophysiologist”, after which the participants alternately worked with the simulator. Data about each pass were saved (the time of passing and the number of errors).
After completion, the data on the functional state of all participants was collected again. Participants were interviewed in order to identify errors or shortcomings in the simulator.
The survey consisted of the following questions:
1. How convenient was working with the simulator? (0–10 points)
2. Is the selected system of movement convenient? (yes/no)
3. Did you experience any errors that are not related to the assessment module (for example, the inability to move to the desired point, interact with the desired subject, and others)?
4. Were there any errors in the evaluation module (for example, actions that you did not perform were counted)?
5. How much do you think the assessment corresponds to your level of knowledge? (0–10 points)
6. Do you feel that knowledge of safety measures has increased after working with the simulator? (yes/no)
7. Do you think this method of acquiring knowledge is more effective than classical methods? (yes/no/equally effective)
Respondents rated the convenience of working with the simulator by 8.4 points on average. The chosen moving system is considered convenient because it does not cause the user to feel “motion sickness”, which can occur when working with virtual reality applications, and allows you to successfully work with the scene.
Based on the answers to questions 3 and 4, several corrections were made to the simulator. Participants in the first group of users mostly encountered minor bugs. Based on the information received from them, some changes were made to the scenes — for example, a bug was fixed that allowed you to move through the wall in one of the scenes, and interactivity was added to several items that were skipped during development. Errors related to the evaluation module were mainly related to the checkpoint system. Several of them were mistakenly marked as invalid. In addition, the position of a few points has been changed in order to more accurately account for the user’s movement.
According to the respondents, the score corresponds to expectations by 7.6 points. In addition, since each participant also took a written test after training, it was possible to compare not only the “expected” score, but also the actual score received after passing the test. The assessment coincided in 82% of cases. In all other cases, the simulator score was lower than during testing.
73% of respondents confirmed the increase in safety knowledge. 68% consider the use of the simulator more effective than traditional methods of obtaining knowledge, 23% — just as effective.
The high level of stress detected by the device “Psychophysiologist” was counted as an error in 93% of cases, which indicates that the stress assessment module copes with the task.
In order to assess how well the acquired knowledge persists over time, the first group of participants was re-invited to work with the simulator a month after the first attempt.
While working with the app again, users showed better results. It means that the knowledge they gained is successfully saved even after a month.
The number of users who successfully completed the task has increased, while the number of errors, on the contrary, has decreased. The stress level also became lower.
Table 1 shows the results of this group during the first and second work with the simulator.
Table 1. The Results of the User Experience
|Condition||Number of participants|
|Total number of participants||20|
|The number of users that have completed the simulation without errors (M < 3 and S = 0)||6; 11|
|The number of users who have completed the simulation with errors of the same type||5 (3 with S = 1, 2 with M > 3);|
3 (2 with S = 1, 1 with M > 3)
|Number of users who failed the task||9; 6|
|Users with significant stress||13; 8|
|Users with unsignificant change of functional state||7; 12|
Virtual reality application with a module for evaluating user behavior was successfully developed.
After the module was developed, experiments were conducted in order to evaluate the quality of the work. The tests confirm that the application is working properly, including the correct operation of the evaluation module. Repeated tests with the same people, conducted some time after the first attempt, also demonstrate that knowledge is successfully preserved over time.
The presented results confirm that:
• the application has a sufficient level of immersion to make the virtual environment feel “almost real” and evoke a reaction from the user;
• assessment module successfully finds mistakes and the occurrence of stress.
In a study, 3 main groups of potential consumers in the market of VR technology were identified:
1. Ordinary users who currently use VR systems mainly for entertainment, but there is a growing interest in applications in the field of health, commerce and education.
2. Representatives of government agencies that are showing increasing interest in VR simulators, as well as in applications that can contribute to attracting the attention of tourists to countries and their regions.
3. Representatives of business, who expect to improve the production process with the help of VR technologies, to provide a new type of service or means for communication with consumers .
The developed application can be of interest to representatives of the last two groups, it can be used in public institutions and commercial firms for training staff. With its help, it is possible to raise the awareness about the necessary actions in emergencies, which can help to reduce the number of victims in various incidents. It is convenient because it provides greater efficiency compared with lectures, and it does not require such time-consuming actions, such as evacuation.
This research was supported by ITMO University. The author thanks Liubov S. Lisitsyna for assistance with the development and for comments that greatly improved the paper.
 Bartle, R. A. Designing Virtual Worlds. New Riders, 2004
 Feng, Zh., González, М., Amor, R. Lovreglio, R. and Cabrera-Guerrero, G. Immersive Virtual Reality Serious Games for Evacuation Training and Research: A Systematic Literature Review. Computers & Education, 2018, no. 127, pp. 252–266
 Chen, C. J. Virtual Reality (VR)-Based Learning Environment: Design, Develop, Evaluate, LAP LAMBERT Academic Publishing, 2010
 Fuchs, P. Virtual Reality Headsets A Theoretical and Pragmatic Approach, CRC Press, 2017
 Ragan, E. D. The Effects of Higher Levels of Immersion on Procedure Memorization Performance and Implications for Educational Virtual Environments, Presence, 2010, no. 19(6), pp. 527–543
 Slater, M. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments, in Philosophical Transactions of the Royal Society B: Biological Sciences, 2009, no. 364, pp. 3549–3557
 Jerald, J. The VR Book: Human-Centered Design for Virtual Reality, Morgan & Claypool Publishers, 2015
 Ruddle, R., Payne, S., and Jones, D. Navigating Large-Scale Virtual Environments: What Differences Occur Between Helmet-Mounted and Desk-Top Displays? Presence: Teleoperators and Virtual Environments, 1999, no. 8, pp. 157–168
 Richardson, A. E., Montello, D. R., and Hegarty, M. Spatial knowledge acquisition from maps and from navigation in real and virtual environments, Memory & Cognition, 1999, no. 27(4), pp. 741–750
 Lovreglio, R., Gonzalez, V., Feng, Z., Amor, R., Spearpoint, M., Thomas, J., Trotter, M., and Sacks, R. Prototyping Virtual Reality Serious Games for Building Earthquake Preparedness: The Auckland City Hospital Case Study, 2018
 Smith, S., and Ericson, E. Using immersive game-based virtual reality to teach fire-safety skills to children, Virtual Reality, 2017, no. 13(2), pp. 87–99
 Chittaro, L., Buttussi, F., and Zangrando, N. Desktop virtual reality for emergency preparedness, Proceedings of the 20th ACM Symposium on virtual reality software and technology, 2014, pp. 141–150
 Li, C., Liang, W., Quigley, C., Zhao, Y., and Earthquake, L. Yu. Safety Training through Virtual Drills, in IEEE Transactions on Visualization and Computer Graphics, 2017, no. 23(4), pp. 1275–1284
 Smith, J., and Veitch, B. A Better Way to Tr ain Personnel to Be Safe in Emergencies, in ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part B: Mechanical Engineering, 2018, no. 5(1)
 Nite, S. Virtual Reality Insider: Guidebook for the VR Industry, in New Dimension Entertainment, 2014
 Connolly, T. M., Boyle, E. A., Boyle, J. M., MacArthur, E., and Hainey, T. A systematic literature review of empirical evidence on computer games and serious games, in Computers & Education, 2012, no. 59(2), pp. 661–686
 Connolly, T., Stansfield, M., and Boyle, L. Current Practices in Serious GameResearch: A Review from a Learning Outcomes Perspective, Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices, 2009, pp. 232–250
 Wouters, P., Van Nimwegen, C., Van Oostendorp, H., and Van Der Spek, E. D. A meta-analysis of the cognitive and motivational effects of serious games, in Journal of Educational Psychology, 2013, no. 105(2), pp. 249–265
 Parisi, T. Learning Virtual Reality: Developing Immersive Experiences and Applications for Desktop, Web, and Mobile, O’Reilly Media, Incorporated, 2015
 Osarek, J. Virtual Reality Analytics: How VR and AR change Business Intelligence, Gordon’s Arcade, 2016