2024 |
Chiossi, Francesco; Ou, Changkun; Mayer, Sven 2024. Abstract | Links | Tags: DSI-VR300, VR @conference{chiossi2024optimizing,Physiologically-adaptive Virtual Reality systems dynamically adjust virtual content based on users’ physiological signals to enhance interaction and achieve specific goals. However, as different users’ cognitive states may underlie multivariate physiological patterns, adaptive systems necessitate a multimodal evaluation to investigate the relationship between input physiological features and target states for efficient user modeling. Here, we investigated a multimodal dataset (EEG, ECG, and EDA) while interacting with two different adaptive systems adjusting the environmental visual complexity based on EDA. Increased visual complexity led to increased alpha power and alpha-theta ratio, reflecting increased mental fatigue and workload. At the same time, EDA exhibited distinct dynamics with increased tonic and phasic components. Integrating multimodal physiological measures for adaptation evaluation enlarges our understanding of the impact of system adaptation on users’ physiology and allows us to account for it and improve adaptive system design and optimization algorithms. |
Klee, Daniel; Memmott, Tab; Oken, Barry In: Signals, vol. 5, no. 1, pp. 18–39, 2024. Abstract | Links | Tags: BCI, DSI-VR300 @article{klee2024effect,Brain responses to discrete stimuli are modulated when multiple stimuli are presented in sequence. These alterations are especially pronounced when the time course of an evoked response overlaps with responses to subsequent stimuli, such as in a rapid serial visual presentation (RSVP) paradigm used to control a brain–computer interface (BCI). The present study explored whether the measurement or classification of select brain responses during RSVP would improve through application of an established technique for dealing with overlapping stimulus presentations, known as irregular or “jittered” stimulus onset interval (SOI). EEG data were collected from 24 healthy adult participants across multiple rounds of RSVP calibration and copy phrase tasks with varying degrees of SOI jitter. Analyses measured three separate brain signals sensitive to attention: N200, P300, and occipitoparietal alpha attenuation. Presentation jitter visibly reduced intrusion of the SSVEP, but in general, it did not positively or negatively affect attention effects, classification, or system performance. Though it remains unclear whether stimulus overlap is detrimental to BCI performance overall, the present study demonstrates that single-trial classification approaches may be resilient to rhythmic intrusions like SSVEP that appear in the averaged EEG. |
2023 |
Chiossi, Francesco; Turgut, Yagiz; Welsch, Robin; Mayer, Sven Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual Reality Journal Article In: Proc. ACM Hum.-Comput. Interact, vol. 7, 2023. Abstract | Links | Tags: DSI-VR300, VR @article{chiossi2023adapting,Biocybernetic loops encompass users’ state detection and system adaptation based on physiological signals. Current adaptive systems limit the adaptation to task features such as task difficulty or multitasking demands. However, virtual reality allows the manipulation of task-irrelevant elements in the environment. We present a physiologically adaptive system that adjusts the virtual environment based on physiological arousal, i.e., electrodermal activity. We conducted a user study with our adaptive system in social virtual reality to verify improved performance. Here, participants completed an n back task, and we adapted the visual complexity of the environment by changing the number of non-player characters. Our results show that an adaptive virtual reality can control users’ comfort, performance, and workload by adapting the visual complexity based on physiological arousal. Thus, our physiologically adaptive system improves task performance and perceived workload. Finally, we embed our findings in physiological computing and discuss applications in various scenarios |
2022 |
Klee, Daniel; Memmott, Tab; Smedemark-Margulies, Niklas; Celik, Basak; Erdogmus, Deniz; Oken, Barry S Target-Related Alpha Attenuation in a Brain-Computer Interface Rapid Serial Visual Presentation Calibration Journal Article In: Frontiers in Human Neuroscience, vol. 16, 2022. Abstract | Links | Tags: BCI, DSI-VR300 @article{klee2022target,This study evaluated the feasibility of using occipitoparietal alpha activity to drive target/non-target classification in a brain-computer interface (BCI) for communication. EEG data were collected from 12 participants who completed BCI Rapid Serial Visual Presentation (RSVP) calibrations at two different presentation rates: 1 and 4 Hz. Attention-related changes in posterior alpha activity were compared to two event-related potentials (ERPs): N200 and P300. Machine learning approaches evaluated target/non-target classification accuracy using alpha activity. Results indicated significant alpha attenuation following target letters at both 1 and 4 Hz presentation rates, though this effect was significantly reduced in the 4 Hz condition. Target-related alpha attenuation was not correlated with coincident N200 or P300 target effects. Classification using posterior alpha activity was above chance and benefitted from individualized tuning procedures. These findings suggest that target-related posterior alpha attenuation is detectable in a BCI RSVP calibration and that this signal could be leveraged in machine learning algorithms used for RSVP or comparable attention-based BCI paradigms. |
2021 |
McLaughlin, Deirdre; Klee, Daniel; Memmott, Tab; Peters, Betts; Wiedrick, Jack; Fried-Oken, Melanie; Oken, Barry Methodology and feasibility of neurofeedback to improve visual attention to letters in mild Alzheimer's disease Journal Article In: Human-Computer Interaction, 2021. Abstract | Links | Tags: DSI-VR300, Neurofeedback @article{mclaughlin2021methodology,Brain computer interfaces systems are controlled by users through neurophysiological input for a variety of applications including communication, environmental control, motor rehabilitation, and cognitive training. Although individuals with severe speech and physical impairment are the primary users of this technology, BCIs have emerged as a potential tool for broader populations, especially with regards to delivering cognitive training or interventions with neurofeedback. The goal of this study was to investigate the feasibility of using a BCI system with neurofeedback as an intervention for people with mild Alzheimer's disease. The study focused on visual attention and language since ad is often associated with functional impairments in language and reading. The study enrolled five adults with mild ad in a nine to thirteen week BCI EEG based neurofeedback intervention to improve attention and reading skills. Two participants completed intervention entirely. The remaining three participants could not complete the intervention phase because of restrictions related to covid. Pre and post assessment measures were used to assess reliability of outcome measures and generalization of treatment to functional reading, processing speed, attention, and working memory skills. Participants demonstrated steady improvement in most cognitive measures across experimental phases, although there was not a significant effect of NFB on most measures of attention. One subject demonstrated significantly significant improvement in letter cancellation during NFB. All participants with mild AD learned to operate a BCI system with training. Results have broad implications for the design and use of bci systems for participants with cognitive impairment. Preliminary evidence justifies implementing NFB-based cognitive measures in AD. |
Kim, Soram; Lee, Seungyun; Kang, Hyunsuk; Kim, Sion; Ahn, Minkyu P300 Brain--Computer Interface-Based Drone Control in Virtual and Augmented Reality Journal Article In: Sensors, vol. 21, no. 17, pp. 5765, 2021. Abstract | Links | Tags: BCI, DSI-VR300, VR @article{kim2021p300,Since the emergence of head-mounted displays (HMDs), researchers have attempted to introduce virtual and augmented reality (VR, AR) in brain–computer interface (BCI) studies. However, there is a lack of studies that incorporate both AR and VR to compare the performance in the two environments. Therefore, it is necessary to develop a BCI application that can be used in both VR and AR to allow BCI performance to be compared in the two environments. In this study, we developed an opensource-based drone control application using P300-based BCI, which can be used in both VR and AR. Twenty healthy subjects participated in the experiment with this application. They were asked to control the drone in two environments and filled out questionnaires before and after the experiment. We found no significant (p > 0.05) difference in online performance (classification accuracy and amplitude/latency of P300 component) and user experience (satisfaction about time length, program, environment, interest, difficulty, immersion, and feeling of self-control) between VR and AR. This indicates that the P300 BCI paradigm is relatively reliable and may work well in various situations |
2018 |
Pereira, Arnaldo; Padden, Dereck; Jantz, Jay; Lin, Kate; Alcaide-Aguirre, Ramses Cross-Subject EEG Event-Related Potential Classification for Brain-Computer Interfaces Using Residual Networks Journal Article In: 2018. Abstract | Links | Tags: BCI, DSI-VR300 @article{pereira2018cross,EEG event-related potentials, and the P300 signal in particular, are promising modalities for brain-computer interfaces (BCI). But the nonstationarity of EEG signals and their differences across individuals have made it difficult to implement classifiers that can determine user intent without having to be retrained or calibrated for each new user and sometimes even each session. This is a major impediment to the development of consumer BCI. Recently, the EEG BCI literature has begun to apply convolutional neural networks (CNNs) for classification, but experiments have largely been limited to training and testing on single subjects. In this paper, we report a study in which EEG data were recorded from 66 subjects in a visual oddball task in virtual reality. Using wide residual networks (WideResNets), we obtain state-of-the-art performance on a test set composed of data from all 66 subjects together. Additionally, a minimal preprocessing stream to convert EEG data into square images for CNN input while adding regularization is presented and shown to be viable. This study also provides some guidance on network architecture parameters based on experiments with different models. Our results show that it may possible with enough data to train a classifier for EEG-based BCIs that can generalize across individuals without the need for individual training or calibration. |
2024 |
Chiossi, Francesco; Ou, Changkun; Mayer, Sven 2024. @conference{chiossi2024optimizing,Physiologically-adaptive Virtual Reality systems dynamically adjust virtual content based on users’ physiological signals to enhance interaction and achieve specific goals. However, as different users’ cognitive states may underlie multivariate physiological patterns, adaptive systems necessitate a multimodal evaluation to investigate the relationship between input physiological features and target states for efficient user modeling. Here, we investigated a multimodal dataset (EEG, ECG, and EDA) while interacting with two different adaptive systems adjusting the environmental visual complexity based on EDA. Increased visual complexity led to increased alpha power and alpha-theta ratio, reflecting increased mental fatigue and workload. At the same time, EDA exhibited distinct dynamics with increased tonic and phasic components. Integrating multimodal physiological measures for adaptation evaluation enlarges our understanding of the impact of system adaptation on users’ physiology and allows us to account for it and improve adaptive system design and optimization algorithms. |
Klee, Daniel; Memmott, Tab; Oken, Barry In: Signals, vol. 5, no. 1, pp. 18–39, 2024. @article{klee2024effect,Brain responses to discrete stimuli are modulated when multiple stimuli are presented in sequence. These alterations are especially pronounced when the time course of an evoked response overlaps with responses to subsequent stimuli, such as in a rapid serial visual presentation (RSVP) paradigm used to control a brain–computer interface (BCI). The present study explored whether the measurement or classification of select brain responses during RSVP would improve through application of an established technique for dealing with overlapping stimulus presentations, known as irregular or “jittered” stimulus onset interval (SOI). EEG data were collected from 24 healthy adult participants across multiple rounds of RSVP calibration and copy phrase tasks with varying degrees of SOI jitter. Analyses measured three separate brain signals sensitive to attention: N200, P300, and occipitoparietal alpha attenuation. Presentation jitter visibly reduced intrusion of the SSVEP, but in general, it did not positively or negatively affect attention effects, classification, or system performance. Though it remains unclear whether stimulus overlap is detrimental to BCI performance overall, the present study demonstrates that single-trial classification approaches may be resilient to rhythmic intrusions like SSVEP that appear in the averaged EEG. |
2023 |
Chiossi, Francesco; Turgut, Yagiz; Welsch, Robin; Mayer, Sven Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual Reality Journal Article In: Proc. ACM Hum.-Comput. Interact, vol. 7, 2023. @article{chiossi2023adapting,Biocybernetic loops encompass users’ state detection and system adaptation based on physiological signals. Current adaptive systems limit the adaptation to task features such as task difficulty or multitasking demands. However, virtual reality allows the manipulation of task-irrelevant elements in the environment. We present a physiologically adaptive system that adjusts the virtual environment based on physiological arousal, i.e., electrodermal activity. We conducted a user study with our adaptive system in social virtual reality to verify improved performance. Here, participants completed an n back task, and we adapted the visual complexity of the environment by changing the number of non-player characters. Our results show that an adaptive virtual reality can control users’ comfort, performance, and workload by adapting the visual complexity based on physiological arousal. Thus, our physiologically adaptive system improves task performance and perceived workload. Finally, we embed our findings in physiological computing and discuss applications in various scenarios |
2022 |
Klee, Daniel; Memmott, Tab; Smedemark-Margulies, Niklas; Celik, Basak; Erdogmus, Deniz; Oken, Barry S Target-Related Alpha Attenuation in a Brain-Computer Interface Rapid Serial Visual Presentation Calibration Journal Article In: Frontiers in Human Neuroscience, vol. 16, 2022. @article{klee2022target,This study evaluated the feasibility of using occipitoparietal alpha activity to drive target/non-target classification in a brain-computer interface (BCI) for communication. EEG data were collected from 12 participants who completed BCI Rapid Serial Visual Presentation (RSVP) calibrations at two different presentation rates: 1 and 4 Hz. Attention-related changes in posterior alpha activity were compared to two event-related potentials (ERPs): N200 and P300. Machine learning approaches evaluated target/non-target classification accuracy using alpha activity. Results indicated significant alpha attenuation following target letters at both 1 and 4 Hz presentation rates, though this effect was significantly reduced in the 4 Hz condition. Target-related alpha attenuation was not correlated with coincident N200 or P300 target effects. Classification using posterior alpha activity was above chance and benefitted from individualized tuning procedures. These findings suggest that target-related posterior alpha attenuation is detectable in a BCI RSVP calibration and that this signal could be leveraged in machine learning algorithms used for RSVP or comparable attention-based BCI paradigms. |
2021 |
McLaughlin, Deirdre; Klee, Daniel; Memmott, Tab; Peters, Betts; Wiedrick, Jack; Fried-Oken, Melanie; Oken, Barry Methodology and feasibility of neurofeedback to improve visual attention to letters in mild Alzheimer's disease Journal Article In: Human-Computer Interaction, 2021. @article{mclaughlin2021methodology,Brain computer interfaces systems are controlled by users through neurophysiological input for a variety of applications including communication, environmental control, motor rehabilitation, and cognitive training. Although individuals with severe speech and physical impairment are the primary users of this technology, BCIs have emerged as a potential tool for broader populations, especially with regards to delivering cognitive training or interventions with neurofeedback. The goal of this study was to investigate the feasibility of using a BCI system with neurofeedback as an intervention for people with mild Alzheimer's disease. The study focused on visual attention and language since ad is often associated with functional impairments in language and reading. The study enrolled five adults with mild ad in a nine to thirteen week BCI EEG based neurofeedback intervention to improve attention and reading skills. Two participants completed intervention entirely. The remaining three participants could not complete the intervention phase because of restrictions related to covid. Pre and post assessment measures were used to assess reliability of outcome measures and generalization of treatment to functional reading, processing speed, attention, and working memory skills. Participants demonstrated steady improvement in most cognitive measures across experimental phases, although there was not a significant effect of NFB on most measures of attention. One subject demonstrated significantly significant improvement in letter cancellation during NFB. All participants with mild AD learned to operate a BCI system with training. Results have broad implications for the design and use of bci systems for participants with cognitive impairment. Preliminary evidence justifies implementing NFB-based cognitive measures in AD. |
Kim, Soram; Lee, Seungyun; Kang, Hyunsuk; Kim, Sion; Ahn, Minkyu P300 Brain--Computer Interface-Based Drone Control in Virtual and Augmented Reality Journal Article In: Sensors, vol. 21, no. 17, pp. 5765, 2021. @article{kim2021p300,Since the emergence of head-mounted displays (HMDs), researchers have attempted to introduce virtual and augmented reality (VR, AR) in brain–computer interface (BCI) studies. However, there is a lack of studies that incorporate both AR and VR to compare the performance in the two environments. Therefore, it is necessary to develop a BCI application that can be used in both VR and AR to allow BCI performance to be compared in the two environments. In this study, we developed an opensource-based drone control application using P300-based BCI, which can be used in both VR and AR. Twenty healthy subjects participated in the experiment with this application. They were asked to control the drone in two environments and filled out questionnaires before and after the experiment. We found no significant (p > 0.05) difference in online performance (classification accuracy and amplitude/latency of P300 component) and user experience (satisfaction about time length, program, environment, interest, difficulty, immersion, and feeling of self-control) between VR and AR. This indicates that the P300 BCI paradigm is relatively reliable and may work well in various situations |
2018 |
Pereira, Arnaldo; Padden, Dereck; Jantz, Jay; Lin, Kate; Alcaide-Aguirre, Ramses Cross-Subject EEG Event-Related Potential Classification for Brain-Computer Interfaces Using Residual Networks Journal Article In: 2018. @article{pereira2018cross,EEG event-related potentials, and the P300 signal in particular, are promising modalities for brain-computer interfaces (BCI). But the nonstationarity of EEG signals and their differences across individuals have made it difficult to implement classifiers that can determine user intent without having to be retrained or calibrated for each new user and sometimes even each session. This is a major impediment to the development of consumer BCI. Recently, the EEG BCI literature has begun to apply convolutional neural networks (CNNs) for classification, but experiments have largely been limited to training and testing on single subjects. In this paper, we report a study in which EEG data were recorded from 66 subjects in a visual oddball task in virtual reality. Using wide residual networks (WideResNets), we obtain state-of-the-art performance on a test set composed of data from all 66 subjects together. Additionally, a minimal preprocessing stream to convert EEG data into square images for CNN input while adding regularization is presented and shown to be viable. This study also provides some guidance on network architecture parameters based on experiments with different models. Our results show that it may possible with enough data to train a classifier for EEG-based BCIs that can generalize across individuals without the need for individual training or calibration. |
Please fill out the form and provide a brief description of your application so we can help match you with products that will meet your specific needs.
Please fill out the form and provide a brief description of your application so we can help match you with products that will meet your specific needs.
