Remote Photonic Detection of Human Senses Using Secondary Speckle Patterns

doi:10.21203/rs.3.rs-724587/v1

Download PDF

Research Article

Remote Photonic Detection of Human Senses Using Secondary Speckle Patterns

https://doi.org/10.21203/rs.3.rs-724587/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 11 Jan, 2022

Read the published version in Scientific Reports →

You are reading this latest preprint version

Neural activity research has recently gained significant attention due to its association with sensory information and behavior control. However, current methods of brain activity sensing require expensive equipment and physical contact with the subject.

We propose a novel photonic-based method for remote detection of human senses. Physiological processes associated with hemodynamic activity due to activation of the cerebral cortex affected by different senses have been detected by remote monitoring of nano‐vibrations generated due to the transient blood flow to specific regions of the brain. We have found that combination of defocused, self‐interference random speckle patterns with a spatiotemporal analysis using Deep Neural Network (DNN) allows associating between the activated sense and the seemingly random speckle patterns.

Photonics/optics

Spectroscopy

sensory information

behavior control

hemodynamic activity

cerebral cortex

With its connection to sensory information and behavior control, neural activity research has recently received considerable attention. However, the current methods of brain activity sensing, however, involve expensive equipment and physical proximity with the subject. Sensation is a physical process involving sensory systems of a body responding to stimuli and providing data for perception¹. Human sensory systems are involved in daily activities, both consciously and unconsciously, and a study of the senses, especially their connection with brain activity, has been gaining in popularity in recent years. Brain activity analysis using electroencephalography (EEG) has received considerable attention^2–6, particularly in relation to the five basic human senses: sight², touch⁷, hearing^8,9, smell^10–13 and taste¹⁴. In addition to EEG, a number of optical techniques have also been employed for monitoring human brain activity using image contrast analysis^15,16 and cross-correlation based analysis of a laser speckle imaging¹⁷. While these methods mainly deal with a laser speckle image and its relation to temporal fluctuations, extracting semantic information from sensory activity is still lacking.

The temple area of the human head is located in front of the cerebral cortex and is not an optical quality surface. Therefore, when illuminated by a laser beam, the back scattered light forms secondary speckle patterns, which are possible to image by a digital camera with defocused optics. Analysis of temporal changes in the spatial distribution of the speckle patterns can be related to nano-vibrations in the illuminated surface due to the hemodynamic process associated with the transient flow of blood occurring during human brain activation¹⁷.

Speckle-based remote sensing has been used for the development of different biomedical applications, such as monitoring heart rate¹⁸, breathing¹⁹, blood pressure²⁰, blood oximetry²¹, blood coagulation^22,23, bone fractures²⁴, melanoma²⁵, and neural activity¹⁷. Prior methods for classifying speckle patterns used a single frame²⁶ or full video frame-by-frame²¹ obtained by averaging the model predictions on all frames of the video and providing a threshold for selecting the desired output. The prior classification methods, used a Convolutional Neural Network (CNN) to encode data from a single frame; however, speckle pattern data recorded over successive periods could be characterized as time series data²⁷. Due to this temporal dependency, we hypothesize that using a recurrent neural network architecture would provide improved results.

In this study, we propose a method for classification and detection of three basic senses: smell, taste, and hearing. The detection of senses is based on projecting a laser beam on a specific area of the human head associated with the cerebral cortex activity, (see Fig. 1) and analyzing the recorded speckle patterns using DNN. To ascertain reliability of our approach, the results were compared with a synchronized and simultaneously recorded EEG, known to be an effective method for detecting brain activity related to human senses². We trained an EEG-based DNN using the recorded EEG data and compared it to the results of the speckle-based DNN to find out conformity between the two approaches.

Our study could be of importance for patients suffering from stroke or cancer^28–31 and experiencing irregularities in their basic senses, especially in taste and smell. The frequently occurring loss of taste and smell associated with COVID-19 is also noteworthy^32,33. The olfactory neurons, which detect odors in the air and send signals to the brain, are one possible pathway for sensory loss³⁴. Predicting sensory loss with relative simplicity and remotely can contribute to the discovery of COVID-19 virus carriers as well.

The experimental setup, shown in Fig. 1, comprises a laser illuminating the temple area of a human head and a defocused, high-speed camera recording the reflected speckle patterns, an EEG device synchronized with the camera and a computer. Five healthy participants ranging in age from 29 to 73 were tested in the laboratory in two conditions: without and under stimulation of each sense.

Figure 2 shows samples of the recorded speckle pattern for each sense in a consecutive timely related order from left to right. Figure 2a displays speckle patterns related to the sense of smell. Figure 2a1 represents activated sense and Fig. 2a2 inactive sense. Figure 2b displays the sense of taste: Fig. 2b1 represents activated sense and Fig. 2b2 inactive sense. Figure 2c displays the sense of hearing: Fig. 2c1 represents activated sense and Fig. 2c2 inactive sense.

Classification of the speckle patterns and its association to a specific brain activity was carried out using DNN. Results of our validation are provided in Table 1 showing that our model achieved precision score of 92% and reached accuracy of 95%, while being faster and maintaining a high recall of 98%. Table 1 also shows comparison of our model with the previous two methods for back scattered laser speckle patterns classification^21,26 having accuracy of 82% and 89%.

Table 1

Speckle-based DNN comparison. Our method reaches an accuracy of 95% while maintaining a high recall of 98% in active-sense classification task. In an active-sense classification task, the full video approach achieved 89% accuracy and 93% precision. Our model takes 2 milliseconds per batch to infer, while the single-frame method takes 4 milliseconds and the full-video method takes 830 milliseconds.
Model	Classes	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	Inference Time (msec)
Single speckle frame Convnet	Sense	82	85	78	82	4
Single speckle frame Convnet	No Sense	82	79	83	82	4
Full video speckle Convnet	Sense	89	93	91	89	830
Full video speckle Convnet	No Sense	89	84	87	89	830
Our model	Sense	95	98	92	95	2
Our model	No Sense	95	92	98	95	2

Figure 3 presents the sensory recognition of our speckle based DNN for each subject individually and the average sensory recognition for each of the three senses, including value margins for all tested subjects. Figure 3a shows Speckle-based DNN predictions for the sense of smell, Fig. 3b for the sense of hearing, and Fig. 3c for the sense of taste. The blue columns in each of the figures representing the active sense predictions, while the orange columns representing the inactive sense.

The main benchmark according to which the results of our model could be verified is the sensing model based on EEG input. Figure 4 shows our speckle-based model and EEG model predictions for one tested subject. The speckle based and EEG inputs were recorded simultaneously during 10 second period while the sense of smell was active. The speckle-based model predicted 120 input batches, each containing 64 frames recorded under 750 frames per second (FPS). In order to compare EEG and the speckle-based model predictions we calculated the percentage of matching values in the sample shown in Fig. 4, For the presented sample the matching value is 92%, indicating that matching of EEG and our optical model are significant and high. For other tested participants the model matching value was in the range of 92–97 %. Table 2 shows that the EEG-based DNN achieved an accuracy of 83% while maintaining a recall of 100% and precision of 75% in active-sense classification task.

Table 2

EEG-based DNN comparison. The EEG based model reaches an accuracy of 83% while maintaining a recall of 100% and precision of 75% in active-sense classification task.
	Precision	Recall	F1-score	Accuracy
Sense	0.75	1.00	0.86	83
No-Sense	1.00	0.67	0.8	83

Current methods for sensing the human brain activity, such as EEG and MRI, usually require significant resources and expensive equipment and close proximity or physical contact with the patient. In this paper, we propose a photonic-based remote monitoring method for detection of human senses by combining a deep learning approach with spatiotemporal analysis of defocused self-interference random speckle patterns reflected from a specific temple area of the head. This work provides further evidence for the hypothesis that physiological processes associated with hemodynamic activity due to stimulation of the cerebral cortex by different senses could be identified by remote monitoring of nano-vibrations produced by transient blood flow to the specific regions of the head.

Temporal changes in the spatial distribution of the random speckle patterns can affect precision of the single frame method : the single frame might not represent whole recording session²⁶.

One major limitation of the secondary speckle patterns classification using full video frame-by-frame method is the noise that would be added due to the multiplicity of frames required to obtain prediction²¹. Namely, the first and the last recording frames could contain irrelevant information due to subject’s behavior and unintentional head movements

The two prior methods were based on a convnet model which does not consider temporal dependency related to speckle pattern signal.

The main difference between our approach and previous methods is related with the design of our model, which includes a ConvLSTM layer that considers temporal dependency found in our data in addition to the image processing capabilities of the convolution layers.

The underlying physiological processes of the human brain activity are time-dependent, hence, ConvLSTM based model allows the model to learn important features that describe the hemodynamic activity due to activation of the cerebral cortex of the human brain.

Figure 3 shows that sensory activity detection can be classified using learning-based methods. No significant difference in the model identification was found between the different types of senses or subjects, since in all cases the model input is expressed through the nano-vibrations associated with neural activity due to activation of the cerebral cortex of the subject's brain.

The comparison between the photonic and EEG- methods for human senses detection and classification shows high conformity between EEG and our speckle-based model.

4.1 Experimental setup

The experimental setup comprises of a laser (770 µW, 1550 nm) and a digital camera with defocused optics to generate and capture the speckle pattern. In addition, EEG electrodes, OpenBCI EEG headband with Ganglion bio-sensing 4-channels boar, were synchronized to record brain signals simultaneously.

Data was collected from five healthy participants, ages 29–73, in a shuttered and controlled laboratory environment to prevent background noises during the course of testing. Each subject was seated 50 cm across from a setup, as shown in Fig. 1. The subject’s head was restrained in a headset equipped with protective gear for the purpose of directing the left side of the head to the sensor and mitigating involuntary head movements. Each subject’s sense-related brain activity was recorded for two cases for each of the senses studied: smell, taste and hearing. First, in a state of rest when there is no sensory stimulation. Second, under stimulation of each sense by performing a relevant action. In order to stimulate the sense of smell the subject smelled an alcohol-soaked cotton ball; the sense of taste was stimulated with sweet chocolate and the sense of hearing with a continuous constant-frequency noise.

A high-speed camera with defocused optics was used to record the temporal changes of the speckle images. The frame rate was set to 750 FPS and spatial resolution of 32×32 pixels. Recording was performed for 10 seconds for each sample taken. Data collection was carried out on separate dates, with each subject being recorded several times in one continuous session. The dataset contained roughly 150,000 frames where each subject's video contained a unique identification, including the subject's ID, the duration of the measurement, the sense type, and a binary sign symbolizing activity or inactivity of the sense.

Videos from different recording days were sub-divided into training and test datasets prior to subdividing them into specific frames. Data for all the subjects was included in the training and test sets. These steps were taken to prevent mixing between the training and test dataset, which could have occurred with a simple random split.

Quantitative assessment and comparison of our proposed method used the metrics provided in Eq. 2–5 with TP: True Positive, TN: True Negative, FP: False Positive and FN: False Negative, which were calculated pixelwise by the logical operators given in Eq. 1.

Where the tuple (x_i, y_i) is the model prediction and label for sample i.

The ethics approval for the study is provided by the institutional review board of Bar-Ilan University. All participants provided informed consent for participation in the study. The experiments were carried out in accordance with relevant guidelines and regulations. The device, although dismantled for lab optimization purposes, is fully laser safe, tissue safe, etc. as was previously obtained from international regulators.

4.2. Model

We propose an approach to process sequential speckle images by using a 2D ConvLSTM layer³⁵ consisting of an LSTM layer with internal matrix multiplications along with 2D convolution operations. As the result, the speckle data passed through the ConvLSTM cells and retained the original input dimensions instead of being projected onto a 1D feature vector, as seen in Fig. 5.

The main equations of the ConvLSTM layer³⁵ are shown below (Eq. 6):

Where ∗ denotes the convolution operator; ◦ is the Hadamard product; X₁…X_t represent the model input; C₁…C_t are the ConvLSTM cell outputs; H₁…H_t are the hidden states; o_t,f_t,i_t are the 3D tensors of the ConvLSTM layer whose last two dimensions are spatial dimensions.

The ConvLSTM is defined by the inputs, past states of its local neighbors and the potential state of a certain cell in the grid. Before implementing the convolution operation, padding is required to ensure that the states have the same number of rows and columns as the inputs.

For the ConvLSTM model, we set the patch size to 1×1, so that each 32×32 frame is represented by a 32×32×3 tensor. The ConvLSTM single layer network contained 64 hidden states and input-to-state for 64 input frames, with state-to-state kernels of size 3×3. The output from the ConvLSTM encoder was fed into a fully connected, binary classification head, which contained 256-unit and Relu activations.

Additional implementation details include binary cross-entropy as a loss function, Adam optimizer and dropout to reduce overfitting. Training was performed on a single 1080Ti GPU and took roughly 20 epochs to converge.

4.3. EEG data classification

In order to verify the validity of the proposed method, EEG data was recorded simultaneously with the speckle patterns. To perform classification of the EEG signal, we used a CNN with three 1D-Conv layers with a Relu activation function and a 1D-MaxPooling operation were followed by two fully connected layers³⁶. No further preprocessing was performed prior to the EEG-based model.

Additional implementation details include using binary cross-entropy as a loss function and Adam optimizer. Training was performed on a single 1080Ti GPU and took roughly 10 epochs to converge.

This paper presents a speckle based photonic method for remote monitoring and detection of the three basic human senses: smell, taste, and hearing.

The method is based on a combination of spatiotemporal analysis of defocused self-interference random speckle patterns reflected from a specific temple area of the head with a deep learning approach.

The study provides further evidence for the hypothesis that physiological processes associated with hemodynamic activity due to stimulation of the cerebral cortex by different senses could be identified by remote monitoring of nano-vibrations produced by transient blood flow to the specific regions of the head. The developed DNN showed high accuracy in classifying active and inactive senses.

Our method offers an alternative and much simpler solution for detecting brain activity which typically require significant resources (for example: EEG and MRI devices). Furthermore, future development of our method could allow remote monitoring and evaluation of human brain activity on a large scale due to the low cost and flexibility of the system.

Corresponding Author

Zeev Kalyuzhner: [email protected]

Author contributions

Z.K created the models and conducted the training. All authors contributed to the design of the study, conducting the tests, interpreting the results, and writing the manuscript.

Competing Interests Statement

The authors declare no competing interests.

Data Availability Statement

The data generated to support the findings of this study are available from the corresponding author upon reasonable request.

Code Availability Statement

The code is available at https://github.com/zeevikal/senses-speckle.

Proctor, R. W. & Proctor, J. D. the Human Factors Fundamentals Chapter 3 Sensation and Perception. Handb. Hum. Factors Ergon. (2006).
Alarcão, S. M. & Fonseca, M. J. Emotions recognition using EEG signals: A survey. IEEE Trans. Affect. Comput. 10, 374–393 (2019).
Zhuang, X., Sekiyama, K. & Fukuda, T. Evaluation of human sense by biological information analysis. 20th Anniv. MHS 2009 Micro-Nano Glob. COE - 2009 Int. Symp. Micro-NanoMechatronics Hum. Sci. 74–79 (2009) doi:10.1109/MHS.2009.5352071.
Lee, M. & Cho, G. Measurement of human sensation for developing sensible textiles. Hum. Factors Ergon. Manuf. 19, 168–176 (2009).
Park, K. H. et al. Evaluation of human electroencephalogram change for sensory effects of fragrance. Ski. Res. Technol. 25, 526–531 (2019).
Hironobu Fukai, Yohei Tomita, Y. M. A design of the preference acquisition detection system using the EGG. Int. J. Intell. Inf. Syst. 2, 19–25 (2013).
Nakamura, T., Tomita, Y. & Mitsukura, Y. A method of obtaining sense of touch by using EEG. Inf. 14, 621–632 (2011).
Zoefel, B. & VanRullen, R. EEG oscillations entrain their phase to high-level features of speech sound. Neuroimage 124, 16–23 (2016).
Christensen, C. B., Harte, J. M., Lunner, T. & Kidmose, P. Ear-EEG-Based Objective Hearing Threshold Estimation Evaluated on Normal Hearing Subjects. IEEE Trans. Biomed. Eng. 65, 1026–1034 (2018).
Lorig, T. S. The application of electroencephalographic techniques to the study of human olfaction: A review and tutorial. Int. J. Psychophysiol. 36, 91–104 (2000).
Martin, G. N. Human electroencephalographic (EEG) response to olfactory stimulation: Two experiments using the aroma of food. Int. J. Psychophysiol. 30, 287–302 (1998).
Saha, A., Konar, A., Chatterjee, A., Ralescu, A. & Nagar, A. K. EEG analysis for olfactory perceptual-ability measurement using a recurrent neural classifier. IEEE Trans. Human-Machine Syst. 44, 717–730 (2014).
Saha, A., Konar, A., Rakshit, P., Ralescu, A. L. & Nagar, A. K. Olfaction recognition by EEG analysis using differential evolution induced Hopfield neural net. Proc. Int. Jt. Conf. Neural Networks (2013) doi:10.1109/IJCNN.2013.6706874.
Park, C., Looney, D. & Mandic, D. P. Estimating human response to taste using EEG. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS 6331–6332 (2011) doi:10.1109/IEMBS.2011.6091563.
Boas, D. A. & Dunn, A. K. Laser speckle contrast imaging in biomedical optics. J. Biomed. Opt. 15, 011109 (2010).
Jiang, M. et al. Dynamic imaging of cerebral blood flow using laser speckle during epileptic events. Biomed. Opt. BIOMED 2012 195–201 (2012) doi:10.1364/biomed.2012.btu3a.45.
Ozana, N. et al. Remote photonic sensing of cerebral hemodynamic changes via temporal spatial analysis of acoustic vibrations. J. Biophotonics 13, 1–12 (2020).
Zalevsky, Z. et al. Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern. Opt. Express 17, 21566 (2009).
Lengenfelder, B. et al. Remote photoacoustic sensing using speckle-analysis. Sci. Rep. 9, 1–11 (2019).
Golberg, M., Ruiz-Rivas, J., Polani, S., Beiderman, Y. & Zalevsky, Z. Large-scale clinical validation of noncontact and continuous extraction of blood pressure via multipoint defocused photonic imaging. Appl. Opt. 57, B45 (2018).
Kalyuzhner, Z., Agdarov, S., Bennett, A., Beiderman, Y. & Zalevsky, Z. Remote photonic sensing of blood oxygen saturation via tracking of anomalies in micro-saccades patterns. Opt. Express 29, 3386 (2021).
Ozana, N. et al. Demonstration of a Remote Optical Measurement Configuration That Correlates with Breathing, Heart Rate, Pulse Pressure, Blood Coagulation, and Blood Oxygenation. Proc. IEEE 103, 248–262 (2015).
Beiderman, Y. et al. Remote estimation of blood pulse pressure via temporal tracking of reflected secondary speckles pattern. J. Biomed. Opt. 15, 061707 (2010).
Bishitz, Y. et al. Noncontact optical sensor for bone fracture diagnostics. Biomed. Opt. Express 6, 651 (2015).
Ozana, N. et al. Remote optical configuration of pigmented lesion detection and diagnosis of bone fractures. Photonic Ther. Diagnostics XII 9689, 968916 (2016).
Kalyzhner, Z., Levitas, O., Kalichman, F., Jacobson, R. & Zalevsky, Z. Photonic human identification based on deep learning of back scattered laser speckle patterns. Opt. Express 27, 36002 (2019).
Connor, J. T., Martin, R. D. & Atlas, L. E. Recurrent Neural Networks and Robust Time Series Prediction. IEEE Trans. Neural Networks 5, 240–254 (1994).
Banerjee, T. K., Roy, M. K. & Bhoi, K. K. Is stroke increasing in India - Preventive measures that need to be implemented. Journal of the Indian Medical Association vol. 103 162, 164, 166 passim–162, 164, 166 passim (2005).
Green, T. L., McGregor, L. D. & King, K. M. Smell and taste dysfunction following minor stroke: a case report. Can. J. Neurosci. Nurs. 30, 10–13 (2008).
Penry, W. H. T. R. J. P. & Kiffin, J. Complex partial seizures Clinical characteristics and differential diagnosis. Handb. Park. Dis. Fifth Ed. 33, 11515--11515 (1983).
Nguyen, M. Q. & Ryba, N. J. P. A smell that causes seizure. PLoS One 7, 1–10 (2012).
Tong, J. Y., Wong, A., Zhu, D., Fastenberg, J. H. & Tham, T. The Prevalence of Olfactory and Gustatory Dysfunction in COVID-19 Patients: A Systematic Review and Meta-analysis. Otolaryngol. - Head Neck Surg. (United States) 163, 3–11 (2020).
Walker, A., Pottinger, G., Scott, A. & Hopkins, C. Anosmia and loss of smell in the era of covid-19. BMJ 370, 1–4 (2020).
Cooper, K. W. et al. COVID-19 and the Chemical Senses: Supporting Players Take Center Stage. Neuron 107, 219–233 (2020).
Shi, X. et al. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Adv. Neural Inf. Process. Syst. 2015-Janua, 802–810 (2015).
Craik, A., He, Y. & Contreras-Vidal, J. L. Deep learning for electroencephalogram (EEG) classification tasks: A review. J. Neural Eng. 16, (2019).

No competing interests reported.

Download PDF

Journal Publication

published 11 Jan, 2022

Read the published version in Scientific Reports →

Editorial decision: Major revision
15 Nov, 2021
Reviews received at journal
13 Nov, 2021
Reviewers agreed at journal
10 Nov, 2021
Reviews received at journal
24 Oct, 2021
Reviewers agreed at journal
14 Oct, 2021
Reviewers agreed at journal
10 Oct, 2021
Reviewers invited by journal
07 Oct, 2021
Editor assigned by journal
06 Oct, 2021
Editor invited by journal
20 Aug, 2021
Submission checks completed at journal
20 Aug, 2021
First submitted to journal
16 Jul, 2021

You are reading this latest preprint version

Remote Photonic Detection of Human Senses Using Secondary Speckle Patterns

Status:

Journal Publication

Version 1

Abstract

Figures

1. Introduction

2. Results

3. Discussion

4. Methods

4.1 Experimental setup

4.2. Model

4.3. EEG data classification

5. Conclusions

Declarations

References

Additional Declarations

Status:

Journal Publication

Version 1