We first describe the setup that was developed for on-line data analysis in real-time. We used DYNAP-SE to process intraoperative ECoG data in the 4–80 Hz and 250–500 Hz frequency bands to detect HFO and IED-HFO (Fig. 1). In the following we describe each step of the processing pipeline (Fig. 2) in detail.
2.1 Signal preprocessing
To validate a real-time scenario when using pre-recorded data, we used the BCI2000 FilePlayback module at real speed. To stream intraoperative data in real-time, we used the MicromedADC module.
We streamed up to 32 channels in parallel. 32 samples of data (64 ms) were buffered and then processed. Filtering was performed in the traditional EEG frequency band (4–80 Hz) and HFO frequency band (250–500 Hz) with a 64th order FIR filter. This choice follows the guideline of employing a high filter order for HFO detection 20.
2.2 Delta-modulation encoding
ECoG data were streamed and converted into discrete digital pulses through an Asynchronous Delta Modulator encoding(ADM, Fig. 3) that we implemented in BCI2000 (ADMFilter module). The ADM processing transformed the signal into UP/DN pulses, focusing on epileptiform patterns as events of interest (EoI). Two parameters govern how the ADM operates: the threshold level δ, and the refractory period τ. At the start of the encoding, the first ECoG sample x(0) is set as baseline, and two thresholds are created. An UP threshold at x(0)+δ, and a DN threshold at x(0)-δ. If the signal crosses one threshold at time t, a corresponding UP or DN pulse is produced, and new thresholds are set at x(t)+δ and x(t)-δ. The minimum allowed inter-pulse interval is equal to the refractory period τ.
The ADMFilter module consisted of an initial tuning phase and an encoding phase. In the tuning phase, streaming data were segmented into non-overlapping windows. The amplitude range of each window was stored, and the ADM threshold level δ was computed as a predefined percentile of the amplitude range distribution. Duration of the tuning phase was set at 5 s, length of non-overlapping windows was set at 50 ms for the EEG band and 5 ms for the HFO band, the percentile level was set at 40% for the EEG band and at 50% for the HFO band. Parameters were kept fixed for all patients.
In our analysis, the selected threshold level preserved the morphology of IED and HFO, while the signal with amplitude below threshold was discarded. The ADM processing thus compresses the continuous ECoG trace and is well suited for remote analyses that require data transfer.
2.3 SNN processing
The discrete UP/DN pulses were then processed with a hardware Spiking Neural Network (SNN) implemented on the Dynamic Asynchronous Neuromorphic Processor (DYNAP-SE) 18 that performed signal compression and focused on EoI (Fig. 4). The silicon neurons in the DYNAP-SE are grouped in four cores of 256 neurons each. Due to their analog nature, the neuron circuits exhibit a variability induced by circuit device mismatch factors that arise during circuit fabrication 21. Although circuit parameters are shared between all neurons in the same DYNAP-SE core, the device mismatch induced variability produces a distribution of parameters, with shared mean values, but with a coefficient of variation that can be as large as 20% 22. Although device mismatch is typically perceived as a limitation in analog computation, in our analysis the inherent neural variability is beneficial since it allows processing the incoming ADM pulses with an ensemble of heterogeneous neurons, which has been shown to improve the information encoding and classification accuracy 23, 24.
UP/DN pulses from the EEG and HFO bands were processed separately on two different DYNAP-SE cores (EEG core and HFO core). One ECoG channel was analyzed by four populations of silicon neurons, two in the EEG core and two in the HFO core. Each population was composed of 10 neurons and accumulated activity only from the UP or DN stream. In the following, we refer to these populations as ACC UP and ACC DN. Since high-dimensional projection is useful for pattern separation in the brain, we implemented these two populations in the DYNAP-SE chip to detect epileptiform patterns and reject artifacts. 40 neurons are therefore allocated for the analysis of one channel, 20 for the EEG band, and 20 for the HFO band. The DYNAP-SE SNN can process up to 8 channels in parallel, for a total of 160 neurons in the EEG core and 160 neurons in the HFO core.
An evolutionary algorithm selected the best performing set of parameters, one for HFO and one for IED detection (Fig. 5). The algorithm worked as follows: parameters were sampled from a volume in the parameter space. All the sampled parameter configurations were tested on a ‘tuning snippet’ that was divided into an IN period, during and shortly after the occurrence of the epileptiform pattern, and two OUT periods, before and long after the occurrence of the epileptiform pattern. A score was associated to each parameter configuration based on the DYNAP-SE activity produced on this snippet. A score was assigned to each neuron \(i\) following \({score}_{i}=-\alpha ⎸spikes IN-1 ⎸+\beta \left(spikes IN-spikes OUT\right)\). \(spikes IN\)are all the SNN events that appeared inside the IN period of the tuning snippet. \(spikes OUT\) are all the SNN events that appeared in the OUT periods. For \(\beta >0\) and \(\alpha >0\), a neuron received an optimal score if it spiked only one time during the IN period. A global score was then assigned to the neural population based on the mean score over all neurons and the percentage of neurons that produced a spike. Best performing configurations were set as the centers of the new sampling space. The algorithm proceeded iteratively. Optimal score was associated with a sparse temporal coding in which the SNN produced spikes shortly after the epileptiform pattern. This coding can increase energy efficiency and processing speed, and it is explored in the context of learning in artificial spiking neural networks 25. The evolutionary algorithm was run as a recalibration step at the beginning of every processing session.
2.4 Detection
We designed simple rules to detect epileptiform patterns from SNN activity. ACC UP and ACC DN activities from EEG and HFO bands were convolved with a linear kernel of duration 100 ms. Contributions from all neurons were summed together, forming one EEG-SNN trace and one HFO-SNN trace. Periods of spiking activity were segmented, and for each period in the HFO-SNN trace we checked for the presence of both ACC UP and ACC DN activities. If both were present, we defined this period as an EoI-HFO. We classify a EoI-HFO that fulfills these criteria as HFO:
-
SNN spikes from the HFO band need to present ≥ 2 neural activations from ACC UP population, ≥ 2 neural activations from the ACC DN population, and a total contribution ≥ 6 neurons. This ensures the rejection of low voltage fluctuations.
-
Sharp transients in the ECoG produce filtering artifacts. These events produce a spiking pattern where ACC UP and ACC DN activities in the HFO band are separated in time. Therefore, the DYNAP-SE activity in the HFO band needs to present ACC UP and ACC DN activities that are temporally mixed.
-
The duration of SNN activity in the HFO band needs to be ≤ 30 ms.
-
If SNN activity is present in the EEG band during EoI-HFO occurrence, its duration needs ≤ 500 ms. This step rejects long high amplitude artifacts in the ECoG trace that may induce high amplitude activity in the HFO band.
If, together with an HFO, we observe SNN activity in the EEG band with well-separated ACC UP and ACC DN activities and duration ≤ 300 ms, we classify this pattern as IED-HFO.
2.5 Pre-recorded ECoG data in Zurich
We analyzed 22 patients who underwent epilepsy surgery in USZ. The resection was guided by intraoperative high-density ECoG (hd-ECoG, 5 mm contact spacing) for 8 patients and standard grid and strip electrodes (10 mm contact spacing) for 14 patients. The USZ ECoG dataset was previously analyzed for HFO detection with two different detectors, the Spectrum detector and the software SNN (SW-SNN). The Spectrum detector was developed on UMCU intraoperative ECoG recordings 11 and then applied to the USZ intraoperative ECoG data 9, 10. The SW-SNN detector was then applied on the same data set for the patients whose resection was guided by hd-ECoG 15, 16.
Pre-recorded data were streamed with the FilePlayback pipeline in BCI2000 at real speed to simulate a real-time scenario. As an example of the results of our processing pipeline, in the pre-resection ECoG of Patient 5 epileptiform patterns detected by the DYNAP-SE SNN had rates ≥ 1 min-1 only in channels that were later resected (Fig. 6). This was similar to the findings obtained with the Spectrum detector.
The Spearman correlation between the maximum HFO rate for each patient in DYNAP-SE SNN and Spectrum detectors amounted to ρ = 0.75 (p = 1e-4). For all patients, our analysis obtained the same seizure outcome predictions as Spectrum and SW-SNN detectors (Table 1). As for the Spectrum and SW-SNN detectors, in the 8 hd-ECoG patients we obtained PPV = 100%, NPV = 100%, sensitivity = 100%, specificity = 100% and accuracy = 100% (CI [63% 100%]). If we consider all patients, we reached PPV = 100%, NPV = 70%, sensitivity = 25%, specificity = 100% and accuracy = 73% (CI [50% 89%]). The decrease in sensitivity might arise from the lower spatial resolution of the standard ECoG contact spacing compared to the hd-ECoG. The Spearman correlation between the maximum IED-HFO rate for each patient in the DYNAP-SE SNN and the HFO rate of the Spectrum detector amounted to ρ = 0.77 (p = 7e-5). If we consider all patients, we reached PPV = 100%, NPV = 70%, sensitivity = 12.5%, specificity = 100% and accuracy = 68% (CI [45% 86%]). The additional decrease in sensitivity arises from the lack of IED in the post-resection recording of Patient 6, therefore classified as a FN.
Table 1
Patient
|
Etiology
|
Follow up (months)
|
Seizure Outcome (ILAE)
|
Spectrum
HFO Rate [min− 1] 9
|
SW-SNN
HFO Rate
[min− 1] 15
|
DYNAP-SE SNN
HFO Rate
[min− 1]
|
Spectrum Outcome Prediction
|
SW-SNN Outcome Prediction
|
DYNAP-SE SNN Outcome prediction
|
Pre
|
Post
|
Pre
|
Post
|
Pre
|
Post
|
USZ 1
|
DNET
|
33
|
1
|
6
|
< 1
|
3
|
< 1
|
6
|
< 1
|
TN
|
TN
|
TN
|
USZ 2
|
FCD 2b
|
24
|
1
|
4
|
< 1
|
10
|
< 1
|
5
|
< 1
|
TN
|
TN
|
TN
|
USZ 3
|
Sturge Weber
|
30
|
1
|
2
|
< 1
|
1
|
< 1
|
1
|
< 1
|
TN
|
TN
|
TN
|
USZ 4
|
Ganglioglioma
|
18
|
1
|
8
|
< 1
|
12
|
< 1
|
2
|
< 1
|
TN
|
TN
|
TN
|
USZ 5
|
FCD 2a
|
13
|
1
|
13
|
< 1
|
30
|
< 1
|
4
|
< 1
|
TN
|
TN
|
TN
|
USZ 6
|
Sturge Weber
|
20
|
3
|
32
|
5
|
45
|
14
|
10
|
1
|
TP
|
TP
|
TP
|
USZ 7
|
Astrocytoma
|
29
|
1
|
1
|
< 1
|
1
|
< 1
|
2
|
< 1
|
TN
|
TN
|
TN
|
USZ 8
|
FCD 2a
|
12
|
1
|
22
|
< 1
|
2
|
< 1
|
30
|
< 1
|
TN
|
TN
|
TN
|
USZ 9
|
FCD 2
|
43
|
1
|
1.8
|
< 1
|
-
|
-
|
< 1
|
< 1
|
TN
|
-
|
TN
|
USZ 10
|
Sturge Weber syndrome
|
30
|
3
|
< 1
|
< 1
|
-
|
-
|
< 1
|
< 1
|
FN
|
-
|
FN
|
USZ 11
|
Ependymoma (WHO II)
|
28
|
1
|
< 1
|
< 1
|
-
|
-
|
2
|
< 1
|
TN
|
-
|
TN
|
USZ 12
|
Anaplastic astrocytoma (WHO III)
|
14
|
1
|
< 1
|
< 1
|
-
|
-
|
< 1
|
< 1
|
TN
|
-
|
TN
|
USZ 13
|
Cavernoma
|
29
|
5
|
1.2
|
< 1
|
-
|
-
|
5
|
< 1
|
FN
|
-
|
FN
|
USZ 14
|
FCD 2b
|
16
|
1
|
1.8
|
< 1
|
-
|
-
|
5
|
< 1
|
TN
|
-
|
TN
|
USZ 15
|
FCD 3
|
35
|
5
|
n.a
|
< 1
|
-
|
-
|
< 1
|
< 1
|
FN
|
-
|
FN
|
USZ 16
|
ODG (WHO II)
|
31
|
1
|
< 1
|
< 1
|
-
|
-
|
< 1
|
< 1
|
TN
|
-
|
TN
|
USZ 17
|
FCD 2a
|
36
|
3
|
n.a
|
< 1
|
-
|
-
|
< 1
|
< 1
|
FN
|
-
|
FN
|
USZ 18
|
FCD 1a
|
38
|
5
|
1.2
|
2.7
|
-
|
-
|
7
|
6
|
TP
|
-
|
TP
|
USZ 19
|
Hippocampal sclerosis
|
40
|
5
|
< 1
|
< 1
|
-
|
-
|
< 1
|
< 1
|
FN
|
-
|
FN
|
USZ 20
|
FCD 1c
|
38
|
1
|
7.0
|
< 1
|
-
|
-
|
5
|
< 1
|
TN
|
-
|
TN
|
USZ 21
|
Cavernoma
|
35
|
5
|
< 1
|
< 1
|
-
|
-
|
< 1
|
< 1
|
FN
|
-
|
FN
|
USZ 22
|
Fibrillary astrocytoma (WHO II)
|
40
|
1
|
< 1
|
< 1
|
-
|
-
|
< 1
|
< 1
|
TN
|
-
|
TN
|
UMCU 23
|
Encephalocele
|
2
|
1
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
-
|
DNET: dysembryoplastic neuroepithelial tumor; FCD: focal cortical dysplasia; ODG: oligodendroglioma; WHO: World Health Organization; TN: true negative; TP: true positive; FN: false negative; HFO: High Frequency Oscillation; SW SNN: Software SNN. |
2.6 Real-time analysis of UMCU ECoG in Zurich
To test whether we can use the procedure in real-time, we performed a remote on-line analysis in collaboration with the UMCU. The ECoG was recorded in the surgical theatre with a Micromed console that was running the Micromed SystemPlus software and streamed the digitized data (2048 Hz) via the UMCU network to a UMCU hospital computer. This computer ran the MicromedADC and the signal processing module in BCI2000 as the real-time framework. There, the ADMFilter performed ADM encoding of the signal into a stream of UP and DN pulses. These lightweight data were then sent to USZ in Zurich, where the DYNAP-SE chip performed further processing (Fig. 7). Two IED-HFO events were detected.
Following the standard UMCU procedure, an expert observer (E.S.) annotated pathological HFO through visual inspection 4. Both IED-HFO patterns were marked as HFO in the Fast Ripple band (250–500 Hz).
2.8 ECoG compression and reconstruction
Event-based ADM encoding allowed compressing the ECoG trace while preserving the morphology of IED and HFO. As an example we present channel 02–03 in the pre resection recording of patient 5 with a duration of ~ 7 min and a sampling rate of 2 kHz. ADM encoding in the EEG band achieved a compression ratio ~ 20. Given the UP/DN pulses, the ECoG trace \(x\left(t\right)\) can be reconstructed. We define the reconstruction as \(\widehat{x}\left(t\right).\) Starting from\(\widehat{x}\left(0\right)=x\left(0\right)\), if a UP pulse occurs at time\({t}_{UP}\), then \(\widehat{x}\left({t}_{UP}\right)=\widehat{x}\left({t}_{UP}-\epsilon \right)+\delta\), where \(\epsilon \to 0\). If a DN pulse occurs at time\({ t}_{DN}\), then\(\widehat{x}\left({t}_{DN}\right)=\widehat{x}\left({t}_{DN}-\epsilon \right)-\delta .\)
The hardware SNN performed a high dimensional projection of the ADM signal and further compressed the ECoG trace. In channel 02–03 of the pre-resection recording of patient 5, the SNN encoding in the EEG band achieved a compression ratio ~ 34. In addition to the benefits of the high dimensional projection in the discrimination between epileptiform patterns and artifacts, we investigated the possibility to reconstruct the ECoG trace from the SNN sparse temporal coding.
For the ECoG reconstruction we used the full-FORCE algorithm 26, 27. This target-based method is typically employed to reproduce neural data recordings and to investigate the relation between neural activity and behavior. The goal of this reconstruction process is to reproduce the original ECoG trace from SNN activity alone. First, a randomly connected RNN (teacher RNN) received the SNN activity together with the target ECoG trace, while the internal activity of the RNN neurons was recorded. This process ensured that the internal activity combined the information of both input and output. A second network (student RNN) was then trained with a recursive least-squares algorithm to match the internal activity of the teacher RNN when receiving only the SNN activity as input. A linear readout then extracted the target ECoG trace from the student RNN activity.
We trained this reconstruction algorithm on the first 100 seconds of recording from the EEG band of the pre-resection recording of patient 5 in channel 02–03. We then tested on the following 300 seconds. We analyzed the Pearson correlation between the ECoG trace and the SNN reconstruction in snippets where both ACC UP and ACC DN activities were present and the SNN activity was ≤ 500 ms. We obtained similar correlation distributions for our train and test sets, with a median Pearson correlation of 0.79 for the train set and 0.73 for the test set. The reconstruction preserved the main morphological features of the ECoG trace (Fig. 8).