Invasive neurophysiology and whole brain connectomics for neural decoding in patients with brain implants

doi:10.21203/rs.3.rs-3212709/v1

Download PDF

Article

Invasive neurophysiology and whole brain connectomics for neural decoding in patients with brain implants

https://doi.org/10.21203/rs.3.rs-3212709/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Brain computer interfaces (BCI) provide unprecedented spatiotemporal precision that will enable significant expansion in how numerous brain disorders are treated. Decoding dynamic patient states from brain signals with machine learning is required to leverage this precision, but a standardized framework for identifying and advancing novel clinical BCI approaches does not exist. Here, we developed a platform that integrates brain signal decoding with connectomics and demonstrate its utility across 123 hours of invasively recorded brain data from 73 neurosurgical patients treated for movement disorders, depression and epilepsy. First, we introduce connectomics-informed movement decoders that generalize across cohorts with Parkinson’s disease and epilepsy from the US, Europe and China. Next, we reveal network targets for emotion decoding in left prefrontal and cingulate circuits in DBS patients with major depression. Finally, we showcase opportunities to improve seizure detection in responsive neurostimulation for epilepsy. Our platform provides rapid, high-accuracy decoding for precision medicine approaches that can dynamically adapt neuromodulation therapies in response to the individual needs of patients.

Biological sciences/Neuroscience/Computational neuroscience/Neural decoding

Physical sciences/Engineering/Biomedical engineering

The advent of clinical brain computer interfaces (BCI) that can adapt neurostimulation therapy to therapeutic demand signifies a unique moment in medical history. Closed-loop therapies have begun to automatize therapeutic adjustment based on signals recorded directly from the brain and body of patients^1–5, with millisecond precision. Raw signals from intracranial electroencephalography (iEEG) electrodes are increasingly available to be transformed into input features for developing machine learning-based therapeutic control policies that can adapt therapy in real-time. Such strategies have unprecedented potential to improve the quality of lives of millions of patients with brain disorders, by reacting precisely and continuously to fluctuating signals of patient-specific brain states.

To realize the full potential of BCIs, robust evaluation and implementation of machine learning models must be enabled. Three major obstacles, however, stand in the way: 1) Machine learning can capitalize on large datasets but current brain signal decoding approaches for BCI still commonly rely on individual patient training sets of short duration. This reliance is impractical for broad clinical adoption and significantly limits the utility of complex models for generalizable performance. 2) Current advances in basic neurosciences have revealed the presence of complex brain signal features such as waveform shape and aperiodic activity^6,7, but standardized methods to extract them for machine learning applications are lacking. 3) Human behavior is encoded in whole brain circuit interactions, but current brain signal decoding solutions often fail to account for the underlying brain networks that give rise to the recorded signals⁸. To overcome these roadblocks, we developed a modularized open-source software platform for invasive brain signal decoding in neuromodulation via BCIs.

In this paper, we introduce this methodology and highlight its utility in a large cohort of patients across thousands of recording sources from invasive brain implants within three key areas of clinical neurotechnology innovation: movement disorders, psychiatry, and epilepsy.

Processing of invasive neurophysiology data for machine learning based brain signal decoding

We developed an open, integrative and modularized platform (Fig. 1), for standardized and reproducible implementation of machine learning based brain signal decoding algorithms (https://github.com/neuromodulation/py_neuromodulation).

The modularized feature estimation chains include a versatile set of state-of-the-art signal processing algorithms. In brief, oscillatory dynamics, waveform shape, interregional coherence^6,7,10,11 and more can be extracted through the modular architecture which further allows for flexible extension for more advanced feature estimation metrics, such as direction of information flow with granger causality or phase amplitude coupling¹² (Supplementary Table 1). A key problem for invasive brain signal decoding is the individualized localization of brain implants across patients, which significantly complicates the development of large-scale models that generalize across patients. To address this, we developed dedicated approaches for across-patient decoding based on normative MRI connectomics and latent embeddings from contrastive learning with CEBRA¹³. In the following, we highlight the utility of this platform across three invasive brain stimulation use-cases.

Generalizable models for movement decoding across 1480 invasive recordings with prospective real-time validation

For the treatment of movement disorders, the decoding of volitional and pathological motor output has the potential to provide critical information for therapeutic adjustment for different diseases such as Parkinson’s disease or essential tremor^1,4. We implemented py_neuromodulation for electrocorticography (ECoG) based movement decoding without individual training (56 patients, 1480 channels, mean age: 50.6 ± 17.8, 24 females) from three independent PD (n=38; Berlin/Pittsburgh/Beijing) and one epilepsy cohort (n=18 from Washington¹⁴) performing different upper limb movements (Fig. 2a, Supplementary Tables 2 and 3). In PD patients, ECoG strips were introduced through DBS burr-holes¹⁵. In a sub-cohort of six patients (Berlin), recordings were repeated during clinically effective STN-DBS.

First, patient specific movement classification performances based on ECoG signals were analyzed. We trained ridge regularized logistic regression classifiers and evaluated their performance using the balanced accuracy metric with 3-fold cross-validation on consecutive data segments, at the single sample level (100 ms precision indicating presence or absence of movement) and at the individual movement level (300 ms or more movement time decoded consecutively). Performance was significantly above chance in every subject, with an average balanced accuracy of 0.8/0.98 ± 0.07/0.04 for single sample/movement detection in the best channel per subject (Fig. 2b-d).

Importantly, we could identify two key factors associated with relatively lower performances that will be crucial to account for in future clinical applications: disease severity and therapeutic DBS. First, we show that decoding performance is negatively correlated with clinical symptom severity in PD patients, as measured with Unified Parkinson’s Disease Rating Scale (Spearman’s rho = -0.36; p = 0.02) (Fig. 2e). We reported a similar result earlier for the Pittsburgh cohort¹¹, and therefore repeated the analysis for Berlin and Beijing alone, excluding all previously reported data, and could empirically reproduce the negative correlation across new cohorts (rho=-0.43, p=0.03). We may speculate that neurodegeneration in PD may impact neural encoding of movement, which may also impede machine learning based decoding performance. Next and equally important, we show that therapeutic electrical stimulation (130 Hz STN-DBS) can significantly deteriorate sample-wise decoding in some but not all patients and that models trained separately for OFF and ON stimulation conditions outperform models trained on either condition alone. Nevertheless, movement detection remained acceptable with 0.88 ± 0.17 movement detection rate across movements even during high-frequency DBS (Fig. 2f,g). Our results highlight the necessity to account for therapies and disease related brain state changes in clinical BCI.

The more serious limitation of the abovementioned results is the dependence on patient individual training. For a real-world clinical application this means that every implant would need to undergo tedious model training sessions, which could be a burden to both patients and medical staff and may hinder a broad clinical adoption. To address this critical limitation, we explored three computational approaches that do not depend on patient individual training while accounting for individual differences in ECoG strip location. In a first approach, the data were spatially extrapolated to a manually defined 38-point cortical surface grid, similar to a previous study¹⁶ (Fig. 2h). The disadvantage of this however, is a) that extrapolation leads to inaccuracy in spatial estimation and b) large amounts of data would be required to train a grid of the whole brain, which limits the application to cohorts with very similar electrode locations.

To overcome these shortcomings, we have developed a connectomic approach for across-patient decoding that optimally accounts for the specific recording localization while being generalizable across the whole brain. It builds on functional or structural connectivity fingerprints extracted from brain signal recording locations in normative space. In brief, voxel-wise correlations between decoding performance and whole-brain connectivity maps seeded from channel MNI coordinates were calculated to identify an optimal connectomic template fingerprint for movement decoding (so called connectomic decoding network map) across all subjects (Fig. 2i). This allows for an optimized a priori channel selection in real-time, by identifying the individual recording channel that has most network overlap with the optimal template. Finally, we have transformed neural features from the selected channel into a lower dimensional embedding¹³. For this, a five-layer convolutional neural network with a temporal filter length of 1 s was trained using the InfoNCE (Noise-Contrastive Estimation) contrastive loss function¹⁷. The resulting embeddings showed exceptionally high consistency across subjects as investigated with linear identifiability (Fig. 2j,k).

All three approaches reached significant above chance balanced accuracy and movement detection metrics (Table 1; Fig. 2l,m, all p<0.05) for cross-validation without patient-individual training within and across cohorts, and even leaving entire cohorts out. This indicates high generalizability across movement types, neurological disorders, recording setups and individual implant trajectories. In addition to the conceptual advantage to account for specific recording location and underlying brain network affiliation, the connectomic approach with contrastive learning (CEBRA) significantly outperformed the linear model for the most challenging across cohort cross-validation in sample-wise balanced accuracy (p=0.001) and overall achieved highest average performances (see Table 1).

Table 1: Leave one subject out and leave one cohort out cross-validation results for generalized movement decoding across patients, diseases and movement types. Decoding performance is depicted as balanced accuracy, which accounts for class imbalances. Single sample estimates provide performance metrics at 100 ms precision. Movement detection estimates were defined as 300 ms consecutive classification output with respect to presence and absence of movement. On average, the connectomic approach combined with contrastive learning (CEBRA) provided best cross-validation performances.

	Grid extrapolation		Connectomics with LM		Connectomics with CEBRA
	Sample	Movement	Sample	Movement	Sample	Movement
Within cohort	0.71 ± 0.08	0.84 ± 0.12	0.72 ± 0.08	0.86 ± 0.10	0.74 ± 0.1	0.86 ± 0.14
Across cohorts	0.64 ± 0.1	0.78 ± 0.2	0.66 ± 0.09	0.78 ± 0.19	0.68 ± 0.09	0.81 ± 0.16
Leave one cohort out	0.54 ± 0.11	0.61 ± 0.22	0.57 ± 0.1	0.68 ± 0.19	0.6 ± 0.11	0.7 ± 0.19
Average	0.63 ± 0.1	0.74 ± 0.18	0.65 ± 0.09	0.77 ± 0.16	0.67 ± 0.1	0.79 ± 0.16

To investigate the individual variability of trainer vs. learner performance and potentially improve the training process according to diversity related factors, we established a subject-to-subject decoding matrix. This allowed to identify individual subjects that provide outstanding training data for other subjects respectively (Fig. 2n). The average sample-wise decoding performance trained on just the best trainer sub_002 (Berlin) for the remaining cohort of 55 patients was 0.71 ± 0.14 for sample-wise prediction and 0.85 ± 0.2 for movement detection. Conceptually, this process could allow for more fine-grained individualized precision medicine approaches that can account for diversity in all its humans forms, e.g. by matching training data to demographic or genetic profiles of patients receiving brain implants.

Finally, after successfully training across patient models offline, we validated the approach prospectively in a newly recruited patient in Berlin (Supplementary Video 1). Importantly, pretrained patient naïve models outperformed individually trained models within the same recording session, with best performance of the model trained on all patients from the Berlin cohort (sample-wise balanced accuracy: 0.71; movement detection rate: 0.97), with slightly lower performances for a model trained on the patient individual data within the same session (45 movements; ~9 minute recording; sample-wise balanced accuracy: 0.67; movement detection rate: 0.97) (Fig. 2o,p). In a third run, we could further validate the patient to patient decoding results, by demonstrating that a model trained on the single best trainer (sub_002) reached comparable performance to the model trained on the patient individual data.

Decoding emotions from subgenual cingulate cortex in major depressive disorder

Invasive brain signal decoding has clinical utility beyond movement decoding. In the following we highlight the clinical potential of py_neuromodulation for brain circuit discovery in the neuropsychiatric domain. In the future, closed-loop therapies for affective disorders may adapt neuromodulation to concurrent mood or may support patients in the valuation of perceived emotion^18,19. Here, we employed py_neuromodulation to investigate optimal circuits, target features and computational approaches to decode perceived emotion from the primary DBS target for major depressive disorder, the subgenual cingulate cortex (SCC).

Machine learning decoders were trained on local field potential signals from the DBS electrodes in SCC in eight patients undergoing DBS for treatment resistant major depressive disorder as part of a clinical trial (mean age: 48 ± 11.4, 4 females; Supplementary Table 4). Neurophysiological recordings were conducted extraoperatively while DBS leads were externalized and acquired while patients participated in a visual emotion task (Fig. 3a). Visual stimuli included pleasant, unpleasant, and neutral stimuli from the international affective picture system (IAPS) database (for further information see²⁰) and were presented for a duration of 1 s with an inter-stimulus interval of 6-8 s. We used py_neuromodulation to estimate a unique and novel feature set that included temporal waveform features, such as discharge prominence, sharpness, decay and rise time, and peak and trough interval in addition to traditional oscillatory FFT features.

Logistic regression classifiers were trained to distinguish emotional valences using samples during stimulus presentation and cross-validated within patients as described above (Fig. 3b). Above-chance performance was obtained in all patients, with performance rising at 150 ms and peaking at 600 ms post-stimulus onset (Fig. 3c). Sample-wise performances were above chance-level for all classifications (balanced accuracy 0.62 ± 0.05; neutral vs pleasant: 0.64 ± 0.07, neutral vs unpleasant performances: 0.6 ± 0.03, pleasant vs unpleasant: 0.63 ± 0.01) (Fig. 3d). Decoding performance was driven by oscillations including high-frequency (200-400 Hz) activity (HFA), low- (60-90 Hz) and high-gamma (90-200 Hz) activity, followed by waveform shape features including rise time and prominence (Fig. 3e).

To investigate a potential relationship with clinical scores, we correlated decoding performances from the most predictive channel contrasting neutral vs. positive/negative per patient with Beck’s Depression Inventory (BDI) at time of recording and after six months of chronic DBS. Decoding performance correlated with DBS induced improvement in BDI scores (rho=0.79, p=0.01), but not with concurrent symptom severity (Fig. 3f). The correlation could be driven by optimal targeting rather than by depressive symptoms themselves, which inspired us to explore the underlying whole-brain networks. To this end, we used both dMRI and fMRI connectomics as above and an additional fiber filtering approach recently introduced in the context of DBS for OCD^21,22. In all cases connectivity fingerprints seeded from LFP channel locations were correlated with channel specific decoder test-set performance. The identified fiber tracts and whole-brain dMRI fingerprints were predictive of decoding performance and robust to leave-one-channel-out (Supplementary Fig. 1; fiber filtering rho=0.48, p<10^-5, whole-brain dMRI fingerprint: rho=0.38, p=0.002) and leave-one-subject-out cross-validation (fiber filtering rho=0.46, p<10^-5,whole-brain dMRI fingerprint: rho=0.46, p<10^-5). Functional connectivity was robust to leave-one-channel out (rho=0.39, p=0.001) but not leave-one-subject out validation (p>0.05).

A consistent left lateralized prefrontal network emerged across modalities that has direct overlap with network targets for the treatment of depression for transcranial magnetic stimulation and has relevant similarity with networks associated with mood change²³, alleviation of depression and affective changes with subthalamic DBS in Parkinson’s disease^24,25 (Fig. 3g,h, Supplementary Fig. 2). In the future, emotion decoding may become relevant to adapt neurostimulation control and select optimal targets from electrophysiological biomarkers and connectomics^5,26,27.

Optimization of seizure detection parameters for responsive neuromodulation in epilepsy patients

Responsive neurostimulation (RNS; Neuropace) is a closed-loop stimulation device for treatment for medication resistant epilepsy²⁹. The device is comprised of a cranially mounted battery and processor connected to two electrodes which can be either an ECoG strip or a stereoencephalography (sEEG) depth-electrode in proximity to the putative seizure focus. RNS promises a superior reduction in the number of disabling seizures by processing LFP signals using programmable detectors of seizures (i.e. ictal) states to deliver temporally targeted stimulation. However, mechanisms of closed-loop stimulation for epilepsy are not well understood. For some patients, this therapy can be life-changing with complete remission of seizures, while for others the therapeutic success remains below expectations². A key differentiating feature versus open loop therapies is the ability to target stimulation to specific neurophysiologic states (such as ictal, inter-ictal, or quiescent) via physician-selected detection parameter settings. Divining appropriate settings from this vast parameter space is a largely manual process taking months or years, if ever, to achieve optimal sensitivity and specificity^30,31. Improvements in parameter selection may lead to faster and better clinical outcomes and identify new features of interest.

Here, we aim to inspire new ways to improve seizure detection accuracy by constraining the decoding platforms to the specifications of clinical brain implants and suggesting improved parameters from offline predictions that are implementable and testable through the clinical patient data management systems (PDMS) provided by Neuropace. For this, we analyzed over 100 hours of invasive human brain signals from neocortical depth electrodes (see Fig. 4a for an example) recorded with RNS devices in a cohort of nine epilepsy patients (mean age: 35.3 ± 8.2, 8 females, all focal epilepsy; mean number of available recordings: 636.9 ± 366.4, mean recording duration: 71.07 ± 10.78 s, Supplementary Table 5). With the aim to reduce the false positive rate of the implemented seizure detector, we performed a systematic feature optimization in a simulation of the RNS bandpass detector algorithm. This embedded algorithm is normally programmed by the clinical team via the selection of a single ictal event (“SimpleStart”) which provides the foundation for a semi-automatized parametrization of detection settings for this specified event in the programming environment.

We extracted brain signals and detection and stimulation parameters from RNS implants (Fig. 4b) using a previously described database access pipeline³². Individual recordings were annotated by a certified epileptologist for electrographic presence and onset of seizure activity. The resulting annotations served as ground truth for using machine learning methods that can identify embedded bandpass algorithm parameters using our platform. Three parameters need to be defined for the detector by the clinical team: i) the threshold direction, i.e. whether increase vs. decrease in band power is associated with ictal activity, ii) the corresponding threshold amplitude and iii) the required duration the threshold is crossed for a seizure event to be detected. We optimized these parameters for sample-wise seizure classification to maximize the F1-score. We focused on F1 scores instead of balanced accuracy, because ‘seizure present’ (true positive) predictions are more critical for clinical scenarios than correct ‘seizure absent’ (true negative) predictions, which is why they are commonly used as a metric for RNS seizure prediction performance³³.

Brain signals aligned to seizure onset revealed high-frequency synchronization followed by activity in lower frequency bands, as previously described (Fig. 4c)^34,35. An exemplar grid-search matrix spanning the threshold direction, amplitude and duration of each feature, demonstrates how py_neuromodulation can directly provide access to RNS detector parameters that can be used in the embedded framework and implemented through the patient data management system (Fig. 4d,e). The identified parameter combination significantly reduced false positives, while maintaining stable true positive rates leading to overall performance increases (F1 score for original RNS settings: 0.41 ± 0.12, F1 score for our optimized settings 0.92 ± 0.06) (Fig. 4f). It is important to note that the embedded RNS programming environment does not have cross-validation implemented which was mirrored in our simulation, and that RNS data are in part recorded because of true or false- positive seizure detection (114.16 ± 86.42 minutes out of 6515 minutes of total recording time were classified as ictal by expert annotations). Thus, the presented data should rather be interpreted as a proof-of-concept that requires further validation in prospective clinical trials.

To highlight the utility of multivariate brain signal decoding for seizure detection beyond the RNS device limitations we further evaluated an extensive set of 264 features (66 features per channel) (Fig. 4g). We then assessed the performance of linear models, support vector machines (SVM), and gradient boosted decision trees (XGBOOST) using a 3-fold cross-validation as before. XGBOOST achieved best performances (F1 score: 0.8 ± 0.2) and outperformed linear models (Linear Model 0.56 ± 0.23, Support Vector Machine 0.4 ± 0.3) (Fig. 4h), potentially by capturing non-linear interactions more robustly than SVMs and linear models.

Our results emphasize two key aspects regarding the utility of our platform for the development of decoding algorithms for brain implants: i) it can be of direct use for the parametrization of currently available and embedded algorithms in available brain implants, ii) it enables the discovery of optimized feature sets and machine learning methods for the next generation of clinical brain computer interfaces for the treatment of epilepsy and other brain disorders.

We introduce py_neuromodulation, a modularized invasive brain signal decoding platform for real-time implementation in the context of clinical brain computer interfaces. We highlight three major advances of this platform in use cases covering key innovation domains for neurotechnology across more than 123 hours of brain signals from seventy patients who have undergone treatment with brain implants.

First, we demonstrated the ability to decode behavior across disease entities, movement types, acquisition systems and cohorts from the US, Germany and China. Importantly, our pipeline achieved high decoding performance in a prospective patient naïve real-time validation of pretrained movement decoding models. We propose a combination of connectomics based channel selection and contrastive learning (CEBRA) for across patient decoding that circumvents tedious patient individual training that would obviate widespread clinical adoption of brain signal solutions. Moreover, we show that training data from an individual subject can generalize to other subjects, which holds promise for across-patient decoding applications even in rare diseases. It may further allow to account for different dimensions of human diversity, such as anatomical variation, genetic differences and/or disease severity.

Next, we highlight the utility of our approach beyond movement disorders, by decoding perceived emotional valence in patients with treatment resistant depression. Here, we additionally demonstrated the predictive relevance of feature sets beyond oscillatory activity such as waveform shape for emotion decoding. Combining brain signal decoding with connectomics in this use case revealed the whole-brain circuits underlying emotion decoding in a left prefrontal network. The resulting network showed resemblance to previously published and validated neuromodulation targets and lesion networks that were reported to affect mood^23,24,37 and may indicate optimal treatment targets to refine DBS for depression.

Finally, we showcase opportunities to directly improve parametrization of RNS detectors for specific neurophysiologic states of seizure networks, such as detection of ictal states, through feature identification using available ECoG recordings and simulation of available detector settings for the treatment of epilepsy. The vast parameter space of closed-loop systems, such as RNS, present an optimization challenge significantly better-suited to machine learning models than the current length and error-prone manual process. Further, the features of seizure networks may change over time, further delaying optimization as physicians struggle to keep up with a moving target. Timely implementation of optimal seizure detection settings, even as the network features evolve, is imperative to directly improving clinical outcomes. Prospective validation of this approach in clinical trials may lead to lower false positive stimulation rates, higher temporal specificity and lower stimulation dose, and promise both faster and superior clinical outcomes with fewer side-effects.

Our use-cases are built on the prediction of brain states in data from neurosurgical patients who have undergone treatment with neural implants for brain stimulation. However, an important limitation is that through the retrospective nature of these use-cases, we did not evaluate the efficacy or temporal precision of resulting brain stimulation algorithms. Nevertheless, decoding performances were investigated in the presence of clinical brain stimulation in two out of three use cases, namely STN-DBS for PD and responsive neurostimulation for epilepsy. Moreover, its application is particularly promising for the combination of neural population activity and LFP for the prediction of low-dimensional decoding targets. Thus, it may have additional utility but will not replace the complex algorithm development in other research domains, e.g. using single unit activity in the context of spatial navigation^38,39 or for complex neuroprostheses to recover speech or to provide brain to text communication^40,41 that have reached astonishing performance through extensive optimization of recording techniques, model architectures and control policies tailored to these specific applications. Finally, some similar and many other use cases for machine learning based invasive brain signal decoding have been reported before^{1,4,5,16,27, 42–45}.

What makes py_neuromodulation in this context unique is that it provides a collaborative open-source platform for standardized and reproducible translation of offline decoding into clinical applications. Meanwhile, it extends previous methods for feature extraction and combines them with novel solutions to across patient decoding with MRI connectomics. With this, it is prepared to serve large-scale multicenter collaborations as demonstrated with this study, to develop machine learning models that can generalize across centers and patients to inspire the next generation of closed-loop neurostimulation.

In the future, our platform may provide the foundation for network specific and brain-state dependent closed-loop neurostimulation approaches that dissociate symptoms⁴⁶, side-effects¹ and volitional behavior^4,43 to offer the best therapy for the individual situation our patients are facing. To give a practical example, symptom specific brain networks have recently been discovered for STN-DBS in PD that differentially underlie improvement in tremor, bradykinesia, rigidity and gait disturbance⁴⁷. The network description however, is static in nature, while symptoms wax and wane. Brain signal decoding may help these networks to come to life in a dynamic closed-loop neurostimulation approach by informing both decoding and stimulation models of symptom specific circuits to optimize symptom decoding and adaptation of stimulation right at the time they occur.

In conclusion, py_neuromodulation provides a novel open-source platform that has the potential to democratize and standardize brain signal decoding in the development of next generation neurotechnology for closed-loop neurostimulation with clinical brain computer interfaces.

The open-source brain signal decoding platform py_neuromodulation and its’ algorithms were developed in the Python programming language. All code and documentation is openly accessible as a Python package (licensed under an open-source-compliant MIT license). The module supports Python version 3.10 or later and is maintained on GitHub (https://github.com/neuromodulation/py_neuromodulation; see the Code availability statement). The GitHub repository includes instructions for installing and contributing to the package, and the documentation materials that are further hosted on (https://neuromodulation.github.io/py_neuromodulation/). The platform handles real-time data streams from neural processors equally to offline data files in Brain Imaging Data Structure format (BIDS)⁴⁸, i.e. data can be streamed from the API of a neural processor or from an offline stream that is simulated from datafiles. This secures equal performance of brain signal decoders in offline and real-time situations, readymade for direct clinical adoption. Preprocessing, re-referencing, normalization and extraction of eight FFT multiband features for state predictions from a single channel on contemporary hardware (Intel i7 laptop computer) takes less than 10 ms.

py_neuromodulation parametrization

Initialization of data streams

py_neuromodulation requires the initialization of an offline or online stream object with the sampling frequency of the recorded data and two parametrization files: nm_channels and nm_settings (see for parametrization of available features Supplementary Table 1, preprocessing Supplementary Table 6 and postprocessing Supplementary Table 7). The nm_channels file provides channel specific information such as re-referencing, decoding target selection, and channel renaming (Supplementary Table 8). It builds upon the channels.tsv file of the BIDS specification for intracranial electroencephalography⁴⁸ and adopts parameters defined therein such as channel name, type and status. The nm_settings file specifies parameters for feature estimation and pre- and postprocessing. Both files can either be created during runtime or be loaded from disk by providing paths to a nm_channels.csv and nm_settings.json file. After initialization, all settings are tested, and the stream object’s run function can be called for batch-wise feature calculation. For offline applications, the stream generates sequential data batches that are processed to simulate a real-time setting. The size of the data batch is specified in nm_settings. In the simplest case, the stream can then be given the entire array of recorded data. The platform additionally provides an interface to automatically load data and metadata such as sampling frequency, channel names and types, and electrode locations from files in BIDS format using MNE-BIDS⁴⁹. When data is directly streamed from a neural device, the same procedure can be called to process arriving data batches in precisely the same way as done for offline analysis and training. When the streams’ run function is terminated, all feature and parametrization files are saved for further analysis. Various visualization and machine learning analysis functions can then be utilized through the nm_analysis module.

Feature estimation

A variety of features were implemented that can capture different characteristics of neural time series data from multiple data analysis domains, which may provide complementary information on brain states and behavior⁸. For extraction of oscillatory features (FFT, STFT, Finite Impulse Response (FIR) bandpass filtering) is implemented in py_neuromodulation with frequency adapted time windows to optimize information content. Spectral parametrization can separate periodic and aperiodic components of power spectra that can provide differential information on neural processing⁷. We developed a wrapper for the FOOOF-toolbox that allows for computation of aperiodic and periodic parameters for real-time decoding. In the temporal domain, the presence, amplitude and duration of oscillatory bursts were shown to provide complementary information to average oscillatory power^50–53. For each feature window, mean burst duration, amplitude, burst rate per second and an in-burst state can be estimated for several frequency bands and different thresholds across a variable time window.

In addition to neural oscillations, temporal waveform shape has previously been demonstrated to reflect different pathological and physiological states^6,54. For multielectrode recordings, interregional functional oscillatory connectivity can provide important information for decoding applications⁵⁵. Therefore, a real-time compatible coherence computation across channels and frequency bands was implemented, as well as a wrapper for the MNE-Connectivity toolbox (https://mne.tools/mne-connectivity/stable/index.html).

Temporal waveform shape estimation is implemented in py_neuromodulation in a batch-wise manner for signal troughs and peaks for which the prominence measures the mean amplitude difference of the trough and the surrounding peaks (or peak with surrounding troughs):

$${V}_{prominence}= \left|\frac{{V}_{peak-left}+{V}_{peak-right}}{2}\right|-{V}_{trough}$$

The sharpness measures the voltage deflection of each trough/peak with respect to the voltage amplitude at time points preceding and following 5 ms:

$${V}_{sharpness}=\frac{\left({V}_{trough}-{V}_{trough-5 ms}\right)+\left({V}_{trough}-{V}_{trough+5 ms}\right)}{2}$$

Additionally, the interval between troughs/peaks can be calculated, as well as the width of the surrounding peaks/troughs, and the rise and decay time and steepness. The maxima of resulting measures per time window can then be used for model training and decoding. Nonlinear measures for dynamical systems (nolds) estimate trends in the data and characterize different metrics of the time series dynamics. The features are obtained through a wrapper for the nolds toolbox, and can calculate sample entropy, correlation dimension, Lyapunov exponent, or detrended fluctuation analysis of raw or bandpass filtered data⁵⁶. Additionally, Hjorth features can be calculated for raw data or bandpass-filtered data separately for different frequency bands⁵⁷. Kalman filters can be used for post-processing of oscillatory features to reduce noise^45,58,59.

Feature analysis and decoding

In the nm_analysis module, features can be visualized across time for specific recording time windows or averaged for a target signal condition (Fig. 1c). Electrode locations can be plotted on a cortical surface and color coded with obtained features or performances. Different decoding methods were implemented in the nm_decode module. Machine learning methods can be evaluated for different validation strategies for individual or combined channels, e.g. as supported by scikit-learn⁹. Multiple samples can be combined for feature analysis for e.g. Wiener filter methods⁴⁵. Principle component analysis or Canonical correlation analysis can be used for dimensionality reduction. Features can, in addition, be selected through the Maximum Relevance and Minimum Redundancy method (https://github.com/smazzanti/mrmr)⁶⁰. Unbalanced datasets can be resampled through random over- or undersampling using the imbalanced-learn framework⁶¹. Since many decoding applications require performance evaluation on not only a sample, but a consecutive group-wise level, detection accuracies can be calculated for a minimum modifiable number of high consecutive predictions, as implemented for movement detection (Fig. 2).

Decoding without patient individual training

Decoding from brain implants without patient individual training needs to account for large individual variance of electrode locations and anatomy¹⁶. We implemented three methods that can achieve this: grid point interpolation, spatial correlation of whole-brain network for channel selection and contrastive representation learning using CEBRA. Grid-point decoding relies on a predefined cortical or subcortical grid definition in a common space (e.g. the non-linear asymmetric version of the Montreal Neurological Institute space MNI152NLin2009bAsym). Data from individual electrode contacts ${x}_{m}$ are interpolated to common grid points ${x}_{gridpoint n}$, and weighted by their normed distances $dist\left(n,m\right)$ to each grid point:

$${x}_{gridpoint n}=\frac{1}{\sum _{contact m=1}^{M}{dist(n,m)}^{-1}}\sum _{contact m =1}^{M}{x}_{m}{dist(n,m)}^{-1}$$

All electrode contacts $M$ are selected through a maximum distance threshold between each electrode contact $m$ and the respective grid point $n$. In this way, localized information is grouped to grid points that are shared across patients. Decoding without patient individual training can then be performed using the grid point data. Therefore, this approach identifies a projection to a standardized space that is comparable to EEG layouts.

Alternatively, single ECoG or depth electrode channels can be selected through optimal connectivity. Functional and structural connectivity measures can for example be calculated within the Lead-Mapper tool of the Lead-DBS toolbox³⁶. An optimal connectomic decoding network map is then constructed through voxel-wise correlation of connectivity profiles with decoding performances from offline training¹¹. This allows prospective channel selection even in previously unseen subjects by comparing the network fingerprints of all available channels to identify the one that is most similar to the optimal connectomic decoding network map that was informed by other subjects.

The selected channels can then be used for learning an embedding based on the py_neuromodulation feature space using CEBRA¹³. Contrastive learning can thus be used to train a neural network to minimize distances between behaviorally and temporally close samples and maximize distances from temporally distinct samples. The obtained embeddings can be compared by computing a linear mapping from one embedding to another, resulting in a “consistency map”⁶². A binned histogram can be computed to take target information into consideration and correlate the resulting embeddings. This provides a powerful ground for across-patient predictions for human brain implants.

Use-case specific patient details and decoding methods

Movement decoding from ECoG signals in epilepsy and PD patients undergoing STN-DBS

Invasive brain signals were analyzed from 56 subjects across four cohorts. The data acquisition methods varied across the four movement decoding cohorts. Patients with Parkinson’s disease (Berlin, Beijing, Pittsburgh) underwent additional ECoG recordings in the context of neurosurgery for subthalamic DBS implantation. Motor sign severity was assessed preoperatively using Unified Parkinson’s disease rating scales (UPDRS-III total score) for all patients. Berlin subjects (12 PD patients, 5 females) were recorded postoperatively through externalized six or twelve contact ECoG electrodes (Ad-Tech 1x6 layout, contact area: 12.56 mm²; Ad-Tech 1x6 layout, double sided, contact area: 12.56 mm²) using a Neuro Omega (Alpha Omega) acquisition system (Berlin subjects 1–2) or with the SAGA amplifier (TMSi; Berlin subjects 3 to 12). In total, there were 38 recordings, of which 17 were performed ON dopaminergic medication (11 patients). All patients were instructed to perform voluntary hand movements with the upper limb contralateral to the hemisphere of ECoG electrode placement. In a sub-cohort of six patients bilateral monopolar STN-DBS was delivered with clinically effective stimulation settings, as determined by monopolar review (stimulation frequency: 130 Hz; pulse width: 60 µs; mean amplitude: 2 mA). Data from the Beijing cohort was also recorded in PD patients undergoing STN-DBS using externalized ECoG electrodes (n = 10, 8 females) using a JE-212 amplifier (Nihon Kohden). Patients carried out a voluntary button press task. Six subjects were measured in the medication ON condition (6 recordings). Data in the Pittsburgh cohort (n = 15 patients, 3 females, 73 recordings) was recorded intraoperatively during STN-DBS implantation using a Grapevine neural interface processor (Ripple). All patients were in medication OFF condition and performed a cued Go/No-Go task that was previously described⁶³. The electrode setup varied across Pittsburgh patients: Ad-Tech electrodes using 6 (layout: 1x6; contact area: 12.56 mm²), 8 (layout: 1x8, contact area: 12.56 mm²), or 28 contacts (layout: 2x16, contact area: 3.14 mm²) were used. All data from epilepsy subjects were measured in the Epilepsy Monitoring Unit (EMU) (n = 18, 7 females) using Synamps2 amplifiers (Compumedics Neuroscan) at the University of Washington and Washington University in St. Louis. The data was previously made publicly available in the publication “A library of human electrocorticographic data and analyses”¹⁴ and was also analyzed in a former study⁶⁴. Here we used the “motor_basic” dataset including hand movements with “synchronous flexion and extension of all fingers”.

For prospective model validation and to showcase the real-time applicability of the py_neuromodulation platform, we recruited an additional PD patient at the Movement Disorder and Neuromodulation Unit at Charité Universitätsmedizin Berlin (age: 65 years, female). ECoG recordings were measured from an Ad-Tech electrode (1x6 contact layout, contact area: 12.56 mm²) and a SAGA (TMSi) amplifier. The patient performed voluntary wrist rotations as described for the Berlin cohort above.

To demonstrate movement decoding across centers, movement types, and diseases, we computed features and applied standardized machine learning decoding methods across the four cohorts described above. Data from all subjects was streamed in batches and the following processing chain was applied to every batch: All data were resampled to 1 kHz, re-referenced by subtracting the common average, and FFT features were calculated for different frequency bands: θ (4–8 Hz), α (8–12 Hz), low β (13–20 Hz), high β (20–35 Hz), low γ (60–80 Hz), high γ (90–200 Hz), and high-frequency activity (200–400 Hz) at a temporal resolution of 100 ms. All features were z-score normalized across the past 30 s and clipped at minus and plus three, to normalize the feature range across subjects and cohorts. Ridge-regularized logistic regression models were trained based on motor output that was provided as the target channel using balanced class-weights for prediction of movement presence using scikit-learn⁹. Performance on each channel was assessed using the balanced accuracy metric, which considers imbalances in target distributions. Three-fold cross-validation was employed on consecutive data segments without shuffling individual samples, which can lead to data leakage through temporal correlation of feature estimates.

For across-patient decoding, electrode contact localization was performed or available from neuroimaging as a prerequisite to the three different approaches described above: first, the features were projected to a custom 38-point sensorimotor grid in MNI space with a maximum channel to grid-point distance of 20 mm. Logistic regression classifiers were trained on grid points instead of channels for each patient. This circumvents the requirement to select single channels at the patient level. Alternatively, in a second approach, channel selection was performed using connectomics. A spherical seed (4 mm radius) was used as a region of interest to compute the connectivity profiles for each recording contact using Lead-Connectome Mapper in the Lead-DBS toolbox³⁶. An optimal connectomic network map for movement decoding was calculated using voxel-wise correlations of channel-specific balanced accuracies connectivity profiles. py_neuromodulation was then used to validate the computed connectomic decoding network map across different cross-validation strategies: leave one subject out within cohorts, leave one subject out across cohorts, and leave one cohort out cross-validation. CEBRA contrastive learning was used with the “offset10-model” five-layer convolutional neural network (32 hidden units for the first layer, followed by three convolutional skip layers with each 32 hidden units and a four-dimensional convolutional output layer). The skip layers resembled a bottleneck across the temporal filter dimension from 10 to 3 samples. A Gaussian Error Linear Unit (GELU) activation function was used for each layer, and normalization was applied to the output layer. The “auto” temperature mode was specified with a learning rate of 0.005. 1000 training iterations were used with a batch size of 100. Positive sampling with the InfoNCE loss function was performed by sampling time samples within a “time_offset” of one and samples respective to the auxiliary movement variable as defined in the target channel. To approximate the movement kinematics, a gaussian filter with sigma 1.5 was used to smooth each label, and color-code the exemplar embeddings. All code required for reproduction of the movement decoding analysis was made publicly available in a dedicated GitHub repository (https://github.com/timonmerk/AcrossCohortMovementDecoding).

All preprocessing, feature estimation, and postprocessing steps of py_neuromodulation were optimized for real-time application. Raw data visualization in real-time is realized using BrainStreamingLayer (https://github.com/bsl-tools/bsl), and features as well as model predictions are visualized using Timeflux⁶⁵. All features are saved using Timeflux in the hdf5 format for post-hoc training and model comparison. The nm_settings and nm_channels specification files can be identical for offline and online analysis. To demonstrate its utility in the movement decoding scenario for a newly recruited patient, the offline data batch generator was replaced by the API of a hardware acquisition system (TMSi SAGA) to stream signal batches directly from the neural implant in real-time. In an initial movement recording run (as described for the remaining Berlin cohort), a pretrained model from all other Berlin cohort patients was applied and real-time decoding performance was objectified for the patient naïve “plug & play” application. Simultaneously, data was stored, and used for offline training of a patient individual model after the first run was concluded. Performance of this patient individual model was validated in real-time in the subsequent run. In a third run a pretrained model was tested, which was based on the single subject with best training outcome from the Berlin cohort (sub_002) to prove that patient to patient prediction is viable at the single subject level, even in prospective real-time applications. All code required for reproduction of the real-time implementation using the TMSi SAGA acquisition system was made publicly available in a GitHub branch of the py_neuromodulation repository (https://github.com/neuromodulation/py_neuromodulation/tree/realtime_decoding_analog_tr).

Emotion decoding in patients with treatment resistant depression undergoing SCC-DBS

Data from treatment-resistant depression subjects (n = 8, 3 females) engaging in an emotion picture viewing task was retrospectively analyzed²⁰. Patients were resistant to treatment of cognitive behavioral therapy, and or electroconvulsive therapy and underwent a clinical trial at Charité – Universitätsmedizin Berlin. Supplementary Table 4 displays the following details: age, preoperative and 24 months DBS BDI follow up scores as reported in⁶⁶. Data was recorded using DBS electrodes (Medtronic model 3387). Through post-operative fMRI images, electrode locations were estimated through the Lead-DBS software and coregistered to MNI space³⁶. Patients were participating in a passive picture viewing task showing pleasant, unpleasant and neutral stimuli of the International Affective Picture System (IAPS)⁶⁷. Pictures were presented with a duration of 1 s and with a randomized inter-stimulus interval of 6–8 s. Additionally, pictures were matched for mean neutral valence and arousal. All recordings were referenced bipolarly and band-pass filtered between 0.5 and 250 Hz through the recording hardware (D360 Digitimer Ltd).

Data with epochs containing different emotional stimuli were concatenated to build the simulated data stream of invasive recordings from patients with treatment-resistant depression. The time periods containing stimulus presentation were labeled according to the emotion class. Since data was already hardware bipolar re-referenced, no rereferencing was specified. Raw data batches from the signal stream were notch-filtered, and FFT features (θ (4–8 Hz), α (8–12 Hz), low β (13–20 Hz), high β (20–35 Hz), low γ (60–80 Hz), high γ (90–200 Hz), and high-frequency activity (200–400 Hz)), as well as temporal waveform shape features (max prominence, mean interval, max sharpness, mean decay time, mean rise time) were calculated. Performances for different classification problems were obtained using a ridge-regularized and class-weight balanced logistic regression model as above.

Healthy subject functional and diffusion-weighted structural MRI connectivity profiles, as well as structural connectivity using fiber-tracts, were calculated for each recording contact location (region of interest seed with 4 mm radius) using openly accessible connectomes (Yeo et al 2011⁶⁸ for fMRI (n = 1000) and Human Connectome Project data⁶⁹ for dMRI and fiber-tracking (n = 985)). This allowed to calculate optimal connectomic emotion decoding network maps that were validated in leave one channel and leave one subject out cross validation approaches (Extended Data Fig. 1). The Lead Connectome Mapper and fiber-filtering was performed using the Lead-DBS toolbox^36,69. Leave one channel and leave one subject out cross-validation approaches were implemented in py_neuromodulation. All code required for reproduction of the emotion decoding analysis was made publicly available in a GitHub repository (https://github.com/timonmerk/TRD_pynm).

Seizure decoding in epilepsy patients with responsive neurostimulation implants

Intracranial data from 9 patients was obtained through the Biophysically Rational Analysis for Informed Neural Stimulation (BRAINStim) platform developed from the Brain Modulation Lab³². Patient individual information such as number of lead depths and ECoG strip electrodes, recording locations, time since RNS implantation, and seizure reduction scores are displayed in Supplementary Table 5. The RNS device can be programmed for storage of scheduled recordings, as well as high risk seizure recordings (called Long Episodes). The concatenated recordings were further annotated for seizure presence and aligned with detection and stimulation events that the BRAINStim platform further logs from the Patient Data Management System (PDMS).

Offline parametrization for seizure presence decoding

The previously published BRAINStim pipeline allows to access streamed recordings from the responsive neurostimulation (RNS) implant³². Electrophysiological data from either lead-depth or ECoG strip electrodes was stored across multiple months in addition to stimulation and detection events in a database. Each recording was linked to a set of stimulation and detection events, as well as seizure presence annotations from an expert neurologist (V.K.). py_neuromodulation was used to first assess the performance of different detection settings using the bandpass detector. FFT features were calculated for different frequency bands (θ (4–8 Hz), α (8–12 Hz), low β (13–20 Hz), high β (20–35 Hz), low γ (35–60 Hz), broadband (20–120 Hz)) with a feature sampling rate of 1 Hz and segment length of 1000 ms. Feature normalization was omitted to ensure feature discriminability across ictal and non-ictal recordings. Since data was hardware bipolar referenced, no additional software re-referencing was performed. Data segments from two artifact components were identified and excluded from further analysis: First, stimulation events result in the RNS stored data as a “flatline” stationary artifact, which does not contain electrophysiological data. Second, suboptimal amplifier gain settings can result in a “clipping” artifact, where all data is clipped to analog-to-digital converter respective maximum bounds. Both artifact types were identified if the voltage derivative did not change in a time range of at least twelve consecutive samples (48 ms). Since stimulation events induce additionally high amplitude fluctuations in the RNS stored recordings, two seconds following each stimulation pulse were annotated to contain artifacts.

A performance of seizure presence decoding was obtained by optimizing different detection parameters. A recording was classified as seizure if for a defined minimum duration the voltage amplitude crossed a certain threshold. In a grid search, the minimum duration was tested across 50 values from 100 to 5000 ms in steps of 100 ms; and the amplitude was sampled from 20 equally spaced values within the minimum and maximum voltage range. Additionally, each feature could be inverted by the RNS implant, resulting in a three-dimensional grid-search across minimum duration, minimum amplitude, and inversion. The obtained seizure classification was measured as a F1-score and compared against the RNS programmed bandpass-detector prediction performance.

In the next step, several additional py_neuromodulation features were computed: raw signal line length, temporal waveform shape features for 5–30 Hz and 5–60 Hz filtered data (temporal waveform-shape features: width, interval, rise time, decay time, rise steepness, decay steepness, prominence, interval, sharpness), bursting features for a 75 percent threshold within a duration of 30 s for signals filtered in the low beta, high beta, and high gamma frequency bands (bursting features: duration, amplitude, burst rate per second, in-burst state), spectral aperiodic components (exponent and offset), and coherence in the high beta and low gamma frequency bands between recording contacts of each electrode as well as across electrodes. Using a non-shuffled three-fold cross-validation, ridge-regularized logistic regression models, support vector machines, and gradient boosted decision trees using the XGBOOST framework were tested⁷⁰. In this manner the potential performance gain of additional features in combination with machine learning models was evaluated. All code required for reproduction of the optimal RNS seizure decoding analysis was made publicly available in a GitHub repository (https://github.com/timonmerk/rns_pynm_optimal_detection_params).

Ethics declaration

The research and brain signal recordings presented in this manuscript were performed according to the standards set by the declaration of Helsinki and after approval of the local independent ethics review board at the included academic centers (Berlin, Beijing, Pittsburgh, Washington). All patients provided informed consent to participate in the respective research. For data from Berlin, the studies were approved by the ethics committee at Charité Universitätsmedizin Berlin (EA2/129/17). The data was collected, stored and processed in compliance with the General Privacy Protection Regulation of the European Union. The data collection for Beijing was approved by the independent review board of Beijing Tiantan Hospital (KY 2018-008-01), registered in the Chinese Clinical Trial Registry (ChiCTR1900026601) and conducted under the supervision of an authoritative third party (China National Clinical Research Center for Neurological Diseases). Data from Washington were obtained under experimental protocols approved by the Institutional Review Board of the University of Washington (#12193) as indicated in the original data repository that is openly available (https://searchworks.stanford.edu/view/zk881ps0522 “A Library of Human Electrocorticographic Data and Analyses”¹⁴, first published in⁶⁴).

Data availability

Data for movement decoding are available for the Pittsburgh (https://doi.org/10.7910/DVN/IO2FLM) and Washington (https://purl.stanford.edu/zk881ps0522) cohorts. Data from Berlin can be made available conditionally to data sharing agreements in accordance with data privacy statements signed by the patients within the legal framework of the General Data Protection Regulation of the European Union. Data from Beijing can be made available upon reasonable request after approval of the local independent review board of Beijing Tiantan Hospital. All trained models are made openly available for direct implementation in clinical research via the GitHub code repositories.

Code availability

All code and documentation for py_neuromodulation is openly accessible as a Python package, licensed under an open-source-compliant MIT license via https://github.com/neuromodulation/py_neuromodulation. Documentation is hosted on https://neuromodulation.github.io/py_neuromodulation/. All code of the real-time implementation using the TMSi SAGA acquisition system was made publicly available in a GitHub branch of the py_neuromodulation repository (https://github.com/neuromodulation/py_neuromodulation/tree/realtime_decoding_analog_tr). All code of the offline movement decoding analysis was made publicly available through https://github.com/timonmerk/AcrossCohortMovementDecoding. All code of the emotion decoding analysis was made publicly available through https://github.com/timonmerk/TRD_pynm. All code of the RNS parameter analysis was made publicly available through https://github.com/timonmerk/rns_pynm_optimal_detection_params.

Funding

The study was funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project-ID 424778371 and a US-German Collaborative Research in Computational Neuroscience (CRCNS) grant from the German Federal Ministry for Research and Education and NIH (R01NS110424). WJN received funding from the European Union (ERC, ReinforceBG, project 101077060). AAK, has received grant funding from the Lundbeck foundation. AH received funding from Deutsches Zentrum für Luft- und Raumfahrt (DynaSti grant within the EU Joint Programme Neurodegenerative Disease Research, JPND), the National Institutes of Health (R01 13478451, 1R01NS127892-01 & 2R01 MH113929) as well as the New Venture Fund (FFOR Seed Grant). Some data used in the preparation of this article were obtained from the PPMI database (https://www.ppmi-info.org/). For up-to-date information on the study, visit https://www.ppmi-info.org/. PPMI, a public–private partnership, is funded by the Michael J Fox Foundation for Parkinson’s Research. For funding partners, see https://www.ppmi-info.org/fundingpartners.

Declaration of Competing Interest

R.M.R. has served as a consultant for NeuroPace.
R.L. reports personal fees from Medtronic.
A.A.K. reports personal fees from Medtronic and Boston Scientific.
G.H.S. reports personal fees from Medtronic, Boston Scientific, and Abbott.

Acknowledgements

We would like to thank all patients who made this research possible.

Gilron, R. et al. Long-term wireless streaming of neural recordings for circuit discovery and adaptive stimulation in individuals with Parkinson’s disease. Nat Biotechnol 1–8 (2021) doi:10.1038/s41587-021-00897-5.
Khambhati, A. N., Shafi, A., Rao, V. R. & Chang, E. F. Long-term brain network reorganization predicts responsive neurostimulation outcomes for focal epilepsy. Sci Transl Med 13, eabf6588 (2021).
Lorach, H. et al. Walking naturally after spinal cord injury using a brain–spine interface. Nature 618, 126–133 (2023).
Opri, E. et al. Chronic embedded cortico-thalamic closed-loop deep brain stimulation for the treatment of essential tremor. Science translational medicine 12, (2020).
Scangos, K. W. et al. Closed-loop neuromodulation in an individual with treatment-resistant depression. Nat Med 27, 1696–1700 (2021).
Cole, S. R. & Voytek, B. Brain Oscillations and the Importance of Waveform Shape. Trends Cogn Sci 21, 137–149 (2017).
Donoghue, T. et al. Parameterizing neural power spectra into periodic and aperiodic components. Nat Neurosci 23, 1655–1665 (2020).
Merk, T. et al. Machine learning based brain signal decoding for intelligent adaptive deep brain stimulation. Experimental Neurology 351, 113993 (2022).
Pedregosa, F. et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011).
Fries, P. Rhythms For Cognition: Communication Through Coherence. Neuron 88, 220–235 (2015).
Merk, T. et al. Electrocorticography is superior to subthalamic local field potentials for movement decoding in Parkinson’s disease. eLife 11, e75126 (2022).
de Hemptinne, C. et al. Therapeutic deep brain stimulation reduces cortical phase-amplitude coupling in Parkinson’s disease. Nat Neurosci 18, 779–786 (2015).
Schneider, S., Lee, J. H. & Mathis, M. W. Learnable latent embeddings for joint behavioural and neural analysis. Nature 617, 360–368 (2023).
Miller, K. J. A library of human electrocorticographic data and analyses. Nat Hum Behav 3, 1225–1235 (2019).
Crowell, A. L. et al. Oscillations in sensorimotor cortex in movement disorders: An electrocorticography study. Brain 135, 615–630 (2012).
Peterson, S. M., Steine-Hanson, Z., Davis, N., Rao, R. P. N. & Brunton, B. W. Generalized neural decoders for transfer learning across participants and recording modalities. J. Neural Eng. 18, 026014 (2021).
Oord, A. van den, Li, Y. & Vinyals, O. Representation Learning with Contrastive Predictive Coding. Preprint at https://doi.org/10.48550/arXiv.1807.03748 (2019).
Figee, M. & Mayberg, H. The future of personalized brain stimulation. Nat Med 27, 196–197 (2021).
Sendi, M. S. E. et al. Intraoperative neural signals predict rapid antidepressant effects of deep brain stimulation. Transl Psychiatry 11, 1–7 (2021).
Huebl, J. et al. Processing of emotional stimuli is reflected by modulations of beta band activity in the subgenual anterior cingulate cortex in patients with treatment resistant depression. Social Cognitive and Affective Neuroscience 11, 1290–1298 (2016).
Baldermann, J. C. et al. Connectivity Profile Predictive of Effective Deep Brain Stimulation in Obsessive-Compulsive Disorder. Biological Psychiatry 85, 735–743 (2019).
Li, N. et al. A unified connectomic target for deep brain stimulation in obsessive-compulsive disorder. Nat Commun 11, 3364 (2020).
Siddiqi, S. H. et al. Brain stimulation and brain lesions converge on common causal circuits in neuropsychiatric disease. Nat Hum Behav 5, 1707–1716 (2021).
Irmen, F. et al. Left Prefrontal Connectivity Links Subthalamic Stimulation with Depressive Symptoms. Annals of Neurology 87, 962–975 (2020).
Fox, M. D. et al. Resting-state networks link invasive and noninvasive brain stimulation across diverse psychiatric and neurological diseases. Proceedings of the National Academy of Sciences 111, E4367–E4375 (2014).
Riva-Posse, P. et al. Autonomic arousal elicited by subcallosal cingulate stimulation is explained by white matter connectivity. Brain Stimul 12, 743–751 (2019).
Xiao, J. et al. Decoding Depression Severity from Intracranial Neural Activity. Biol Psychiatry S0006-3223(23)00048–3 (2023) doi:10.1016/j.biopsych.2023.01.020.
Makris, N. et al. Decreased volume of left and total anterior insular lobule in schizophrenia. Schizophr Res 83, 155–171 (2006).
Nair, D. R. et al. Nine-year prospective efficacy and safety of brain-responsive neurostimulation for focal epilepsy. Neurology 95, e1244–e1256 (2020).
Hixon, A. M. et al. RNS modifications to eliminate stimulation-triggered signs or symptoms (STS): Case series and practical guide. Epilepsy Behav 112, 107327 (2020).
Jobst, B. C. et al. Brain-responsive neurostimulation in patients with medically intractable seizures arising from eloquent and other neocortical areas. Epilepsia 58, 1005–1014 (2017).
Sisterson, N. D., Wozny, T. A., Kokkinos, V., Constantino, A. & Richardson, R. M. Closed-Loop Brain Stimulation for Drug-Resistant Epilepsy: Towards an Evidence-Based Approach to Personalized Medicine. Neurotherapeutics 16, 119–127 (2019).
Peterson, V. et al. Deep net detection and onset prediction of electrographic seizure patterns in responsive neurostimulation. Epilepsia (2023) doi:10.1111/epi.17666.
Grinenko, O. et al. A fingerprint of the epileptogenic zone in human epilepsies. Brain 141, 117–131 (2018).
Zaher, N. et al. Ictal Onset Signatures Predict Favorable Outcomes of Laser Thermal Ablation for Mesial Temporal Lobe Epilepsy. Front Neurol 11, 595454 (2020).
Neudorfer, C. et al. Lead-DBS v3.0: Mapping deep brain stimulation effects to local anatomy and global networks. NeuroImage 268, 119862 (2023).
Weigand, A. et al. Prospective Validation That Subgenual Connectivity Predicts Antidepressant Efficacy of Transcranial Magnetic Stimulation Sites. Biol Psychiatry 84, 28–37 (2018).
Stangl, M. et al. Boundary-anchored neural mechanisms of location-encoding for self and others. Nature 589, 420–425 (2021).
Topalovic, U. et al. A wearable platform for closed-loop stimulation and recording of single-neuron and local field potential activity in freely moving humans. Nat Neurosci 26, 517–527 (2023).
Moses, D. A. et al. Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria. New England Journal of Medicine 385, 217–227 (2021).
Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M. & Shenoy, K. V. High-performance brain-to-text communication via handwriting. Nature 593, 249–254 (2021).
Basu, I. et al. Closed-loop enhancement and neural decoding of cognitive control in humans. Nat Biomed Eng 1–13 (2021) doi:10.1038/s41551-021-00804-y.
He, S. et al. Closed-Loop Deep Brain Stimulation for Essential Tremor Based on Thalamic Local Field Potentials. Movement Disorders 36, 863–873 (2021).
Moses, D. A., Leonard, M. K., Makin, J. G. & Chang, E. F. Real-time decoding of question-and-answer speech dialogue using human cortical activity. Nature Communications (2019) doi:10.1038/s41467-019-10994-4.
Shah, S. A., Tan, H., Tinkhauser, G. & Brown, P. Towards Real-Time, Continuous Decoding of Gripping Force From Deep Brain Local Field Potentials. IEEE transactions on neural systems and rehabilitation engineering: a publication of the IEEE Engineering in Medicine and Biology Society 26, 1460–1468 (2018).
Lauro, P. M. et al. Concurrent decoding of distinct neurophysiological fingerprints of tremor and bradykinesia in Parkinson’s disease. Elife 12, e84135 (2023).
Hollunder, B. et al. Toward personalized medicine in connectomic deep brain stimulation. Prog Neurobiol 210, 102211 (2022).
Holdgraf, C. et al. iEEG-BIDS, extending the Brain Imaging Data Structure specification to human intracranial electrophysiology. Sci Data 6, 102 (2019).
Appelhoff, S. et al. MNE-BIDS: Organizing electrophysiological data into the BIDS format and facilitating their analysis. Journal of Open Source Software 4, 1896 (2019).
Feingold, J., Gibson, D. J., DePasquale, B. & Graybiel, A. M. Bursts of beta oscillation differentiate postperformance activity in the striatum and motor cortex of monkeys performing movement tasks. Proc Natl Acad Sci U S A 112, 13687–13692 (2015).
Lofredi, R. et al. Subthalamic beta bursts correlate with dopamine-dependent motor symptoms in 106 Parkinson’s patients. NPJ Parkinsons Dis 9, 2 (2023).
Tinkhauser, G. et al. The modulatory effect of adaptive deep brain stimulation on beta bursts in Parkinson’s disease. Brain 140, 1053–1067 (2017).
Torrecillos, F., He, S., Kühn, A. A. & Tan, H. Average power and burst analysis revealed complementary information on drug-related changes of motor performance in Parkinson’s disease. NPJ Parkinsons Dis 9, 93 (2023).
Cole, S. R. et al. Nonsinusoidal Beta Oscillations Reflect Cortical Pathophysiology in Parkinson’s Disease. J Neurosci 37, 4830–4840 (2017).
Yao, L., Baker, J. L., Schiff, N. D., Purpura, K. P. & Shoaran, M. Predicting task performance from biomarkers of mental fatigue in global brain activity. J Neural Eng 18, (2021).
Schölzel, C. Nonlinear measures for dynamical systems. (2019) doi:10.5281/zenodo.3814723.
Hjorth, B. EEG analysis based on time domain properties. Electroencephalography and Clinical Neurophysiology (1970) doi:10.1016/0013-4694(70)90143-4.
Eliseyev, A. & Aksenova, T. Penalized Multi-Way Partial Least Squares for Smooth Trajectory Decoding from Electrocorticographic (ECoG) Recording. PLoS One 11, e0154878 (2016).
Yao, L., Brown, P. & Shoaran, M. Improved detection of Parkinsonian resting tremor with feature engineering and Kalman filtering. Clinical Neurophysiology (2020) doi:10.1016/j.clinph.2019.09.021.
Zhao, Z., Anand, R. & Wang, M. Maximum Relevance and Minimum Redundancy Feature Selection Methods for a Marketing Machine Learning Platform. Preprint at https://doi.org/10.48550/arXiv.1908.05376 (2019).
Lemaitre, G., Nogueira, F. & Aridas, C. K. Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning. Preprint at https://doi.org/10.48550/arXiv.1609.06570 (2016).
Roeder, G., Metz, L. & Kingma, D. P. On Linear Identifiability of Learned Representations. Preprint at https://doi.org/10.48550/arXiv.2007.00810 (2020).
Kondylis, E. D. et al. Movement-related dynamics of cortical oscillations in Parkinson’s disease and essential tremor. Brain 139, 2211–2223 (2016).
Miller, K. J. et al. Spectral changes in cortical surface potentials during motor movement. J Neurosci 27, 2424–2432 (2007).
Clisson, P., Bertrand-Lalo, R., Congedo, M., Victor-Thomas, G. & Chatel-Goldman, J. Timeflux: an open-source framework for the acquisition and near real-time processing of signal streams. in BCI 2019-8th International Brain-Computer Interface Conference (2019).
Merkl, A. et al. Deep brain stimulation of the subcallosal cingulate gyrus in patients with treatment-resistant depression: A double-blinded randomized controlled study and long-term follow-up in eight patients. J Affect Disord 227, 521–529 (2018).
Lang, P. J., Bradley, M. M. & Cuthbert, B. N. International affective picture system (IAPS): affective ratings of pictures and instruction manual. (NIMH, Center for the Study of Emotion & Attention, 2005).
Thomas Yeo, B. T. et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of Neurophysiology 106, 1125–1165 (2011).
Van Essen, D. C. et al. The WU-Minn Human Connectome Project: an overview. Neuroimage 80, 62–79 (2013).
Chen, T. & Guestrin, C. XGBoost: A Scalable Tree Boosting System. in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (Association for Computing Machinery, 2016). doi:10.1145/2939672.2939785.

Yes there is potential Competing Interest. R.M.R. has served as a consultant for NeuroPace. R.L. reports personal fees from Medtronic. A.A.K. reports personal fees from Medtronic and Boston Scientific. G.H.S. reports personal fees from Medtronic, Boston Scientific, and Abbott.

SupplementaryTable5RNSPatients.xlsx
Table 5
SupplementaryVideo1.mp4
Real-time movement decoding of a Parkinson’s disease patient at Charité University Medicine Berlin without patient individual training
Extendeddata.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Invasive neurophysiology and whole brain connectomics for neural decoding in patients with brain implants

Status:

Version 1

Abstract

Figures

Main

Results

Discussion

Methods

Feature estimation

Feature analysis and decoding

Decoding without patient individual training

Use-case specific patient details and decoding methods

Movement decoding from ECoG signals in epilepsy and PD patients undergoing STN-DBS

Emotion decoding in patients with treatment resistant depression undergoing SCC-DBS

Seizure decoding in epilepsy patients with responsive neurostimulation implants

Offline parametrization for seizure presence decoding

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1