Terminology definition. The intricate nature of the individual components within the MAP necessitates a standardized terminology to eliminate potential confusion. In response, recent research has advocated for a comprehensive summary of terms and definitions pertinent to MAP.28 Expanding upon this effort, we present a detailed elucidation of the structural framework underlying MAP, which encompasses four primary components, platforms, modules, tasks and actions (Supplementary Figure S1).
The platform serves as the foundational framework that interconnects experimental resources, facilitating automated experiments to cater to the diverse needs of multiple users. The concept of platforms has been increasingly emphasized in recent advancements.29–31 Within a platform, modules represent the fundamental building blocks, each dedicated to executing a specific experimental process. Examples of modules include "BatchSynthesis", "FlowSynthesis", "SprayCoating", "Filtration" and "UV‒Vis". Each module encapsulates the requisite functionalities necessary for its designated experimental process.
A task delineates the granular steps comprising a specific experimental process within a module. For instance, tasks within the "BatchSynthesis" module include "PrepareContainer", "AddSolution", "Stir", "Heat", "Mix" and "React", among others. Actions, on the other hand, denote the discrete operations of the experimental devices required for task execution. Experimental devices such as robotic arms, pipettes and stirrers execute various actions to accomplish a task within a module. For example, the "PrepareContainer" task involves sequential actions by a robotic arm of (1) opening vial storage, (2) picking vials and (3) placing vials on a stirrer (Supplementary Figure S1).
Through this structured framework, a MAP harmoniously orchestrates all the components, empowering users to execute desired experiments with high precision and efficiency. This hierarchical terminology definition not only enhances clarity and comprehension within the field but also facilitates seamless communication and collaboration among researchers and practitioners working within the automated experimentation realm.
Architecture of OCTOPUS. The architectural blueprint of OCTOPUS is depicted in Fig. 1, which highlights and draws inspiration from the adaptive nature of the octopus. OCTOPUS, which comprises three integral components, the interface node, master node and module nodes, orchestrates the seamless execution of the experimental modules within the MAP.
The interface node serves as the nexus, bridging multiple users (or clients) in remote environments to the MAP infrastructure. Next, the master node plays a central role in managing a myriad of tasks and actions, employing a sophisticated suite of six components, a job scheduler, a task generator, a task scheduler, an action translator, an action executor and a resource manager. Finally, the module nodes of OCTOPUS are meticulously modularized to accommodate individual experimental processes, as illustrated in Fig. 1. Upon receiving experimental action directives from the master node, the module nodes initiate modularized robotic operations, encompassing a spectrum of actions such as robotic arm maneuvers, solution dispensation and stirrer activation, among others. The hierarchical structure of OCTOPUS (interface node → master node → module nodes) and the harmonious integration of these components elucidate the intricate workflow of the platform.
Job submission via the interface node and job scheduler of the master node. The interface node within OCTOPUS functions as a command line interface (CLI), enabling simultaneous job submissions from multiple clients, as illustrated in Fig. 2. Notably, recent advancements in the user interface design for MAP have primarily been tailored to receiving a single job within an OS.13,14,19–24 However, the ability to accommodate multiple job submissions concurrently is paramount in enhancing the efficiency of a MAP.25 Thus, the interface node, bolstered by a multithread pool, enables numerous clients to seamlessly request remote experiments across multiple applications, akin to a computing server (Fig. 2). Notably, OCTOPUS operates real chemical experiments rather than computational tasks, distinguishing it from traditional computing server systems32,33.
Clients are required to generate job scripts based on the JavaScript object notation (JSON) format, and examples of job scripts are provided in Supplementary Figure S2. These job scripts cater to a spectrum of client demands, ranging from manual experiments with predefined input conditions (e.g. a model with the name of "Manual" in Supplementary Figure S2a) to autonomous experiments enabled by AI-decision processes (e.g. model with the names of "BayesianOptimization", "DecisionTree" and "BayesianNeuralNetwork" in Figure S2b). The model names could be changed by the clients depending on the type of AI model built for experimental planning on the platform. Initially, the interface node takes responsibility for managing the client login process, enforcing stringent security policies through hash functions while awaiting client commands (Supplementary Figure S3). Subsequently, clients submit job scripts via CLI, leveraging portable batch system commands, with detailed examples provided in Supplementary Figure S4.32,33 This transformation of the local platform into a client/server-based ubiquitous platform has profound implications for research continuity, particularly in scenarios constrained by temporal and spatial limitations, such as those imposed by COVID-19.34,35
The master node of OCTOPUS embodies a central management system capable of orchestrating diverse experiments for multiple clients.18 Within the master node, the job scheduler, which comprises a job ID generator, job modeling unit, job storage and three distinct queues (waiting, executing and holding), plays a pivotal role in scheduling jobs for the execution of real experiments, as illustrated in Fig. 2. Upon job submission, the job ID generator assigns a unique identifier (ID) to each job, facilitating orderly processing based on submission order. This job ID is subsequently transferred to the job modeling unit, which generates specific job configurations based on the provided job script, comprising process recommendations (manual vs. autonomous), module and task sequences and experimental conditions. Subsequently, the job storage organizes the generated jobs, awaiting the triggering decision.
The triggering decision, determined by various factors, including module resource availability and experimental device status, directs the job execution sequence. If the job trigger decides to execute the next job, it transitions the job ID from the waiting queue to the executing queue for real experiment executions (Supplementary Figure S5). Moreover, a holding queue addresses the safety concerns inherent in surveillance-free MAP environments. In the event of severe safety hazards involving, for example, inflammable or hazardous chemicals,12 the platform automatically places the job on hold, ensuring user safety. Clients retain the autonomy to restart or terminate holding jobs via predefined commands. Several examples of job management (submission, status check, hold, restart and deletion) via CLI are provided in Supplementary Figure S6. We also provide Supplementary Video 1, where the process of logging in, and performing remote and simultaneous job submissions by two users in the interface node of OCTOPUS is recorded for clarity.
Job executions in the master node. Following the job submissions in the interface node, we present the job execution workflow in the master node, incorporating the seven components of job scheduler, task generator, task scheduler, action translator, action executor, resource manager and database, which are specifically tailored for closed-loop experiments, as illustrated in Fig. 3.
We empower the task generator to retrieve experimental device information, including physical specifications and the setup environment, from the resource manager. For instance, when preparing to execute the “AddSolution” task, the task generator accesses information detailing the device specifications and setup environments of stock solution types and concentrations, among others. Notably, the dynamic updating of experimental device information from the resource manager enhances operational flexibility (Supplementary Figures S7a and S7b). The task generator integrates the job script with actual experimental device information to complete the predefined task template (in JSON format) containing the experimental conditions and task sequences (Fig. 3 and Supplementary Figures S7c-e). Subsequently, the task scheduler supervises resource allocations for task executions by obtaining information on the location indices from the resource manager (Fig. 3 and Supplementary Figure S8).
Crucially, the abstraction of tasks, as exemplified in Chemputer14 and HELAO-async24, ensures adaptability across different platforms by omitting specific device operation and location details.14,24 To concretize abstracted tasks, we implement an action translator, which translates abstracted tasks into concrete actions, incorporating predefined action sequences and device location information. For instance, the "AddSolution" task involves the serial actions of initializing a pump, moving a dispenser and injecting a solution, each executed by different devices (Fig. 3 and Supplementary Figure S9). Next, the action executor transmits device commands for real experiment execution, employing a transmission protocol that follows predefined data types to module nodes. These data types, including the job ID, device name, action type, action data and mode type, are encoded and transmitted to module nodes via the Transmission Control Protocol/Internet Protocol (TCP/IP) (Fig. 3 and Supplementary Fig. S10). The modules execute experimental action and return the resulting data to the database for subsequent experimental planning. More detailed information on the resulting data structure is provided in Supplementary Figure S11.
Network protocol-based modularization. To manipulate multiple jobs within OCTOPUS, process modularization is key for efficiently operating experimental processes. We conceptualized four key concepts to explain the benefits of using experimental process modularization and network protocols, which include homogeneity, scalability, safety and versatility, as illustrated in Fig. 4.
First, a challenge may arise when different manufacturers utilize various programming languages and operating systems for their own experimental devices, and such heterogeneity may hinder seamless communication between module nodes and devices. Examples of the heterogeneous environments among devices in the "BatchSynthesis" and "UV‒Vis" modules are provided in Supplementary Figure S12a. To achieve a homogeneous environment in OCTOPUS, we opted to connect the experimental devices via the TCP/IP technique (Fig. 4a). Since most programming languages, including C, C++, Python, JAVA and JavaScript, support TCP/IP regardless of the operating system (both Linux and Windows), we independently created device servers to facilitate communications between module nodes and devices via TCP/IP, ensuring platform homogeneity (Supplementary Figure S12b).
Second, the TCP/IP network protocol not only ensures homogeneity but also offers significant advantages in terms of scalability. Scalability in MAP refers to the ability to connect experimental modules of multiple laboratories, even across different nations, as illustrated in Fig. 4b. When multiple experimental modules are joined in a TCP/IP-based network, it logically becomes part of the same network, overcoming spatial constraints in MAP and providing infinite possibilities for platform scalability. Notably, synthesized materials cannot be transferred across different regions due to practical issues, including time costs and material degradation. Thus, the absence of material synthesis modules in a specific laboratory could hinder flexible utilization for diverse applications, despite all experimental modules (modules for material synthesis, characterization and property evaluation) being connected within a single network. To address this, we proposed customizing material synthesis modules in each laboratory and integrating all the other local modules into a single platform despite remote distances via the TCP/IP-based network protocol (Fig. 4b).20,21,36,37 The method for integrating network protocols is detailed in Supplementary Figure S13.
Third, to minimize safety concerns arising from physical disconnections, we implemented the user datagram protocol (UDP) with TCP/IP to monitor physical connection issues for platform safety, as illustrated in Fig. 4c. If a miscommunication occurs between the master node, module nodes due to a physical disconnection between module nodes and experimental devices, the UDP-based broadcast executes an emergency stop for the other modules in the internet network to prevent safety issues in surveillance-free systems (Fig. 4c and Supplementary Figure S14a). In addition to the emergency stop, an alert system based on messengers was implemented to provide researchers with notification and monitoring systems, which aids in swiftly recovering the platform from safety issues (Fig. 4c and Supplementary Figure S14b-e).
Fourth, a key challenge in MAP is that most platforms have thus far been developed only for a specific purpose; experimental devices are limited to a specific application.14,38–40 However, with TCP/IP-based process modularization, a static platform can evolve into a versatile platform (Fig. 4d). Researchers can selectively adopt modules tailored to their own applications, allowing customized process design within a single platform and ensuring the versatility of the platform.
Job parallelization to address the module overlap challenge. The compositions of the master node and module nodes and their functionalities are investigated with detailed examples. In the following sections, the user-optimal scheduler (US) developed within OCTOPUS is explained, incorporating three techniques, job parallelization, task optimization and CPS.
When multiple clients attempt to utilize a MAP with many modules, module overlap issues may occur. Customized scheduling algorithms are necessary to address module overlap issues effectively. One conventional approach is job serialization based on the FCFS (First-Come-First-Serve) concept (Fig. 5a), where the priorities of module usage are assigned based on the job submission order. However, in job serialization, device standby times, where no significant actions are performed in terms of devices, may substantially impair job execution efficiency. An example of device standby times is the chemical reaction period ("React" task) in the "BatchSynthesis" module, where chemical reactions occur in chemical vessels but no actions are performed in terms of devices.
To improve module resource utilization, we introduced job parallelization by leveraging device standby times (Fig. 5a and Supplementary Figure S15). During the device standby times of a preceding job, the following job can run in parallel without hindering previously started tasks. As illustrated in Fig. 5b, while job ID 0 progresses during the "React" task in the "BatchSynthesis" module, the "PrepareContainer" task in the next job (job ID 1) can be executed simultaneously by leveraging the bottleneck time (chemical reaction period) of the prior "React" task (Fig. 5b). This specific example of job parallelization is provided in Supplementary Video 2.
To evaluate the impact of job parallelization on time efficiency, we conducted virtual experiments on catalysis processes, which typically involve long device standby times for chemical reactions. In this test, we define 10 virtual modules related to catalyst synthesis ("BatchSynthesis", "BallMilling"), preprocessing ("Washing", "Filtration", "Drying", "InkPreparation", "SprayCoating"), characterization ("XRD"), and property evaluation ("HalfCellTest", "FullCellTest"). Each module involves both a device execution time and device standby time, as described in Supplementary Figures S16 and S17. Seven different types of catalysis experiments utilizing parts of these modules were performed (Fig. 5c, Supplementary Figures. S16 and S17). Fig. 5d compares the results between job serialization and parallelization using bar charts. Three performance metrics were implemented, including "job waiting time", "job turnaround time" and "job total time" (Supplementary Figure S18). Job waiting time reflects the difference between the job submission and start time, providing client-centric performance insights.41 "Job turnaround time" indicates the duration between the start of a job and completion of a job, and it offers administrator-centric performance insights. "Job total time", the sum of "job waiting time" and "job turnaround time", reflects the overall job execution efficiency.
Job serialization entails the sequential execution of modules based on a priority order. Analysis of the performance metrics for time efficiency reveals that, in serialized jobs, the lower-priority jobs, determined by job submission order, may experience substantially increased waiting times due to module overlaps. In contrast, job parallelization aims to reduce waiting times by leveraging the device standby times within each module. Consequently, for all seven jobs in these virtual tests, a significant reduction in “job total time” is achieved via job parallelization compared to job serialization (Fig. 5d). For example, for job ID 3, "job total time" decreased significantly with job parallelization (from 10 hours to 6.5 hours) by leveraging the device standby time of modules utilized in the preceding jobs, such as the "BatchSynthesis" module of job ID 1. Similarly, in Job ID 4, "job total time" was nearly halved with job parallelization (10 hours to 5.5 hours), leveraging device standby times in the module, such as the centrifugation of the "Washing" module, oven usage of the "Drying" module and scanning of the "XRD" module, as shown in Supplementary Figure S16. As a result, job parallelization significantly improves the time efficiency, reducing both "job waiting time" and "job turnaround time", ultimately shortening "job total time" (Fig. 5d and Supplementary Table S1).
Task optimization with masking table for preventing device overlaps. Up to this point, we operated under the assumption that each module functions independently, with the experimental devices operating without interference. However, as multiple clients submit job requests, the likelihood of multiple tasks within the same module running concurrently increases. In such scenarios, experimental device collisions may occur, raising potential safety concerns (Fig. 6a and Supplementary Figure S19). Moreover, the MAP cost implications are substantial, as each experimental device typically entails significant expenditures to ensure experimental reliability and precision.15–17 In this regard, implementing device sharing between different modules has become imperative for cost savings within limited budgets, as demonstrated in recent advances.30,42–44 For instance, a robotic arm may serve for both the "BatchSynthesis" and "UV‒Vis" modules, as illustrated in Fig. 6b and Supplementary Figure S19, enabling cost savings through shared utilization.8 However, when multiple modules attempt to execute individual tasks simultaneously, conflicts may arise regarding the prioritization of tasks for the robotic arm. Consequently, task optimization is urgently needed to prevent device collisions within the same module, and device overlap challenges across different modules.
To overcome these challenges, we introduce a task optimization technique with a masking table. In our approach, each module continuously updates the device status in a tabular format retrieved from the resource manager. Devices currently in use by ongoing tasks are assigned as "True", while unused devices are marked as "False", as shown in the device status table example in Fig. 6c. The resource manager dynamically updates the device status table in real time. Next, it is essential to predefine masking tables for each task, which identify the devices required for a specific task as "True" and those not needed as "False". For instance, when performing the "GetAbsorbance” task in the “UV‒Vis” module, devices such as UV‒Vis spectroscopy, robotic arms and pipettes are involved, and are thus marked as "True", while the other devices are marked as "False" to create task-specific masking tables (Fig. 6c). Prior to performing a specific task, the AND Boolean operation is performed between the device status table and the masking table for the task, resulting in a table for “hold criteria” to determine whether to proceed with the next task (Supplementary Figure S20). If all hold criteria indicate "False", the tasks are executed in parallel, whereas if any of the logical results are found "True", the system waits until the ongoing task is completed, and all the hold criteria indicate “False”. Examples of masking tables are provided in Supplementary Figure S21, and an actual example of a task optimization process with a masking table is provided in Supplementary Video S3. In summary, when experimental devices are shared between modules, task optimization with masking tables enables cost-saving benefits and enhances the efficiency of device utilization while avoiding device collisions and ensuring safety.
The closed-packing schedule for optimizing module resources. In computing server systems with a finite number of cores, resource allocation considerations are paramount, and are often documented in job scripts.32,33 Similarly, in the context of MAP, resource management extends to modules and experimental devices. Here, device resources denote the maximum number of experiments that each module can accommodate. For example, the “BatchSynthesis” module’s resource could be defined by the maximum number of vials simultaneously processable by a magnetic stirrer, while in the “UV‒Vis” module, it could also be determined by the maximum number of vial holders available for UV‒Vis spectrum measurements (Supplementary Figure S22). Thus, given the limited availability of resources in each module, scheduling methods that account for these constraints are essential in practical platforms.
To efficiently execute multiple jobs within such constraints, we introduce the closed-packing schedule (CPS) algorithm (Fig. 7 and Supplementary Figure S23). The core concept of CPS lies in splitting tasks in a single job into multiple batches to maximally utilize the remaining resources via compact packing. An example of CPS is illustrated in Fig. 7, where three jobs (job IDs 1, 2 and 3) are sequentially submitted with different batch sizes (the number of required vials) of eight, four and eight, respectively, and a stirrer can accommodate up to 16 vials. With active job parallelization, at the execution of job ID 3, 12 resources out of a total of 16 are already occupied by preceding jobs, leaving only 4 resources unused. Then, the CPS divides the tasks of job ID 3 into two batches to allow the first batch to be executed first by fully occupying the 4 unused resources (compact packing), and to allow the other batch to be executed later as soon as the preceding jobs are completed and resources become available. As demonstrated in the example, the CPS aims to effectively minimize the waste of module resources by splitting the tasks within a job based on the computation of the remaining resources.
Performance test of the user-optimal scheduler. In the previous sections, the user-optimal scheduler developed within OCTOPUS was explained, and involved three distinct scheduling methods, job parallelization, task optimization and CPS. To maximize scheduling efficiency, these three scheduling methods were integrated within OCTOPUS, resulting in a united scheduling system named the user-optimal scheduler (US) in this paper. To benchmark the US, we introduced the conventional FCFS to prioritize jobs based on their order of submission (Supplementary Figure S24).45 FCFS executes jobs sequentially based on a priority order. We chose the FCFS algorithm as the baseline scheduler for comparison due to its fairness in executing jobs based on its priority order, which is crucial in multiclient systems.
Figs. 8a and 8b visualize the execution time efficiency in processing multiple jobs across the two different scheduling schemes of FCFS and US. We employed a platform capable of simulating device collision and sharing issues to reflect a realistic environment.8 In these comparative tests, 11 jobs with different batch sizes are submitted sequentially, all of which require the use of either the "BatchSynthesis" or "UV‒Vis" module (Supplementary Figure S25). It is important to clearly understand that, in the tests, the techniques of job parallelization, task optimization and CPS-based resource allocation, developed before, are active only for the US, whereas they are not active in the FCFS scheduling scheme.
Through the observation of the scheduling results, we demonstrate the efficacy of US, particularly in terms of "job waiting time". The benefits of combined job parallelization and CPS are pronounced in many cases. For example, job IDs 1, 8 and 10 significantly reduced "job waiting time" (8.92 hours to 0.63 hours for job ID 1, 10.21 hours to 0.72 hours for job ID 8, and 10.97 hours to 0 hours for job ID 10). The time reductions are attributed to both leveraging device standby times and splitting jobs via CPS. Notably, these jobs are parallelized during the "React" task (device standby time) in the "BatchSynthesis" module of the preceding jobs (Fig. 8b, Supplementary Figure S26 and Table S2). Similarly, job ID 4 also benefits from both job parallelization and CPS, leading to a reduction in "job waiting time", as shown in Fig. 8c (11.72 hours to 0 hours for job ID 4). This job is parallelized with the preceding job ID 3 as the CPS splits the job according to the remaining resources of the "UV‒Vis" modules (Fig. 8b and Supplementary Figure S27). Meanwhile, the benefit of the task optimization technique is well observed in many cases, such as job IDs 2, 3, 4, 6, 7 and 9. Since the “BatchSynthesis” module and “UV‒Vis” module share a robotic arm, the simultaneous execution of these two modules may cause safety hazards from the device sharing environments. However, with active task optimization, concurrent executions of both modules were safely performed, as demonstrated in Supplementary Figure S28, Table S2 and Video S3.
By minimizing "job waiting time", the US also proved more efficient in terms of "job total time" compared to the conventional FCFS scheme. However, we observed an overall and slight increase in "job turnaround time" for US, potentially resulting from a duplication of device standby time (Supplementary Figure S29). For Job IDs 1, 8 and 10, our scheduling system duplicates the device standby time associated with the "React" task, as the jobs are split based on the remaining resources. This redundancy contributes to the accumulation of "job turnaround time". Overall, while US significantly reduces "job waiting time", some delays in "job turnaround time" may occur due to device standby time duplications.
Overall, by analyzing the performance metrics for time efficiency, US emerged as highly effective, saving time across all jobs in terms of "job waiting time" (Fig. 8c). Although there is a slight increase in "job turnaround time" due to duplications of device standby time by CPS (Fig. 8d and Supplementary Figure S27), the significant improvement in efficiency in terms of "job waiting time" renders US more efficient, leading to a substantial reduction in “job total time", compared to conventional FCFS (Fig. 8e and Supplementary Table S2).