Director of the ASU Human-Oriented Robotics and Control (HORC) Lab, School for Engineering of Matter, Transport and Energy, Arizona State University, USA
Received Date: January 06, 2017; Accepted Date: January 08, 2017; Published Date: January 16, 2017
Citation: Artemiadis P (2017) Brain-Swarm Control Interfaces: The Transition from Controlling One Robot to a Swarm of Robots. Adv Robot Autom 6: e127. doi: 10.4172/2168-9695.1000e127
Copyright: © 2017 Artemiadis P. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Visit for more related articles at Advances in Robotics & Automation
Brain-Machine Interfaces (BMIs) have been one of the most influential and disruptive science fields of the past decades. Prosthetic or remotely operated robotic devices being controlled by brain signals has transitioned from science fiction to reality. Advances in the recording electrodes technology and the machine learning and signal decoding algorithms were critical in the realization of those systems. The second decade of the 21st century brings new challenges found in both frontiers; first, advancements in neuroscience are sought via high-resolution mapping of the brain for better understanding of its function and decision making processes. On the robotics frontiers, the challenge of the human controlling many robots simultaneously is of utmost importance for applications spanning from industrial and entertainment, to disaster response and military. As the swarming paradigm, deriving inspiration from the behaviour of natural swarms such as bird flocks and fish schools, offers myriad advantages to a team of robots, the way humans interact and control a robotic swarm creates new avenues of research. This article summarizes recent developments and novel methods for brain-swarm interfaces, and poses challenges for the future researchers.
Brain-machine interfaces; Electroencephalographic; Electromyography; Robotic devices
Brain Machine Interfaces (BMI) have gained increased attention over the last decades because they offer intuitive control in a plethora of applications where other interfaces (e.g. joysticks) are inadequate or impractical. Moreover, these kinds of interfaces allow people with motor disabilities due to amyotrophic lateral sclerosis (ALS), spinal cord injuries (SCI) etc. to interact with the world. This interaction is found in many forms and can vary between controlling the motion of a cursor on a screen to interacting with an actual robotic platform. However, most of the existing systems allow only binary control, while the number of degrees of freedom directly.
Brain-machines interfaces have been widely used in many applications ranging from the control of prosthetics [1,2] to humancomputer interfaces . Electroencephalographic (EEG) signals in particular have been used in the past for this scope [4-7]. There are two main approaches towards BMI using EEG signals: one is based on event related potentials (ERPs) and another is based on the multiple sensor EEG activities recorded in the course of ordinary brain activity. The latter approach is more comprehensive and does not require any particular stimulus. The author has extensive research back-ground on BMIs using electromyography (EMG) signals from upper limb muscles [8-19], and neural recordings [20,21]. We recently proposed that instead of using the decoder-based technique for BMIs, human subjects can learn to map their neural activity into control actions for an artificial system [22-28]. More specifically we have shown that subjects can control artificial systems using muscular activation, without requiring a decoding function to map one to the other. This method requires no training of the interface itself, therefore no decoder. We have also shown that once the subjects learn to control a system, their learned techniques are transferable to different tasks. This result led us to propose new avenues for BMIs, going beyond decoder-based techniques, and significantly improving human-machine embodiment.
Focusing on EEG signals, previous studies have demonstrated the ability of the subjects to develop control of their own brain activity using biofeedback [29,30]. More specifically, it was shown that human subjects gained voluntary control over brain rhythms. Leveraging this result, and based on our recent findings, we recently proposed to develop a novel framework of embedded human controllers using EEG signals. More-over, in contrary to all previous studies on BMIs that built methods for the control of a single system (usually a teleoperated or prosthetic device), we proposed to extend the current state of the art by introducing the control of a multi-agent system (swarm) using brain interfaces. This article presents the motivation of the brain-swarm control interfaces project, as well as recent results.
Without loss of generality, this article focuses on the control of unmanned aerial vehicles (UAVs) by a human. State of the art systems usually involve one human controller for a single UAV. The human has spatial feedback of the controlled vehicle, and provides it with highlevel commands (e.g. fly to a specific location or follow a predefined surveillance path) [31,32]. However, the swarming paradigm, deriving inspiration from the behaviour of natural swarms, offers myriad advantages to a team of UAVs. A swarm system consists of a large group of relatively inexpensive, interchangeable vehicles that execute autonomous decisions using information obtained via local sensing and communication. The redundancy in a swarm makes its operation robust to vehicle failures and disturbances, which also enable the use of sacrificial platforms and its distributed activity, can conceal the system’s mission from an opponent. Recent advances in computing, sensing, actuation and control technologies are currently enabling the development of swarms of aerial vehicles, varying in complexity, size and overall scale [33,34]. The integration of very large teams of robots into comprehensive systems enables new tasks and missions ranging from search, exploration, rescue, surveillance, pursuit, up to deploying infrastructure.
The trend of deploying multi-agent systems, however, poses a challenge for the control of such systems, especially for human operators. Currently, most large robotic systems are controlled by multiple operators, often via remote control. For larger systems with more agents, such an approach is not practical. Although most of these agents can act autonomously, the distributed algorithms and complex dynamics of those systems pose another challenge to the human operators. Therefore, as the “power of the many” UAVs is facilitating an increasing number of applications, the human role in the high-level control architecture of this population is becoming more and more significant.
We recently demonstrated a hybrid control interface for a human and a swarm of UAVs using both a manual controller (joystick) and brain interfaces via EEG. The EEG signals were recorded using a noninvasive set of 64 electrodes placed on the head of human subjects. The data were recorded at 500 Hz. A 5th order Butterworth band pass filter between the frequencies of 1 and 40 Hz was applied to the data in order to remove low-frequency trends and line noise. In order to accommodate for the volume conduction effects that are typical in scalp EEG measurements , a large Laplacian filter was applied to each of the channels of interest. We focused our analysis on 11 channels located over the sensorimotor cortex, namely C3, Cz, C4, FC3, CP3, C1, FCz, CPz, C2, FC4, and CP4. An electrooculogram (EOG) artifact removal algorithm  was applied before the large Laplacian referencing in order to eliminate any artifacts from eye blinks and eye movements.
After the pre-processing step, Fast Fourier Transform (FFT) was applied to the data in order to extract the spectral features of the signals. For each channel, a dedicated algorithm selected automatically a frequency band of interest. Its goal was to find for each channel the frequency band that the user was activating the most. In this work, we were interested in ERD/ERS phenomena on the motor cortex while the subjects were performing limb movement imagination or actual limb movement (joystick movement) and we focused our search on the alpha (α) and beta (β) bands (i.e. 7 to 30 Hz). In order to further extract features that would guarantee good differentiation among the tasks, we applied Principal Component Analysis (PCA) to the final FFT features. Finally, a Hidden Markov Models (HMMs) classifier was developed. The final output of the hybrid system combines a continuous measure about the amplitude of the EEG signals with the classification decision about the brain state of the subject and a joystick input in order to output a command vector where each of its elements regulates a specific DOF of the robotic platform.
A swarm of 3 quad rotors was controlled using the pro-posed hybrid BMI system. In Figure 1, we show snapshots of the experiment, where the user changes the formation of the quad rotors, passes them through the hoop and then returns them in their original formation. A video of the experiment is included in ref. . The joystick was used for the directional motion control of the swarm, while the cohesion of the swarm, defined as their inter-distance in the lateral axis, was controlled from the subject’s brain signals. As it can be seen in the figures, the subject was able to use the hybrid interface, i.e. use simultaneously both the joystick and the brain interface to pass the swarm through the narrow hoop. This was a real-time demonstration of controlling a swarm of quad rotors using our proposed hybrid BMI using both EEG activations and joystick inputs.
Figure 1: Snapshots of the three quadrotors passing through the rectangular hoop during the second experiment. The top row shows a side view of the motion of the swarm, while the bottom row shows the top view of the quadrotors. A: Initial formation, B: Change of formation, C: Passing the quadrotors through the hoop, D: Returning to initial formation. Video at .
The system we proposed is going to generate a novel generation of Brain-Swarm Control Interfaces that will provide human operators with a wealth of control capabilities over multi-agent systems. Advancing our understanding of swarm perception and control at the brain level offer a myriad of applications that involve human-in-the-loop multiagent systems, spanning from industrial and entertainment, to disaster response and military situations. The avenues of multidisciplinary research required to address the challenges are numerous and exciting. The transition from controlling one robot to a swarm of them using brain-machine interfaces has just started.
This work is supported by the U.S. Defence Advanced Research Projects Agency (DARPA) grant D14AP00068, and U.S. Air Force Office of Scientific Research (AFOSR) award FA9550-14-1-0149. Controlled through those interfaces is quite limited, i.e. only one or two in most cases.