Detecting anomalies in telemetry data captured on-board a spacecraft is a critical aspect of its safe operation, and it allows us to effectively and timely respond to failures and hazards. There exist three main types of anomalies that should be considered for such complex missions. In point anomalies, telemetry values fall outside the nominal operational range. The collective anomalies refer to the overall sequences of consecutive telemetry values that are anomalous (a single data point does not necessarily manifest an anomaly), whereas in contextual anomalies, the single values are anomalous within their local neighborhood. There have been various approaches for automating the process of detecting anomalies from telemetry data. The basic yet widely exploited algorithms include the out-of-limit techniques that are built upon the assumption that we have the prior expert knowledge allowing us to exploit a rule-based approach for detecting unexpected events. There are machine learning algorithms for this task, but they are commonly heavily parameterized and require large amounts of ground-truth (manually delineated) data, ideally with captured anomalies. Since acquiring such data is infeasible in practice, unsupervised techniques have attracted the research attention, as they do not require having large training samples to train well-generalizing models. In this paper, we not only review the current state of the art in anomaly detection from telemetry data, but also present our algorithm for this task – being developed as a part of our Antelope on-board computer with predictive maintenance capabilities – which exploits recurrent neural networks (we are currently utilizing long short-term memory networks) to model the expected telemetry signal. Such models can be trained from a set of the simulated nominal telemetry signals (e.g., using the software or hardware-in-the-loop simulators), or from a set of real-life telemetry presenting the nominal operation. Importantly, we can learn from the correct examples that do not contain anomalous events – it allows us to abstract from the type of anomalies that we want to target. Once the expected signal is elaborated, it is confronted with the actual one, and the obtained error triggers the alert showing that the anomaly has appeared. We additionally show how to thoroughly verify the anomaly detection techniques in a quantitative way, and what kind of metrics comprehensively reflect the underlying abilities of such deep learning techniques. Finally, we will present our visualization tools that help us better understand the advantages and shortcomings of various anomaly detection methods and will discuss our experiments performed over benchmark one-dimensional signal, and real-life telemetry captured on-board the European Space Agency’s OPS-SAT satellite. Since the Antelope will be exploited on-board a satellite, our resource-frugal models will ultimately help us respond to the events quicker and could be used to reduce the amount of data to transfer back to Earth through annotating the most important parts of the signal that enable further analysis and interpretation.

Space operations are performed in a dynamic and complex environment, which exhibits nondeterministic action outcomes and where unexpected events may require human intervention. In this regard, tasks' scheduling and optimization of onboard resources are crucial points in the problem of enabling autonomous capabilities aboard satellites. Reasoning agents and autonomous decision-making systems represent valid approaches to such kind of problem. Nevertheless, these solutions require large efforts in defining consistent knowledge bases and models about the operative environment and about the agent itself.

Reinforcement Learning (RL) is currently one of the most compelling research fields in AI. Specifically, an RL algorithm allows agents to learn how to perform actions in an autonomous way through interaction with the surrounding environment. Bearing that in mind is possible to start exploring the advantages of such kinds of algorithms applied to the problem of autonomous space missions. Specifically, the objective of the research hereby presented is the implementation of an autonomous agent, which emulates an operating Earth Observation satellite, capable of scheduling downlink operations in advance, taking actions accordingly to its available resources and the priority of the data generated, aiming to optimize its tasks outcome at the same time.

We start from a brief analysis of the state-of-the-art in the research field and the most promising works regarding Reinforcement Learning agents' implementation. A preliminary implementation was tested by developing a baseline environment used for training the algorithm. The model under development will reach TRL 8 by Q3 2021, providing an innovative and effective alternative to nowadays solutions.

The focal points of the entire research and implementation process were the problem formalization in terms of the Markov Decision Process (MDP) and the selection of a suitable reward function; the latter, in particular, directly impacts the learning process of the agent and his behavior. More sophisticated approaches were taken into account after the first results, aiming to reach a possible generalized training procedure.

In the presentation, we will cover an introduction part describing the RL framework and how our problem is approached in terms of MDP. Afterward, the structure of the environment and the choice of algorithms to train the agent are described. In the final part, key results are discussed and some limitations are presented together with possible alternative solutions to address them.