Reinforcement Learning for Satellite Communications: From LEO to Deep Space Operations

Abstract

the National Aeronautics and Space Administration (NASA) is in the midst of defning and developing the future space and ground architecture for the coming decades to return science and exploration discovery data back to investigators on Earth.
美国国家航空航天局（NASA）正在为未来几十年定义和开发未来的太空和地面架构，以将科学和探索发现数据返回给地球研究人员。
Optimizing the data return from these missions requires planning, design, standards, and operations coordinated from formulation and development throughout the mission.
The use of automation enhanced by cognition and machine learning are potential methods for optimizing data return, reducing costs of operations, and helping manage the complexity of the automated systems.
要优化从这些任务返回的数据，需要计划，设计，标准和操作，这些任务需要在整个任务过程中从制定和开发中进行协调。
通过认知和机器学习增强的自动化功能的使用是优化数据返回，降低运营成本并帮助管理自动化系统的复杂性的潜在方法。

An experiment using NASA’s Space Communication and Navigation Testbed onboard the International Space Station and the ground station located at NASA John H. Glenn Research Center demonstrates for the first time the benefits and challenges of applying machine learning to space links in the actual flight environment.
在本文中，我们讨论了机器学习在通信系统的链接到链接方面的潜在作用。
使用国际空间站和位于美国国家航空航天局（NASA）的地面站上的美国国家航空航天局（NASA）的空间通信和导航测试平台进行的一项实验，约翰·格伦（John H. Glenn）研究中心首次展示了在实际飞行环境中将机器学习应用于空间链接的好处和挑战。
The experiment used machine learning decisions to confgure a space link from the ISS-based testbed to the ground station to achieve multiple objectives related to data throughput, bandwidth, and power. Aspects of the specifc neural-network-based reinforcement learning algorithm formation and on-orbit testing are discussed.

实验使用机器学习决策来配置从基于ISS的测试台到地面站的空间链接，以实现与数据吞吐量，带宽和功率有关的多个目标。 讨论了基于特定神经网络的强化学习算法的形成和在轨测试的各个方面

Intruduction

With the U.S. Space Council’s recent recommendation and the President’s Space Policy Directive 1, NASA is once again setting its sights to return to the Moon and later on to Mars for “an innovative and sustainable program of both robotic and human exploration” [1]. The exciting exploration missions of planetary bodies require careful and comprehensive planning for safe voyage, science rich discovery and learning of the solar system, and inspiration for future generations. One key to these successful missions is the communications between and among robotic and crewed spacecraft, science or relay satellites, surface and space elements, ground stations on Earth, and ultimately principal investigators and mission controllers.
根据美国航天理事会的最新建议和总统的《太空政策指令1》，NASA再次将目光投向月球，然后又返回火星，进行“一项创新的，可持续的机器人和人类探索计划” [1]。 行星体令人兴奋的探索任务需要进行仔细而全面的计划，以进行安全航行，科学丰富的太阳系发现和学习以及对子孙后代的启发。 这些成功的任务的关键之一是机器人与乘员航天器，科学或中继卫星，地表与空间元素，地球上的地面站以及最终的主要研究人员和任务负责人之间的通信。
The future of human exploration of space requires new capabilities to support exploration and colonization missions. The future space and ground architecture will provide communications, navigation, and internetworking services for space missions from within Earth’s orbit out through Mars and other planetary exploration in deep space. At the center of both the mission satellites and the ground stations,
planetary surface elements and relay satellite infrastructure are the communications systems providing seamless operations across the solar system back to controllers on Earth. This multitude of systems is a complex system of systems that relies on compatibility and interoperability among user spacecraft and the communications infrastructure.
As these space communication systems become larger and more complex, new methods to control, operate, and enhance the communications infrastructure are needed to more quickly adapt to science opportunities; interpret, understand, and quickly respond to system anomalies; and ease or relieve human operators from constant and detailed interaction with the systems of systems and their multifaceted interaction.
人类太空探索的未来需要新的功能来支持探索和殖民任务。未来的太空和地面架构将为通过火星和其他深空行星探索从地球轨道内的太空任务提供通信，导航和互联网络服务。在任务卫星和地面站的中心，
行星表面元件和中继卫星基础设施是通信系统，可提供整个太阳系到地球控制器的无缝运行。众多系统是一个复杂的系统，它依赖于用户航天器和通信基础结构之间的兼容性和互操作性。
随着这些空间通信系统变得越来越大，越来越复杂，需要新的方法来控制，操作和增强通信基础设施，以更快地适应科学机会。解释，理解并快速响应系统异常；并减轻或减轻人工操作员与系统系统及其多方面交互的持续和详细的交互。
Software-defined radio (SDR) is one such technology that provides the flexibility and configurability needed for NASA’s future cognitive communication systems. SDRs provide the needed on-orbit reconfgurability where a portion of their functions can be updated via software in situ while conducting a mission. This reprogrammable capability provides system flexibility and adaptability to change the operating or signaling characteristics, or mitigate or recover from system anomalies
软件无线电（SDR）是一种这样的技术，可提供NASA未来的认知通信系统所需的灵活性和可配置性。 SDR提供所需的在轨可配置性，在执行任务时，可以通过软件现场更新其部分功能。 这种可重新编程的功能为系统提供灵活性和适应性，以改变操作或信号特性，或减轻系统异常或从系统异常中恢复
SDRs have been around for a number of years for ground-based and defense applications. However, NASA’s John H. Glenn Research Center (GRC) has been testing the viability of SDRs for space through an experimental communications system called the Space Communications and Navigation (SCaN) Testbed [2, 3]. The SCaN
Testbed is a communications research platform installed on the International Space Station (ISS), and it comprises three SDRs with the aim to facilitate on-orbit reconfgurable communication experiments to reduce the risk and increase the experience of using SDRs in space missions. The resultant network topology is illustrated in Fig. 1
对于地面和国防应用而言，SDR已经存在了很多年。 但是，美国宇航局的约翰·格伦研究中心（GRC）已通过称为“太空通信与导航（SCaN）”试验台的实验通信系统测试了SDR在太空中的可行性[2，3]。 SCaN
Testbed是一个安装在国际空间站（ISS）上的通信研究平台，它包括三个SDR，目的是促进在轨可重构通信实验，以降低风险并增加在太空飞行中使用SDR的经验。 最终的网络拓扑如图1所示。