PhiSat-1 Nanosatellite Mission
PhiSat-1 (Φ-Sat-1) is the first European satellite to demonstrate how onboard artificial intelligence can improve the efficiency of sending Earth observation data back to Earth. This revolutionary artificial intelligence technology will fly on one of the two CubeSats that make up the FSSCat (Federated Satellite System) mission – a Copernicus Masters winning idea. 1)
As the overall 2017 Copernicus Masters winner, FSSCat, was proposed by Spain’s UPC (Universitat Politècnica de Catalunya) and developed by a consortium of European companies and institutes.
FSSCat is an innovative mission concept consisting of two federated 6U CubeSats in support of the Copernicus Land and Marine Environment services. They carry a dual microwave payload (a GNSS-Reflectometer and a L-band radiometer with interference detection/mitigation), and a multispectral optical payload to measure soil moisture, ice extent, and ice thickness, and to detect melting ponds over ice. It also includes a radio/optical inter-satellite link and an Iridium intersatellite link to test some of the techniques and technologies for upcoming satellite federations. FSSCat will be the precursor of a constellation of federated small satellites for Earth observation achieving high temporal resolution and moderate spatial resolution in a cost-effective manner. 2)
«The Federated Satellite System 6U tandem mission for sea ice and soil moisture monitoring captured the interest of the challenge experts immediately. Not only because the mission concept shows a high degree of well thought through technical novelties, but also because it will provide data that is complementary to the Sentinel fleet. This is especially true for the soil moisture monitoring component, which is not part of the current Sentinel portfolio. The FSSCat mission development is good to go and due to its disruptive approach, we are confident that it will be seen as a breakthrough in procuring future small missions at ESA.»
The two CubeSats will collect data, which will be made available through the Copernicus Land and Marine Environment services, using state-of-the-art dual microwave and hyperspectral optical instruments. They also carry a set of intersatellite communication technology experiments.
During Φ-week (9-13 September 2019 at ESA/ESRIN), ESA’s Director of Earth Observation Programs, Josef Aschbacher, said, “We see that there is huge interest in Φ-Sat and thanks to our partners, it is ready to be launched. We live in exciting times, the pace at which digital technology is developing coupled with the wealth of satellite information being delivered and, indeed, the growing demand for such data, means there are many opportunities to make a step change for the future of Earth observation. And, with Φ-Sat – Europe’s first artificial intelligence in space – we are going to do just this.”
The hyperspectral camera on one of the CubeSats will collect an enormous number of images of Earth, some of which will not be suitable for use because of cloud cover. To avoid downlinking these less than perfect images back to Earth, the Φ-Sat artificial intelligence chip will filter them out so that only usable data are returned.
Marco Esposito, from cosine Remote Sensing, the company that led the development of the artificial intelligence algorithm, explained, “While compact, the instrument – which covers the visible and near infrared with hyperspectral capability, enhanced with bands in the thermal infrared – is very powerful and will acquire terabytes of data that can be used to monitor vegetation changes and to assess water quality, for example.
“However, generating this amount of data actually poses a problem, as the data have to be handled efficiently so that they can reach the users in a timely manner. With Φ-Sat we have effectively given the instrument its own brain, which processes the data onboard to detect clouds in the images. This not only ensures better quality data, but makes the delivery much more efficient.”
ESA’s Massimiliano Pastena, noted, “Indeed, this will be the first satellite to demonstrate the use of artificial intelligence in orbit and we are very much looking forward to it being launched in the coming months.”
Φ-Sat-1 is the first scientific and technology initiative of the Earth Observation directorate of the European Space Agency (ESA) in the NewSpace field. 3) Φ-Sat-1 is the result of an enhancement of the FSSCAT mission using AI (Artificial Intelligence) and improved imaging capabilities of the hyperspectral camera HyperScout-1 VNIR imager. cosine Remote Sensing is leading the development and execution of the demonstration, including the highly integrated spectral imaging in the VNIR and TIR, as well as the integration of state-of-art AI accelerators including the first AI algorithms for cloud screening. The HyperScout-2 spectral camera is the successor of HyperScout-1, a miniaturized reflective optical instrument equipped with enhanced processing capabilities, in orbit since February 2018 as part ESA GOMX-4B technology demonstration mission.
Φ-Sat-1 will demonstrate the capabilities of small instruments for scientific applications, and will validate in-orbit innovative technologies. 4) The technology demonstration includes the miniaturization and highly integration of visible, near infrared and thermal spectral channels, as well as state-of-art processors and machine learning algorithms. The hardware co-registration of the VNIR and TIR channels will enable a variety of applications for terrain classification and change detection. Φ-Sat-1 will demonstrate an AI-based inference engine for cloud detection and will be launched as enhancement of the FSSCat mission, 5) during the VEGA SSMS (Small Satellite Mission Service) Proof Of Concept flight.
Sensor complement (HyperScout-2)
The HyperScout product series consists of very compact spectral imagers, in the order of 1-2 kg. The instruments are rather inexpensive, which are not meant to replace or compete with larger institutional satellites like Sentinel or Landsat, but more to complement large observation spacecrafts, being exploited for early detection of anomalies, as well as other applications. HyperScout is the one of the first hyperspectral imagers with such a compact envelope, therefore enabling applications which rely on the potential of very high temporal resolution (i.e. order of hours) in order to detect anomalies as early as possible and to refer to other space or ground assets for more detailed investigations.
A spectral imager with such a small envelope becomes an attractive solution for deployment in constellations, and therefore offers to the user the possibility of achieving high revisit times over regions of interest. The HyperScout concept is a combination of state of the art optics packed in an extremely compact format, in combination with very powerful processing abilities. It was first designed to be installed on CubeSat and small satellites, however it can be easily incorporated into larger platforms given the small engineering budgets and the compact volume.
HyperScout-2 is based on the building blocks of HyperScout-1 that was successfully demonstrated in flight by cosine Remote Sensing and ESA onboard the ESA GOMX-4B satellite and it is operational since February 2018. 6) The HyperScout-2 specifications are reported in Table 1. The optical front-end is based on a monolithically and reflective TMA configuration, based on freeform optical design and manufactured using diamond turning machining technique. The dual channel implementation envisages a beam splitter at the end of the front end optics, diverting the shorter wavelengths towards the VNIR sensor, and allowing the longer wavelengths to impinge on the TIR sensor.
Table 1: HyperScout-2 specifications
Figure 1: Left: HyperScout-1 Protoflight Model, Right: HyperScout-2 Protoflight Model (image credit: cosine Remote Sensing, ESA/ESTEC)
This configuration was envisaged during other ESA activities related to navigation and planetary science, with goal of designing a multispectral camera for relative navigation. 7) A very compact relay has been conceived in order to enhance the numerical aperture of the TIR channel. The VNIR Focal Plane Array (FPA) is a CMOS sensor with a linearly variable hyperspectral filtering element that separates the different wavelengths from 400 to 1000 nm, while the TIR FPA is based on a microbolometer and a set of four spectral filters from 8 to 14 µm. Both FPAs are 2D sensors operated in push broom mode. The TIR sensor is actively thermally controlled via a thermoelectric cooler (TEC) to limit the dependency of the sensor readout on the sensor temperature fluctuations.
HyperScout-2 has the following subsystems:
• Telescope assembly;
• VNIR Focal Plane Array (VNIR FPA);
• TIR Focal Plane Array (TIR FPA);
• Back-End Electronics unit (BEE);
• On-Board Data Handling (OBDH);
• Mass Memory Units (MMUs): operated in hot and cold spare configuration;
• Instrument Control Unit (ICU): main point of contact from the S/C bus for data interface and main instrument control;
• Eyes of Things board (EOT): data processing board for on-board artificial intelligence algorithms.
The architectural diagram of HyperScout-2 is depicted in Figure 2, while the subsystems locations in the payload is indicated in Figure 3. The back-end electronics (BEE) is a compact and versatile FPGA based single board image acquisition system. The On-Board Data Handling (OBDH) is physically installed on the BEE. The Instrument Control Unit (ICU) controls and monitors the operational status of the payload subsystems. The Vision Processing Unit (VPU) consists of a Myriad 2 processor located on the Eyes of Things (EoT) board. The ICU can independently power the BEE, OBDH, the EoT board and each MMU. The FPAs are independently powered via the BEE. All subsystems have independent latch-up and overcurrent protection.
Figure 3: Illustration of the HyperScout-2 instrument (image credit: cosine Remote Sensing, ESA/ESTEC)
The majority of the HyperScout subsystems can be operated independently from each other, offering an additional degree of freedom to optimize the power budget by powering up only the components that play a role within the specific operating mode. The ICU is always powered to serve as the contact point for communication with the S/C bus and to monitor the status of the payload throughout the orbit. During acquisition the FPAs, BEE and MMU are mainly used. In this phase for example, the OBDH will consume minimal power transferring the image data to disk. During processing both FPAs and BEE are powered off completely. The OBDH will normally consume the most power in the processing mode. The dependency of the subsystems per operating mode is reported in Figure 4.
The frequency of the observation mode can vary according to the mission planning. The acquired data is cached in the instrument MMUs for subsequent processing or transmission to ground, as requested by the S/C bus. The processing operating mode can be run using different processing subsystems. For this scope HyperScout-2 offers a powerful computational set of processors, including a FPGA, CPU, GPU and a dedicated board for AI inference represented by an Eyes Of Things (EOT) board consisting of the Intel® Movidius™ Myriad™ vision processing unit (VPU) designed for accelerating machine vision tasks. The Intel's Movidius™ Myriad™ 2 VPU (Ref. 5) (second generation VPU from Movidius™, an Intel® company) is specialized for AI, vision and imaging applications where both performance and low power consumption are important.
HyperScout-2 is therefore capable of performing a vast range of processing operations, that can be continuously updated during flight, even with completely new applications.
Figure 5: Right: Intel's Movidius Myriad 2 board, Left: as integrated on the top of the HyperScout-2 electronics stack (image credit: cosine Remote Sensing, ESA/ESTEC).
HyperScout-2 calibration and applications
In-flight Calibration approach: HyperScout-2 has been characterized in the laboratory before integration into the spacecraft. For what concerns in-flight calibration, typically on-board stimuli are used to calibrate the full optical chain for the long waves infrared channel, such as blackbodies. For a miniaturized instrument like HyperScout this is currently not an option. A lean system approach is adopted that relies on vicarious calibration in combination with a thermally stable instrument and detector.
For the absolute calibration of the HyperScout VNIR spectral channel, instrumented test sites of RadCalNet are considered. Test sites consist of large, homogeneous, cloud-free areas used as radiance or reflectance reference targets. The method relies on the comparison of satellite date with a simultaneous ground truth measurements. Two methods are distinguished:
• Reflectance based: the reflectance of the earth surface is measured on-ground, together with extinction depth and other meteorological parameters. Subsequently a radiative transfer model is used to convert these values to a TOA (Top of Atmosphere) radiance, which can be compared to the sensor data. These methods allow absolute radiometric calibration with a accuracy <5%.
• Radiance based: the radiance of the earth scene is measured at an altitude much above aerosol scattering. After correction for residual scattering and absorption, the radiance can be compared directly to the sensor for absolute radiometric calibration within 2.8% accuracy.
Pseudo invariant characterization sites (PICS), are sites with a high temporal or spatial stability. Desert sites are the most promising candidates as they are characterized by large, homogeneous areas and constant atmospheric conditions (no clouds). An extensive catalogue with site information is available on the websites of the USGS and CEOS. The most common application is to monitor the relative radiometric response of the sensor, without the need of an on-ground reference measurement.
Data will be crossed calibrated on the long term using Sentinel-2A/-2B, acquired over the same region of interest (ROI). For the purposes of cross-calibration of HyperScout, the Sentinel-2 satellites are considered thanks to the good match between the spectral bands of the Sentinel Multispectral Instrument (MSI) and HyperScout and the in-orbit HyperScout data collection opportunity provided by the FSSCat spacecraft with limited costs for cosine.
Figure 6: Diagram of a cross acquisition between HyperScout and Sentinel satellites. Given the 180º separation between the Sentinel-2A and Sentinel-2B orbit, the crossed acquisition with HyperScout can be performed with only one Sentinel at the time (image credit: cosine Remote Sensing, ESA/ESTEC)
For the TIR channel in-flight calibration, a spacecraft maneuver with the instrument pointed at deep space is used to measure the instrument thermal background signal. As there is no black body calibrator on-board the payload for in-flight calibration, two options are considered: 1) intercalibration or cross-calibration, 2) vicarious calibration. Intercalibration calibrates a satellite instrument by relating its measurements to those of a well calibrated reference satellite instrument. This technique relates measurements in similar spectral bands. Well-calibrated reference satellite instruments that match each TIR band of HyperScout-2 are for example Sentinel-3, MODIS or Landsat-8. Vicarious calibration uses natural or artificial sites on the surface of the earth to calibrate satellite instruments. Within this technique, well-calibrated ground-based or airborne radiometers take measurements of a spectrally and spatially homogeneous test sites at the time of the satellite instrument overpass.
Two methods can be used to perform vicarious calibration in the thermal infrared:
• Radiance-based method;
• Temperature-based method.
The radiance-based method requires that the spectral response of the well-calibrated reference radiometer matches the spectral response of the satellite instrument. In practice, this condition is most likely not going to be fulfilled. Consequently, this method is also not considered to be a suitable option for inflight monitoring and calibration. In contrast to the radiance-based method, the temperature-based method in principle does not impose restrictions on the spectral response of the instrument. Hence, this method is considered a suitable option for in-flight monitoring and calibration. The following calibration sites could be envisaged for the in-flight calibration of HyperScout:
• Lake Tahoe on the California/Nevada border
- ideal thermal calibration target due to thin atmosphere above the lake thanks to the high altitude
- does not freeze in winter as it is very deep
- annual temperature ranges from about 4 to 20°C
- excluding instrument noise, uncertainty in the predicted at-sensor spectral radiance for TIR bands and skin surface temperature of 300 K is about 0.43 K - expressed in equivalent apparent temperature
- freely available data
• Salton Sea in Southern California
- less ideal target than Lake Tahoe due to thicker atmosphere (lower altitude)
- annual temperature variation ranges from about 4 to 35°C (extends temperature range compared to lake Tahoe)
- excluding instrument noise, uncertainty in the predicted at-sensor spectral radiance for TIR bands and skin surface temperature of 300 K is about 0.43 K - expressed in equivalent apparent temperature
- freely available data
• NOAA Ocean and Great Lakes
- larger surface area than Lake Tahoe and Salton Sea
- temperature variation ranges from about 3 to 30°C depending on the season and location
- excluding instrument noise, uncertainty in the predicted at-sensor spectral radiance for TIR bands and skin surface temperature of 300 K is about 0.48 K - expressed in equivalent apparent temperature
- freely available data.
cosine is also leading a Dutch consortium for the development and commercialization of a miniaturized spectroradiometric calibration stimuli, expected to be integrated and tested in orbit in late 2020 / early 2021. This miniaturized calibration source will be considered for future flights of HyperScout related products as well as third parties compact spectrometers.
HyperScout-2 data products
A number of data products have been selected for HyperScout-1 and this is being extended for the HyperScout-2 mission adding the potential applications enabled by the coexistence of reflectance and thermal information of the target. The target applications are: agriculture, thermal inertia and soil moisture, urban heat island, fire hazard and monitoring, and water quality.
In-Orbit Test Bed: A Platform for Artificial Intelligence Experiments
Applying Artificial Intelligence processing algorithms on the data acquired by HyperScout-2 directly onboard and in real time may represent a big leap forward in inferring and delivering Earth Observation derived information at a very high temporal resolution and with a very short timeliness. Alternatively, an interesting use case is the screening of the data before being downloaded. This is the case for the first demonstration that will be performed in orbit as part of the Φ-Sat-1 mission, where AI is leveraged to select the data to be downloaded based on the cloud coverage of the acquired scene. The application has been selected by ESA as the presence of clouds is one of the fundamental problems of optical remote sensing and it becomes particularly relevant considering the large data volume generated by Earth Observation missions based on hyperspectral imaging.
However, it should be noted that the HyperScout-2 processing chain has been designed in order to accept third party AI based software that can be run on the either on the VPU, or CPU/GPU, for testing purposes before being employed in operational scenario. The processing chain is depicted in Figure 7.
The first step prepares the data and corrects it for known aberrations. After the raw data is corrected, the spectral cube can be computed in order to produce spectral bands. The second pre-processing step is mandatory if the VPU is used. In this case the spectral cube is prepared and loaded into the VPU, ready to be analyzed by the AI algorithm. The inference step is use-case based, and as discussed can be performed on the VPU if high time efficiency is required.
Figure 8: Φ-Sat-1 is the first experiment to demonstrate how onboard artificial intelligence can improve the efficiency of sending Earth observation data back to Earth. This revolutionary artificial intelligence technology will fly on one of the two CubeSats that make up the Federated Satellite Systems (FSSCat) mission (image credit: ESA)
Launch: The Φ-Sat-1 6U CubeSat mission is scheduled for launch on 19 June 2020 as enhancement of the FSSCat mission, during the VEGA SSMS Proof Of Concept flight. 8)
Orbit: Sun-synchronous orbit, altitude of 540 km, eccentricity: ~0, LTAN = 10:30 hours.
The FSSCat mission the mission will demonstrate for the first time worldwide a reliable optical intersatellite link (O-ISL) between two 6U CubeSats flying in LEO (Low Earth Orbit). We have developed a technology demonstration with a full duplex O-ISL terminal of 1.5U in size (15cm x 10cm x 10cm), 85mm optical aperture, suitable for LEO to LEO operations on coarse pointing CubeSats (pointing accuracy < 0.5º).
The terminal adopts a novel adaptive variable divergence laser mechanism with a hybrid payload/platform seek-and-track pointing algorithm, and an amplitude modulated signal to ensure reliable optical communications at 1 Gbit/s at a nominal intersatellite distance of up to 2,000 km in LEO. In addition of establishing an intersatellite link, the terminal provides the ability for simultaneous imaging through the optical aperture and the improvement of the attitude determination capability of the hosting satellite through the mixing of optical information in the overall platform attitude feedback control loop. 9)
Figure 9: FSSCat is a constellation of two 6U CubeSats that provide data on Earth’s ice and soil moisture content to complement the Sentinel fleet. FSSCat took the top prize at the 2017 Copernicus Masters Competition (image credit: UPC)
1) ”First Earth observation satellite with AI ready for launch,” ESA, 12 September 2019, URL: http://www.esa.int/Our_Activities/Observing_the_Earth
2) ”Smallsats win big prize at Copernicus Masters,” ESA, 8 November 2017, URL: http://www.esa.int/Our_Activities/Observing_the_Earth
Marco Esposito, Bernardo Carnicero Domınguez, Massimiliano Pastena,
Nathan Vercruyssen, Simon Silvio Conticello, Chris van Dijk, Pierluigi
Foglia Manzillo, Rick Koeleman,”Highly integration of
hyperspectral, thermal and artificial intelligence for the ESA PhiSat-1
mission,” Proceedings of the 70th IAC (International Astronautical Congress), Washington DC, USA, 21-25 October 2019, URL: https://iafastro.directory/iac/proceedings
Massimiliano Pastena, Bernardo Carnicero Domínguez, Pierre
Philippe Mathieu, Aman d a Re ga n,Marco Esposito, Simon Conticello,
Chris Van Dijk, Nathan Vercruyssen, Pierluigi Foglia Manzillo, Rick
Koelemann, John Hefele, ” Earth Observation Directorate NewSpace
Initiatives,” Proceedings of the 70th IAC (International Astronautical Congress), Washington DC, USA, 21-25 October 2019, paper: SSC19-V-05, URL: https://digitalcommons.usu.edu/cgi
5) A Camps, A. Golkar, A. Gitierrez, J. Ruit-de-Azua, J. Muñoz‐Martin, L. Fernandez-Capon, C. Diez, A. Aguilella, S. Briatore, N. Garzaniti, F. Nichele, R. Mozzillo, M. Vanotti, M. Esposito, B. Carnicero, G. Filippazzo, and A. Regan, “FSSCat: A Cubesat-based Tandem Mission for Earth Observation of the Polar Regions” Living Planet Symposium ‘19, Milan, May 2019
6) Marco Esposito, Alessandro Zuccaro Marchi, “In-Orbit Demonstration of the first hyperspectral imager for nanosatellites” ICSO ( International Conference on Space Optics) 2018, Chania, Greece
Pierluigi Foglia Manzillo, Ljubisa Babic, Marco Esposito, Cristian
Corneliu Chitu, Marco Canetri, Paolo Corradi, Jesus Gil Fernandez,
Michael Kueppers. ”TIRI: A Multi-Purpose Thermal Infrared Payload
for Planetary Exploration” Proceedings of the 69th IAC
(International Astronautical Congress) Bremen, Germany, 1-5 October
2018, paper: IAC-18,A3,4B,4, URL: https://iafastro.directory/iac/proceedings/IAC-18
8) ”FSSCat/Φ-Sat-1 ready for launch,” ESA Applications, 16 June 2020, URL: https://www.esa.int/Applications/Observing_the_Earth
9) ”Overview of the FSSCAT Optical Intersatellite Link (O-ISL) - Technology Demonstration,” EPFL, 2019, URL: https://memento.epfl.ch/event/overview-of-the-fsscat-optical-intersatellite-link/
The information compiled and edited in this article was provided by Herbert J. Kramer from his documentation of: ”Observation of the Earth and Its Environment: Survey of Missions and Sensors” (Springer Verlag) as well as many other sources after the publication of the 4th edition in 2002. - Comments and corrections to this article are always welcome for further updates (email@example.com).