Incorporating Redundant Systems to Capture the Kentucky Money Shot

The Kentucky Eclipse Ballooning Project began in early 2015 when students and faculty from The University of Kentucky attended the NASA Marshall Space Flight Center BalloonSat Workshop in Huntsville, Alabama. The students accelerated their preparations after the Eclipse Ballooning Project Workshop hosted in Bozeman, Montana where they built and learned systems designed by Montana Space Grant. In 2016, the students began a sequence of 10 balloon launches in preparation for the total solar eclipse on August 21, 2017. In the early stages of this project, University of Kentucky students set the goal to capture footage of a separate high-altitude weather balloon in front of the solar eclipse, an image dubbed “The Kentucky Money Shot.” After establishing that goal, students began working on approaches and designs to capture this picture with one overarching theme: redundancy. Every aspect of the project from the number of balloons and imaging systems to tracking systems and launch procedures were designed with redundant aspects and through collaboration among the payload, ground station, launch, and mission control teams. The short time window of eclipse totality, 2 minutes 28 seconds, motivated design iterations throughout the progressive practice launches and ground tests including launching two balloons simultaneously, streaming and storing footage of the flight from multiple cameras, and using SPOT Trackers and Iridium systems as multiple tracking approaches. All of these practices and tests led to flying the final redundant designs on August 21 st , 2017 to successfully capture “The Kentucky Money Shot”.


I. Introduction
The Kentucky Eclipse Ballooning Project began in early 2015 when students and faculty from The University of Kentucky attended the NASA Marshall Space Flight Center BalloonSat Workshop in Huntsville, Alabama.The students' efforts then accelerated after the Eclipse Ballooning Project Workshop hosted in Bozeman, Montana where they built and learned systems designed by Montana Space Grant. 8,9In 2016, the students began a sequence of 10 balloon launches in preparation for the total solar eclipse on August 21, 2017.
In the early stages of this project, University of Kentucky students set the goal to capture footage of a separate high-altitude weather balloon in front of the solar eclipse, an image dubbed "The Kentucky Money Shot".After establishing that goal, students began working on approaches and designs to capture this picture with one overarching theme: redundancy.
In this paper, examples of redundancy in successful NASA missions are considered first.Following that, the UK Eclipse Balloon Team's approach to incorporating redundancy is presented, along with results of the UK team's dry run and eclipse balloon missions for the 2017 Eclipse.

II. Background
In 2015, early in the planning stages for the UK Eclipse Balloon Team project, redundancy was adopted as a primary concept to ensure success capturing the Kentucky Money Shot (KMS).The NASA Systems Engineering Handbook introduces redundancy under requirements for mission reliability and safety. 1 To satisfy mission reliability requirements, redundancy, fault tolerance and design robustness have trade-offs to be considered.For safety, diverse functional redundancy is a primary design driver.
Identical or near-identical components or subsystems (i.e., redundant subsystems) are frequently used in NASA missions.The 2005 final report of the NASA Exploration Systems Architecture Study 2 presented redundancies and contingencies related to the rendezvous and docking mission sequence.Redundancies combined with performance margins are credited with lowering risk in the event of failures and off-nominal situations, contributing to the 100-percent success rate of U.S. rendezvous and docking missions.Redundancies are either "like" or "unlike" highlighting the importance of different approaches -having different failure modes -for key subsystems.Consequently, the overall system does not include single-point failures.Along with redundancies, operational contingencies are planned and can be executed if needed for mission success.
In 2004, Paté-Cornell, et.al.evaluated the benefits and costs of redundancy in NASA spacecraft programs. 3edundancy adds costs affecting system weight and other limited resources, as well as adding financial costs and development time, especially for "unlike" redundancies.For a system of parallel components where only one has to be operational at a time, the cost of redundancy and probability of success are closely tied to performance independence.Identical subsystems may have identical failure modes from a common design flaw or when subjected to the same external conditions.Parallel subsystems that are entirely different require considerable additional design, evaluation, and integration effort.However, parallel redundant subsystems with some aspects in common may realize reduced development and financial costs while maintaining a level of performance independence to increase mission probability of success.Successful NASA missions that used dual identical or near-identical spacecraft, seen in Figure 1, include the following (launch year): Viking (1975), Voyager (1977), Mars Exploration Rover (MER) mission Spirit and Opportunity (2003), and STEREO space weather detection satellites (2006).Viking's identical orbiter-lander pairs reached orbit and landed in 1976, where they were the first U.S. missions to land on Mars and transmit images back to Earth. 4 Viking 1 and 2 rovers both significantly exceeded their design life of 90 days, continuing for four and two years, respectively.The Voyager 1 and 2 twin spacecraft completed their primary 5-year mission to explore Jupiter and Saturn and are still operational today for the Voyager Interstellar Mission to explore the outer solar system and extent of the sun's influence. 5Launching identical systems at essentially the same time can preclude using a failure in one system to prevent the same failure in the other.Even so, a failure of the primary communication system of Voyager 2 resulting from operational procedures enabled operational changes for Voyager 1 to prevent the same failure. 3e twin Mars robotic geologists, Spirit and Opportunity, arrived at Mars in 2004 and landed at different sites for exploration. 6The MER program faced trade-offs of launching two identical systems within a single near-term 2003 window of opportunity, risking identical failures due to common external conditions or potential flaws from an overly ambitious schedule. 3As a final example, selective redundancy was included in the two nearly-identical Stereo observatories that lead and trail the Earth in its orbit to focus on the flow of Sun-Earth energy and matter.The two-year primary mission was successfully completed, but extended missions continue, including attempts to regain communications with the Stereo B spacecraft that were interrupted in 2014 and discontinued in 2016. 7e NASA missions inspired the use of dual/near-identical systems by the UK Eclipse Ballooning Team for their two balloons launched on 21 August, 2017.Details follow of how redundant parallel subsystems that are identical and different (i.e., "like" and "unlike") were implemented by the UK team for success.
Note also that 70 nearly-identical parallel systems were launched for the nationwide Eclipse Live Stream creating an overall mission redundancy more similar to that of envisioned future CubeSat swarms than to the dualidentical redundant UK team mission described herein.Determining probabilities of success is beyond the scope of this paper, but could be completed as a follow-on effort.

III. Methods
In order to ensure success both for the national goal of live streaming the eclipse from the edge of space and the UK Eclipse Ballooning Team goal of capturing the Kentucky Money Shot, various methods of redundancy were used in four areas of the project: the launch procedures, ground station systems, payload designs, and personnel assignments.

Launch
The UK Eclipse Ballooning Team determined early on that the single point of failure for the project would be lack of helium.Due to the potential national shortage of helium it was more difficult and more expensive to obtain and store helium.Contingencies for a failed launch required having enough helium to fill three balloons, the two launch balloons plus one spare.In preparation, the team stored upwards of 25 tanks of helium for use by multiple universities on Eclipse Day.
With two balloons being launched, the UK Eclipse Ballooning Team was comprised of two completely independent launch teams.Each team had all the tools required to successfully launch a high altitude weather balloon.This ensured that in the event of a malfunction of any one part, there was a spare owned by the second team.This process began after an early practice launch in which the gasket of the only helium regulator failed, forcing removal of part of the flight string payloads so that the system had enough lift to ascend.This redundancy not only ensured all equipment was accounted for, but allowed for a simultaneous double balloon launch.This cut helium fill times to minutes, allowing the ground station and payload teams to power up systems closer to launch, saving battery life and extending the capture time of payloads in flight.

Ground Stations
A major component in achieving the goals set by the UK Eclipse Ballooning Team was the successful operation of ground stations in parallel and implementing redundancy throughout.Each team across the country had identical ground stations for use of receiving still images, streaming video, and tracking the balloon.These stations had a pair of antennas for receiving the still images, a dish antenna for receiving video, and tracking software and hardware for using the satellite-communicated GPS data received at the ground station via wireless or hardwired connection to the internet from the Iridium satellite payload on the balloon 8,9 .As it became clear the ground stations were mission critical for accomplishing the team's goals, incorporating redundant ground station systems became the path to success.
Early on, the team decided to launch two balloons, which requires two ground stations.The original set-up provided to all the teams involved only one ground station.The UK team decided to have four; one main system for each balloon and one backup system.The original design also only used a single laptop, a setup the UK team used early in project.While the team had success using one computer to stream video and track the balloon, the team's final design had three computers.The differences in set-ups can be observed in Figure 2. The use of three laptops allowed for a lighter workload on each system with one computer used for tracking, one for streaming to Stream for the national eclipse live stream, and the third for streaming to the team's YouTube channel.Even though only one computer was necessary to operate the system, three allowed the system to be more efficient.Three computers added redundancy to the system including two backup computers for each station.With a total of six laptops for both systems, there were four extra laptops if needed.In early tests, computers used for tracking and streaming were connected to the Internet via Wi-Fi.During the national dry run in June 2017, the team was unable to test video streaming capabilities due to poor internet connection.For a reliable connection with enough bandwidth for the multiple video streams, the team decided to use a hardwired connection, with Wi-Fi or wireless hotspot as contingency backup.The team also moved to a different launch site with more reliable wired and wireless connection and additional bandwidth.
Each ground station setup was identical for the two balloons.Each balloon carried the same Iridium, still image, and video payload 8,9 .If needed, these three payloads all had an additional identical spare payload ready.

Payloads
The UK Eclipse Ballooning team payloads were intended to meet two goals; live stream the eclipse from the edge of space, and capture the KMS.Much like the other aspects of this project, the main strategy to accomplish these goals was the use of redundant systems.The live streaming goal was to be achieved using a system developed by the Montana Space Grant at Montana State University.This system utilized a Rocket M5 modem to transmit live video from an onboard Raspberry Pi to the ground station.The use of two identical but separate systems allowed for dual live stream vantage points, one observing the eclipse happening while looking up, and the other viewing the shadow of totality on the earth while looking out.
In order to ensure that images and video were captured, GoPro cameras were flown on both flight strings.This redundancy did not allow for real-time viewing, but allowed for more reliable footage due to the consistency of the cameras.Each flight string held four GoPro cameras, three facing out to view the earth, and one looking directly up at the eclipse.
The final redundant camera system was the implementation of a Garmin 360 camera.This allowed us to view the eclipse in all directions in a higher quality than other video stitching methods.This device was also able to provide another method of recording in flight metrics such as pressure, altitude, and speed.Editing software with the camera allowed for stabilization of footage, increasing the quality of video captured.
Most of the remaining equipment flown was tracking equipment so that recovery of the team's expensive camera equipment was swift and simple.The most accurate of the tacking devices was an Iridium modem.The FAA required each team participating in the national ballooning project to fly this device.Through the Iridium network, the FAA could see the exact location of all 50+ teams balloons at once.This tracking device was able to provide real-time updates of altitude, velocity, and latitude-longitude data.Unlike other tracking methods employed, the altitude and colder temperatures did not affect the performance of the Iridium system, so connection was maintained throughout the entire flight.The longest tenured tracking device the UK Eclipse Ballooning Team used was the SPOT satellite personal tracker.This device gave precise GPS location through packets received every five minutes.While SPOT trackers are accurate, its operating altitude does not exceed over 20,000 feet, so the team would lose connection until the balloon popped and descended by parachute below this altitude again.Not only was SPOT a different redundant tracking device than the Iridium, but each flight string also used two individual SPOT trackers.The use of two trackers ensured there would still be communication if one did not reconnect after descending.
In redundant fashion, the UK Eclipse Ballooning Team decided to reintroduce a high altitude tracking system, or H.A.T.S., which utilizes the 908 MHz frequency to communicate altitude, temperature, and latitude longitude coordinates, in real time.This addition marked the third different tracking system implemented the flight string along with SPOT and Iridium tracking.This device had been a consistent tracking method early in the project, but was replaced by the Iridium tracker when it was received.
The use of two balloons and two flight strings was a redundancy in itself (see Figure 3).If a balloon failed to make it to altitude during the eclipse, there was another balloon to increase the chances of meeting the project goals.In order maximize success, equipment was split into two near identical groups, then assigned to a flight string.Each flight string consisted of at least five cameras and three tracking devices, with Balloon 1's flight string having the additional H.A.T.S. tracker, and Balloon 2's flight string having the Garmin 360 camera.

People
In order to support launching two balloons, the team leaders decided that more people would need to be brought onto the project to help with tasks on launch day.This project offered multidisciplinary hands-on experience accessible even for underclass engineering students.After recruiting 30 students the first semester, we focused on teaching the general basics of high altitude ballooning and introducing the existing systems.These new students were divided into three groups based on their interests: learning the basics and further developing the ground station systems, GoPro payload development, and balloon fill and launch procedures.The work that semester culminated in the first large team practice balloon launch in November 2016.The next semester, groups were reorganized in order to better accommodate the needs of future launches.As progress continued, every system had at least two students learning and handling it, so that there were experts for redundant payloads on the first double balloon practice launch in April 2017.In addition, the timeline leading to launch was created to be systematic, so each practice launch was similar to the expected timeline on Eclipse Day.This way, if a problem arose, there was either time built into the schedule or a contingency written to smoothly continue with the timeline.

IV. National Dry Run Results
On June 20 th 2017, the UK Eclipse Ballooning Team, along with 50+ other universities, performed a dry run of all the systems for Eclipse Day.The team's goal was to test the equipment and perfect a double balloon launch.Many of the final-design redundancies were first tested on this date, or stemmed from the lessons learned on the dry run.
During the national dry run, the UK team launched two 3000g balloons both with nine pound, three ounce flight stings.A simultaneous fill using the two separate launch teams described earlier was implemented for the first time.The process went very smoothly and both balloons were launched at precisely the same time.Having two separate fill teams allowed for a smoother launch procedure.The accuracy of payload weights, fill volumes, and timeline management allowed for both balloons to stay close to each other during the hour and a half ascent.Footage of Balloon Two was captured from Balloon One at an altitude of about 80,000 feet and a separation of about 1.25 miles.The close proximity in launch also resulted in close landing locations, approximately 1.16 miles seen in Figure 5.   Ground station tracking was connected on the ground but failed at takeoff.After receiving packets from the Iridium server, the ground station did not automatically track the balloons, so the team was forced to switch to manual tracking to continue to live stream video.Eventually the live stream feed cut out due to lack of internet connection at the launch location.Numerous speed tests would later determine that the 0.5 Megabytes per second (Mbps) upload speed would not be sufficient to upload a stream to YouTube or the national NASA Eclipse Balloon Project website at stream.live.With the results from the dry run, the team was able to reduce numerous risk probabilities, but also found many new problems that would have to be solved before the eclipse.
The dry run exposed inconsistencies of some payloads, in particular the still image payload.Prior to launch, an uncommon software error was preventing successful image capturing on both still image systems.After unsuccessful attempts to resolve the error, the team was forced to follow the timeline and fly the faulty packages.The team was still able to capture images and video from the redundant GoPros.These payloads had recently been updated so that the GoPro cameras could be powered on without opening the package.This design change allowed for power-up to occur at T-1 minute to capture the most footage in the air.Also, the team had success in capturing footage via the live streaming payloads.While the streaming portion was not fully working, the footage was saved onboard and was able to be evaluated post flight.
The number of redundancies for tracking ensured the team was able to recover payloads.There were six trackers, three on each balloon that could lead us directly to the landing site.While ground station automatic tracking did not work (see Figure 6 for progression of system performance through the practice launches), all six trackers maintained connection at their various ranges and specifications, leading directly to the landing points.The two dry run balloon strings landed within just over a mile from each other after crossing the Barren River Lake.One just missed being lost in the lake, which highlighted the increased risk of two identical systems experiencing identical environments.Both could easily have been lost in the lake one month prior to the Eclipse.

Ground Station
Ground station success on August 21, 2017, stemmed directly from redundant computer systems implemented in the system's design leading to eclipse.Every challenge that arose was quickly fixed or overcome due to the redundant design.Early in the pre-flight timeline, the first issue encountered was missing critical hardware for one of the primary ground stations.The ground station leaders quickly decided to use a spare ground station system instead.As the timeline continued, one computer began to overheat and could no longer be used.As previously mentioned, the ground station system was originally designed to work only on one computer, but even though a computer overheated in the new three-computer setup, the ground station students easily transitioned to two computers with one for balloon tracking and one for streaming video.The other ground station did not have any complications, so all three computers were used.A third challenge appeared after one of the balloon's Iridium tracking system failed upon launch.While there was not a spare Iridium flown on that flight string, the team had previously practiced tracking the balloon manually using signal strength indicators during several of the practice launches.This technique successfully captured live stream video.
While the systems used allowed the team to overcome challenges with success, they also led to the team having the most successful launch yet.The team held connections with video payloads for the longest they had since starting to work on the project.This came from the successful tracking of balloons.It also allowed streaming video to Stream, accomplishing the national goal of the project, and to YouTube, one of UK's goals.The team previously decided that no still images would be received "live" on Eclipse Day, but the still image systems still flew saving the images onboard.

Launch
The launch procedure on Eclipse Day very closely followed the successful model of the dry run.Separate teams filled two balloons simultaneously.Unlike the dry run, during the launch of Balloon One, the flight string snapped due to the jerk at launch.The team went ahead and launched Balloon Two while starting the contingency for a failed balloon launch.A spare 2000 gram balloon was filled with the extra helium brought by the team, and twenty minutes after the first balloon launched, the second balloon followed.This is a direct result of the contingencies plan of the launch team to have extra balloons and helium in case of failure.When the helium regulator failed during one of the early practice launches, the launch team members therefore had experienced an unexpected launch failure, enabling their rapid recovery and eventual success on Eclipse Day.

Payloads
On Eclipse Day, the team had the most success over all payloads.The video streaming payloads were able to stream to both YouTube and stream.livefor a range of over 14 miles.The GoPro cameras were able to capture video through the totality for footage of the approaching eclipse, the shadow on the earth, as well as upward video of the eclipse itself.The Garmin 360 camera provided great footage that was stabilized post flight, but did not have the battery life to record totality.The still image cameras were successful as well, capturing upward pointing images of the eclipse entering totality.The team was not able to view photos in real time, however photos were stored onboard for post flight recovery and viewing.Figure 6.The UK Eclipse Balloon Team timeline of all payloads and their success as the project progressed.

Tracking
All four SPOT tracking devices functioned through flight, as well as the H.A.T.S. payload.One Iridium worked the entire flight as well, while the second malfunctioned at takeoff.The working trackers allowed for recovery of both flight strings that landed in the middle of nearby soybean fields.

People
For the eclipse, due to predictions of excessive traffic and unavailability of additional lodging at the first hotel booked two years in advance, the team ended up split into two groups.The team was divided by balloon responsibilities, which provided an unlike redundancy to address traffic concerns by traveling to the designated launch location in Russellville, KY from two different directions.The groups were split so that, if one team could not make it to the launch site, then the other team had full autonomy for their systems.For example each group had one student leader for general management and organization, two students for ground station set-up and operation, a leader for balloon fill, and at least one student for payload operation.Assigning people mirrored tasks for each group lowered the risk of not launching a balloon for the eclipse.
While the entire team made it to Russellville County High School, balloon fill, payload startup, and ground stations were still operated by the same students assigned to the separate travel groups.Each balloon group, Balloon One or Balloon Two, worked through the timeline starting 90 minutes before launch and 2.5 hours before the eclipse.All of the practice launches allowed for seamless collaboration and communication among the balloon groups during the timeline.Even though the first flight string launched came crashing down due to the cut down string failure, a preplanned contingency was enacted flawlessly to continue to launch Balloon Two while refilling another balloon for Flight String One.Everyone on Balloon One knew exactly what to do for this setback and, due to the redundant responsibilities, the Balloon One group quickly reset the flight line and refilled the new balloon, which launched 15 minutes later.
The second balloon launched, officially labeled Balloon 1, included the H.A.T.S. tracking system.This system sent packets of information with altitude, latitude, longitude, and temperature to a receiving antenna at one ground station during the flight.Figures 7 and 8 show the data for the eclipse flight.
Overall, the 24-person team on Eclipse Day, which was 40% women and represented eight different STEM majors, accomplished both the national and team goals due to incorporating redundancy in all aspects of the system design.

VI. Money Shots
All of the efforts mentioned previously led to the capturing of these pictures that the UK Eclipse Balloon Team selected as their "money shots."

Figure 2 .
Figure 2. Montana single ground station design (top) with Wi-Fi connection to the internet, and UK's Eclipse Day redundant balloon and ground station design (bottom), including hardwired connection to the internet and the addition of multiple computers at each ground station to separate operations for tracking, streaming to YouTube and streaming to Stream.

Figure 3 .
Figure 3. Payload manifests for the two balloons on Eclipse Day

Figure 4 .
Figure 4.A time-lapse of fill and launch procedures from national dry run.The first three images taken from the ground while the final image take from a fixed-wing aircraft.

Figure 5 .
Figure 5.A map with the two landing locations of the dry run l balloons plotted over Google maps.

Figure 7 .
Figure 7. Balloon One altitude during the flight vs time, with point of eclipse totality indicated.

Figure 8 .
Figure 8. Plot showing the temperature vs altitude for Balloon One on Eclipse Day (top).Plot showing the path of Balloon One on Eclipse Day (bottom).Point of eclipse totality indicated.

Figure 9 .
Figure 9. Money Shot 1: The Eclipse's umbra coming.The white speck is Temple University's weather balloon.

Figure 11 .
Figure 11.Money Shot 3: Eclipse totality looking up at a University of Kentucky's weather balloon (left).Money Shot 4: Balloon burst after the eclipse around 104,000ft (right).