Recoding Voyager 1—NASA’s interstellar explorer is finally making sense again

Engineers have partially restored a 1970s-era computer on NASA's Voyager 1 spacecraft after five months of long-distance troubleshooting, building confidence that humanity's first interstellar probe can eventually resume normal operations.

Several dozen scientists and engineers gathered Saturday in a conference room at NASA's Jet Propulsion Laboratory, or connected virtually, to wait for a new signal from Voyager 1. The ground team sent a command up to Voyager 1 on Thursday to recode part of the memory of the spacecraft's Flight Data Subsystem (FDS), one of the probe's three computers.

“In the minutes leading up to when we were going to see a signal, you could have heard a pin drop in the room," said Linda Spilker, project scientist for NASA's two Voyager spacecraft at JPL. "It was quiet. People were looking very serious. They were looking at their computer screens. Each of the subsystem (engineers) had pages up that they were looking at, to watch as they would be populated."

Finally, a breakthrough

Launched nearly 47 years ago, Voyager 1 is flying on an outbound trajectory more than 15 billion miles (24 billion kilometers) from Earth, and it takes 22-and-a-half hours for a radio signal to cover that distance at the speed of light. This means it takes nearly two days for engineers to uplink a command to Voyager 1 and get a response.

In November, Voyager 1 suddenly stopped transmitting its usual stream of data containing information about the spacecraft's health and measurements from its scientific instruments. Instead, the spacecraft's data stream was entirely unintelligible. Because the telemetry was unreadable, experts on the ground could not easily tell what went wrong. They hypothesized the source of the problem might be in the memory bank of the FDS.

There was a breakthrough last month when engineers sent up a novel command to "poke" Voyager 1's FDS to send back a readout of its memory. This readout allowed engineers to pinpoint the location of the problem in the FDS memory. The FDS is responsible for packaging engineering and scientific data for transmission to Earth.

After a few weeks, NASA was ready to uplink a solution to get the FDS to resume packing engineering data. This data stream includes information on the status of the spacecraft—things like power levels and temperature measurements. This command went up to Voyager 1 through one of NASA's large Deep Space Network antennas Thursday.

Then, the wait for a response. Spilker, who started working on Voyager right out of college in 1977, was in the room when Voyager 1's signal reached Earth Saturday.

"When the time came to get the signal, we could clearly see all of a sudden, boom, we had data, and there were tears and smiles and high fives," she told Ars. "Everyone was very happy and very excited to see that, hey, we're back in communication again with Voyager 1. We're going to see the status of the spacecraft, the health of the spacecraft, for the first time in five months."

Throughout the five months of troubleshooting, Voyager's ground team continued to receive signals indicating the spacecraft was still alive. But until Saturday, they lacked insight into specific details about the status of Voyager 1.

“It’s pretty much just the way we left it," Spilker said. "We're still in the initial phases of analyzing all of the channels and looking at their trends. Some of the temperatures went down a little bit with this period of time that's gone on, but we're pretty much seeing everything we had hoped for. And that's always good news.”

Relocating code

Through their investigation, Voyager's ground team discovered a single chip responsible for storing a portion of the FDS memory stopped working, probably due to either a cosmic ray hit or a failure of aging hardware. This affected some of the computer's software code.

"That took out a section of memory," Spilker said. "What they have to do is relocate that code into a different portion of the memory, and then make sure that anything that uses those codes, those subroutines, know to go to the new location of memory, for access and to run it."

Only about 3 percent of the FDS memory was corrupted by the bad chip, so engineers needed to transplant that code into another part of the memory bank. But no single location is large enough to hold the section of code in its entirety, NASA said.

So the Voyager team divided the code into sections for storage in different places in the FDS. This wasn't just a copy-and-paste job. Engineers needed to modify some of the code to make sure it will all work together. "Any references to the location of that code in other parts of the FDS memory needed to be updated as well," NASA said in a statement.

Newer NASA missions have hardware and software simulators on the ground, where engineers can test new procedures to make sure they do no harm when they uplink commands to the real spacecraft. Due to its age, Voyager doesn't have any ground simulators, and much of the mission's original design documentation remains in paper form and hasn't been digitized.

"It was really eyes-only to look at the code," Spilker said. "So we had to triple check. Everybody was looking through and making sure we had all of the links coming together."

This was just the first step in restoring Voyager 1 to full functionality. "We were pretty sure it would work, but until it actually happened, we didn't know 100 percent for sure," Spilker said.

"The reason we didn’t do everything in one step is that there was a very limited amount of memory we could find quickly, so we prioritized one data mode (the engineering data mode), and relocated only the code to restore that mode," said Jeff Mellstrom, a JPL engineer who leads the Voyager 1 "tiger team" tasked with overcoming this problem.

"The next step, to relocate the remaining three actively used science data modes, is essentially the same," Mellstrom said in a written response to Ars. "The main difference is the available memory constraint is now even tighter. We have ideas where we could relocate the code, but we haven’t yet fully assessed the options or made a decision. These are the first steps we will start this week."

It could take "a few weeks" to go through the sections of code responsible for packaging Voyager 1's science data in the FDS, Spilker said.

That will be the key payoff, Spilker said. Voyager 1 and its twin spacecraft, Voyager 2, are the only operating probes flying in the interstellar medium, the diffuse gas between the stars. Their prime missions are long over. Voyager 1 flew by Jupiter and Saturn in 1979 and 1980, then got a gravitational boost toward the outer edge of the Solar System. Voyager 2 took a slower trajectory and encountered Jupiter, Saturn, Uranus, and Neptune.

For the last couple of decades, NASA has devoted Voyager's instruments to studying cosmic rays, the magnetic field, and the plasma environment in interstellar space. They're not taking pictures anymore. Both probes have traveled beyond the heliopause, where the flow of particles emanating from the Sun runs into the interstellar medium.

But any scientific data collected by Voyager 1 since November 14 has been lost. The spacecraft does not have the ability to store science data onboard. Voyager 2 has remained operational during the outage of Voyager 1.

Scientists are eager to get their hands on Voyager 1's science data again. "With the results we got on Saturday, we have new confidence that we can put together the pieces we need to now get back the science data," Spilker said.

"One thing I'm particularly excited about, there's this feature in the Voyager 1 data. We nicknamed it pressure front 2," Spilker said. "Pressure front 2 is a jump in both the density of the plasma around the spacecraft and the magnetic field. It's lasted for three-and-a-half years.

"We'd like to see, is this still there?" she continued. "It's different from what we've seen in the past, and we're trying to figure out, is it some influence coming from the Sun, or is it actually something coming from interstellar space that's creating this feature? So we'd like to see it again, get more data, and be able to study it more carefully.”

Listing image by NASA