NASA Engineers Are Racing to Fix Voyager 1

A computer glitch has put the future of humanity’s farthest-flung space probe in doubt.

Voyager 1 is still alive out there, barreling into the cosmos more than 15 billion miles away. However, a computer problem has kept the mission's loyal support team in Southern California from knowing much more about the status of one of NASA's longest-lived spacecraft.

The computer glitch cropped up on November 14, and it affected Voyager 1's ability to send back telemetry data, such as measurements from the craft's science instruments or basic engineering information about how the probe was doing. As a result, the team has no insight into key parameters regarding the craft's propulsion, power, or control systems.

"It would be the biggest miracle if we get it back. We certainly haven't given up," said Suzanne Dodd, Voyager project manager at NASA's Jet Propulsion Laboratory, in an interview with Ars. "There are other things we can try. But this is, by far, the most serious since I’ve been project manager."

Dodd became the project manager for NASA's Voyager mission in 2010, overseeing a small cadre of engineers responsible for humanity's exploration into interstellar space. Voyager 1 is the most distant spacecraft ever, speeding away from the sun at 38,000 mph (17 kilometers per second).

Voyager 2, which launched 16 days before Voyager 1 in 1977, isn't quite as far away. It took a more leisurely route through the solar system, flying past Jupiter, Saturn, Uranus, and Neptune, while Voyager 1 picked up speed during an encounter with Saturn to overtake its sister spacecraft.

For the past couple of decades, NASA has devoted Voyager's instruments to studying cosmic rays, the magnetic field, and the plasma environment in interstellar space. They're not taking pictures anymore. Both probes have traveled beyond the heliopause, where the flow of particles emanating from the sun runs into the interstellar medium.

There are no other operational spacecraft currently exploring interstellar space. NASA's New Horizons probe, which flew past Pluto in 2015, is on track to reach interstellar space in the 2040s.

State-of-the-Art 50 Years Ago

The latest problem with Voyager 1 lies in the probe's Flight Data Subsystem (FDS), one of three computers on the spacecraft working alongside a command-and-control central computer and another device overseeing attitude control and pointing.

The FDS is responsible for collecting science and engineering data from the spacecraft's network of sensors and then combining the information into a single data package in binary code—a series of 1s and 0s. A separate component called the Telemetry Modulation Unit actually sends the data package back to Earth through Voyager's 12-foot (3.7-meter) dish antenna.

In November, the data packages transmitted by Voyager 1 manifested a repeating pattern of 1s and 0s as if it were stuck, according to NASA. Dodd said engineers at JPL have spent the better part of three months trying to diagnose the cause of the problem. She said the engineering team is "99.9 percent sure" the problem originated in the FDS, which appears to be having trouble "frame syncing" data.

So far, the ground team believes the most likely explanation for the problem is a bit of corrupted memory in the FDS. However, because of the computer hangup, engineers lack detailed data from Voyager 1 that might lead them to the root of the issue. "It's likely somewhere in the FDS memory," Dodd said. "A bit got flipped or corrupted. But without the telemetry, we can't see where that FDS memory corruption is."

When it was developed five decades ago, Voyager's Flight Data Subsystem was an innovation in computing. It was the first computer on a spacecraft to make use of volatile memory. Each Voyager spacecraft launched with two FDS computers, but Voyager 1's backup FDS failed in 1981, according to Dodd.

The only signal Voyager 1's Earthbound engineers have received since November is a carrier tone, which basically tells the team the spacecraft is still alive. There's no indication of any other major problems. Changes in the carrier signal's modulation indicate Voyager 1 is receiving commands uplinked from Earth.

"Unfortunately, we haven't cracked the nut yet, or solved the problem, or gotten any telemetry back," Dodd said.

Breaking Out the Binders

In the next few weeks, Voyager's ground team plans to transmit commands for Voyager 1 to try to isolate where the suspected corrupted memory lies within the FDS computer. One of the ideas involves switching the computer to operate in different modes, such as the operating parameters the FDS used when Voyager 1 was flying by Jupiter and Saturn in 1979 and 1980. The hope among Voyager engineers is that the transition to different data modes might reveal what part of the FDS memory needs a correction.

This is a lot more complicated than it might seem on the surface. For one thing, the data modes engineers might command Voyager 1 into haven't been used for 40 years or more. Nobody has thought about doing this with Voyager's flight data computer for decades.

Voyager 1 and 2 have an outsized public profile compared to the resources NASA commits to keeping the spacecraft going. Fewer than a dozen people typically work on the Voyager mission. This number has slightly increased since the computer problem appeared in November, with a small "tiger team" of around eight experts in flight data systems, software, and spacecraft communications assigned to help troubleshoot the glitch.

"Not to be morose, but a lot of Voyager people are dead," Dodd said. "So the people that built the spacecraft are not alive anymore. We do have a reasonably good set of documentation, but a lot of it is in paper, so you do this archaeology dig to get documents."

Imagine rummaging through a user's manual for an antique car. The book's weathered pages are probably fraying. That's not unlike what Voyager engineers, some of whom weren't alive when the mission launched, are experiencing now.

"We have sheets and sheets of schematics that are paper, that are all yellowed on the corners, and all signed in 1974," Dodd said. "They’re pinned up on the walls, and people are looking at them. That's a whole story in itself, just how to get to the information you need to be able to talk about the commanding decisions or what the problem might be."

This is a familiar task for Voyager engineers. In the last few years, the mission's core team at JPL has consulted archived documents to troubleshoot other, less serious computer problems and develop a new way to operate thrusters on both spacecraft to stave off the accumulation of residue in fuel lines.

While spacecraft engineers love redundancy, they no longer have the luxury of backups on the Voyagers. That means, in any particular section of the spacecraft, a failure of a single part could bring the mission to a halt. Both spacecraft run off nuclear batteries, which produce a little less electricity each year as their plutonium power sources decay. Toward the end of the 2020s, the declining power will force NASA to start turning off instruments on each spacecraft.

Most of NASA's modern missions exploring the solar system have simulators on the ground to test commands and procedures before sending them to the real spacecraft. This practice can reveal commanding errors that could put a mission at risk.

“It is difficult to command Voyager," Dodd said. "We don't have any type of simulator for this. We don't have any hardware simulator. We don't have any software simulator. There's no simulator with the FDS, no hardware where we can try it on the ground first before we send it. So that makes people more cautious, and it's a balance between getting commanding right and taking risks."

Managers are also aware of Voyager 1's age. It's operating on borrowed time. "So we don't want to spend forever deciding what we want to do," Dodd said. "Something else might fail. The thrusters might fail. We want to do the right thing, but we can't hem and haw over what the right thing is. We need to look at things methodically and logically, make a decision, and go for it."

When it comes time to send up more commands to try to save Voyager 1, operators at JPL will have to wait more than 45 hours to get a response. The spacecraft's vast distance and position in the southern sky require NASA to use the largest 230-foot (70-meter) antenna at a Deep Space Network tracking site in Australia, one of the network's most in-demand antennas.

"The data rates are very low, and this anomaly causes us not to have any telemetry," Dodd said. "We're kind of shooting in the blind a little bit because we don't know what the status of the spacecraft is completely."

Source