Ambient light sensors can reveal your device activity. How big a threat is it?

An overwhelming majority of handheld devices these days have ambient light sensors built into them. A large percentage of TVs and monitors do, too, and that proportion is growing. The sensors allow devices to automatically adjust the screen brightness based on how light or dark the surroundings are. That, in turn, reduces eye strain and improves power consumption.

New research reveals that embedded ambient light sensors can, under certain conditions, allow website operators, app makers, and others to pry into user actions that until now have been presumed to be private. A proof-of-concept attack coming out of the research, for instance, can determine what touch gestures a user is performing on the screen. Gestures including one-finger slides, two-finger scrolls, three-finger pinches, four-finger swipes, and five-finger rotates can all be determined. As screen resolutions and sensors improve, the attack is likely to get better.

Always-on sensors, no permissions required

There are plenty of limitations that prevent the attack as it exists now from being practical or posing an immediate threat. The biggest restrictions: It works only on devices with a large screen, in environments without bright ambient light, and when the screen is displaying certain types of content that are known to the attacker. The technique also can’t reveal the identity of people in front of the screen. The researchers, from Massachusetts Institute of Technology, readily acknowledge these constraints but say it’s important for device makers and end users to be aware of the potential threat going forward.

“We aim to raise the public awareness and suggest that simple software steps can be made to make ambient light sensors safer, that is restricting the permission and information rate of ambient light sensors,” Yang Liu, a fifth-year PhD student and the lead author of the study, wrote in an email. “Additionally, we want to warn people of the potential privacy/security risk of the combination of passive (sensor) and active (screen) components of modern smart devices, as they are getting ‘smarter’ with more sensors. The trend of consumer electronics pursuing larger and brighter screens can also impact the landscape by pushing the imaging privacy threat towards the warning zone.”

There’s a large body of existing attacks that use sensors on phones and other devices as a side channel that can leak private details about the people using them. An attack devised by researchers in 2013, for instance, used the embedded video camera and microphone of a phone to accurately guess PINs entered. Research from 2019 showed how monitoring a device accelerometer and gyroscope output can also lead to the accurate guessing of PINS entered. Research from 2015 used accelerometers to detect speech activity and correlate it with mood. And an attack presented in 2020 shows how accelerometers can recognize speech and reconstruct the corresponding audio signals.

Exacerbating the potential risk: This sensor data is always on, and neither Android nor iOS limit the permissions required to access it. End users are left with few, if any, effective recourses.

The MIT researchers add to this existing corpus with an eavesdropping technique that can capture rough images of objects or events taking place directly in front of the device screen. The device used in the experiments was a Samsung Galaxy View2, a tablet that runs on Android. The researchers chose it because of its large (17.3-inch) screen. Under current conditions, large screens are necessary for the attack to work because they provide the large amount of brightness needed. The Galaxy View2 also provided easy access to the light sensor. MIT researcher Liu said iOS devices and light sensor-embedded TVs from a host of manufacturers are also likely vulnerable.

Turning the sensor into a camera

The springboard for the attack is the target viewing a series of known sequences, such as a video, that the attacker has already analyzed and knows is being displayed at the time the attack is performed. By knowing the intensity of the light being emitted from the screen at a given time and correlating it to the ambient light measured at that point, the researchers turn the sensor into a crude form of camera that can capture the shape of objects touching, or possibly very close to, the screen.

The other three co-authors of the research are Gregory W. Wornell, William T. Freeman, and Frédo Durand. In much more technical terms, the researchers described the technique this way:

We argue that the ambient light sensor can enable imaging if one uses the screen as a controllable active source of illumination displaying a known video sequence. The ambient light sensor measures the corresponding intensity variation of light reflected off or blocked by the scene. These sequential measurements and the corresponding known illumination sequence form a linear inverse problem, which can be solved to reconstruct an image from the perspective of the screen. Here, the problem is linear only when the measurements are not quantized with full precision of floating-point numbers. However, the ambient light sensor is of low sensitivity (at 1 lux level), and the contribution of screen fluctuation is heavily quantized to ≤5 bits per measurement. This type of imaging inverse problem is known as ghost imaging or single-pixel imaging, which was considered to be a quantum effect, was independently explored as dual photography, and can be accelerated by compressive sensing. A similar idea using arrays of light-emitting diodes as virtual sensors has also been explored in the internet of things community for skeleton posture and hand pose estimation. However, it has not been shown in any privacy settings. The imaging capability that we explore is a form of dual photography, where Helmholtz reciprocity indicates that the flow of light can be computationally reversed to swap out the sensor (ambient light sensor) and the illumination (screen). In our case, the light path from the screen to the scene and then to the ambient light sensor (primal path) can be reversed, resulting in a path from the ambient light sensor to the scene and eventually to the screen (dual path). The primal configuration, where the sensor has a single pixel and the illumination has good resolution, can be interpreted by its duality, where the pixelated screen works as a virtual sensor and the ambient light sensor as a virtual light source.

The crudeness of the objects that can be detected by the attack is illustrated in the following images. Images C and D show how an open hand and a pointing index finger appear to an attacker.

The image below further illustrates the crudeness, but it also illustrates that the roughness is nonetheless sufficient for an attacker to identify touch gestures used to control or interact with devices.

In the event the attack one day meets the threshold for posing a real threat, there are several things that can mitigate the risk. One is for device manufacturers and OS or app developers to tighten restrictions on sensor output, possibly through the current permission system both iOS and Android employ. Another possible mitigation is for one or more of these parties to reduce the precision and speed of the sensor. A third option is to change the location of the sensor from the front of the device to elsewhere. For end users, an effective defense is to perform sensitive operations in a well-lit environment or to turn off sensor-based light adjustments in device settings.

There’s no need for end users or device or app builders to take any of these precautions now. That said, attacks only improve over time. It’s possible these measures may make good sense in the future.