How Does Google Night Sight Work?

Today we're taking a peek with Google at their Night Sight feature for the Google Camera app on Pixel phones. The basic concept begins with keeping the Pixel's camera's shutter open longer than normal – allowing more light in on the camera's image sensor. With more light comes more information with which the camera can create a less dark image. That's where Google started – they then brought in the Machine Learning to do some magic.

Night Sight is almost like – and might be considered – an extension of what Google introduced in the year 2014 with the first Google Pixel smartphone. That device brought on Google's HDR+, a system with which dynamic range is improved to a great degree by capturing a "burst of frames." Capturing several photos happens with every HDR system, but with Google's HDR+, some machine learning magic takes place, frames are aligned, and computational photography makes the end product look very, very good in most cases.

Google Product Manager for Computational Photography, Alexander Schiffhauer, provided some insight this morning on all the information Night Sight takes in as the Pixel's camera captures a photo.

"Before you press the shutter button, Night Sight measures your natural hand shake, as well as how much motion is in the scene. If Pixel is stable and the scene is still, Night Sight will spend more time capturing light to minimize noise," said Schiffhauer. "If Pixel is moving or there's significant scene motion, Night Sight will use shorter exposures, capturing less light to minimize motion blur."

Google suggested this week that Night Sight is made to capture scenes with which even HDR+ doesn't do well. In a feature from Marc Levoy, Distinguished Engineer and Yael Pritch, Staff Software Engineer on the Google AI blog, the exact wording is as follows: "With Night Sight, our goal was to improve picture-taking in the regime between 3 lux and 0.3 lux, using a smartphone, a single shutter press, and no LED flash."

Above you'll see a chart provided by Google which identifies a range of lux. Lux is the amount of light hitting a surface, light per unit area, lux. Lux is measured in lumens per meter squared. According to Google, Night Sight is for light amounts where you either wouldn't walk through the house without a flashlight or you'd be unable to find your keys on the floor – that's dark!

When you press the shutter button in the Google Camera app in a Pixel, it captures the most recent 9 or 15 frames to send to its HDR+ (or Super Res Zoom, but that's not important right now) software. Per Google, "As soon as you open the camera app, it begins capturing image frames and storing them in a circular buffer that constantly erases old frames to make room for new ones."

Incidentally, this is a great way to make your Pixel's battery drain faster than in almost any other on-screen situation. Just keep the camera open!

Google suggests that with this system, they provide "zero-shutter-lag." With the Night Sight feature, you do not use the most recent frames, you only start to capture a scene when the shutter button is pressed. With Night Sight, you're using what Google calls positive-shutter-lag (PSL).

Here's what Night Sight does:

1. Captures 6 to 15 frames

1b. Captures 15 frames if device detects it's handheld, 1/15 of a second (or less) each

1c. Captures 6 frames if sitting perfectly still, at 1 second each

"Or anything in between" Dependent on the detected situation.

2. Aligns multiple frames

3. Merges multiple frames

Does both of these things with HDR+'s merging algorithm on Pixel 1 and 2 or using Super Res Zoom's merging algorithm on the Pixel 3.*

4. Uses Fast Fourier Color Constancy algorithm to determine most accurate colors

5. Determine ideal light-to-dark tone mapping to make sure the photo doesn't look like it was captured with full-on daytime lights

6. Present final photo with per-frame exposure time attached

* Because of the Google Pixel 3's faster processor, it's able to produce better results with Night Sight. This happens because it uses Super Res Zoom's algorithm instead of HDR+'s algorithm. Super Res Zoom requires the faster processor of the Google Pixel 3 – at the moment.

This feature is thus far relegated to Pixel smartphones, but might (unofficially) spread to others in the future. If you own a Google Pixel smartphone, of any of the first three generations, you can head over to Google Play and check if you need an update to the Google Camera app. If you don't need an update, either the update isn't ready yet, or you already have it.

If you already have the update, Google Night Sight will be available in your Google Camera app – your standard camera app on your Google Pixel smartphone. If you've never used Night Sight before, just open a dark closet and point your Pixel's camera lens inside the darkness. Night Sight should appear as a button just above your other camera modes. Otherwise you'll find it by sliding on your camera modes to the left.

If you do not have a Pixel: First, be aware there is no official Google Night Sight feature for non-Pixel smartphones. Then, head over to our early Night Sight download info article to learn of a place or two were some unofficial builds may be found. And be sure to do so of your own free will – all downloads are your own responsibility!