How Memorable are TIME’s Top 10 Photos of the Year?

The braniacs at MIT have created an algorithm to determine the “memorability” of a photograph. The deep learning-based technology “learned” what makes a photo memorable by evaluating the rankings from 5,000 human volunteers, and even indicates which portions of an image it considers to be memorable with a heatmap depiction. The algorithm is allegedly as good as a human in determining what makes a memorable photograph.

But ranking a photo based purely on aesthetics isn’t necessarily how humans associate photographs. Context matters. Where was I? What was I doing? What is this photo representative of? I decided to test the algorithm against TIME magazine’s top 10 photos of 2015. Let’s take a look:

1. James Nachtwey for TIME

Memorability: HIGH (0.783)

This is an incredibly moving photo. The grief on the faces of the two women in the foreground is palpable, and the pyramid of bodies is visually intriguing.

2. Goran Tomasevic/Reuters

Memorability: Medium (0.666)

This is a clear example of the importance of context: A man in Nigeria was struck by a motorcycle while celebrating the presidential elections. The photo is devoid of faces, and is difficult to parse without the caption.

3. Damon Winter/The New York Times

Memorability: Medium (0.628)

An eye-tracking study would probably contradict the heatmap. The foreground house is clearly the most important element in the photo, but the algorithm indicates that most of the photo is “memorable.”

4. Nilufer Demir/DHA-AP

Memorability: Medium (0.692)

This photo scored most egregiously. The death of 3-year old Aylan Kurdi sent shockwaves through mass media, and was an inflection point in the Syrian refugee crisis. The algorithm indicates that the adult is the point of memorability. It is obviously wrong.

5. Georgi Licovski/EPA

Memorability: Medium (0.526)

The anguish and distress on the faces of the two children is disturbing, and simultaneously makes the picture memorable. But the algorithm scores the upper third of the photo as the point of interest.

6. Ross McDonnell

Memorability: Low (0.449)

I would surmise that the “distracting” foreground trees confuse the algorithm. And of course, the algorithm doesn’t understand the context of this eastern Ukraine photo and the conflict with Russia that it illustrates.

7. Justin Sullivan/Getty Images

Memorability: Very Low (0.111)

The algorithm seems to be biased against landscapes, so it’s not surprising that this wildfire image scored poorly. But again, the context of this photo (i.e. raging California Wildfires) matters in its memorability.

8. Peter Dejong/AP

Memorability: Medium (0.582)

This slow shutter image during the Paris terrorist attacks is a fairly stunning composition: A father with two crying children amidst the chaos of a terrorist attack. But to the algorithm, it’s of medium memorability.

9. NASA

Memorability: High (0.809)

The highest resolution photo we’ve ever taken of Pluto represents an incredible achievement of science. It is also a damn fine photo, and the algorithm gets it right.

10. Feidin Santana

Memorability: Low (0.485)

This video grab of Walter Scott fleeing from police officer Michael Thomas Slager before he is killed with multiple shots to the back is an unremarkable composition. But contextually it is an amazing photo illustrating police brutality, citizen journalism, and the cultural importance of cameras. This highly memorable image isn’t understood by the algorithm.

***

The algorithm was trained by humans, and it’s arguable that a human without the contextual information would rank the images the same. But we do have the contextual information, and until the algorithm can understand context, it will continue to be “wrong” in assessing the most important images of the year.