This Is An Umbrella (055) identifies objects using Yolo, a neural network, and removes them from their context. The images are then situated in a sequence, acquiring new context and meaning.
Sequencing underscores the accuracy of the model, and the discrepancy between human and algorithmic-statistical identification, prompting the inevitable question: is this “person” really a person?
Sequencing also establishes a comparison of similarly-categorized images, where one “person” is evaluated against the next “person”. Does comparison aid or frustrate identifications made by the viewer?
Some of the images were taken from the same frame, some were taken from different frames. As a result, certain elements periodically re-appear, but not consistently. The ticket booth might be identified as a truck, but then might not be identified at all. An umbrella is always identified as an umbrella, but it isn’t always identified.
Finally, objects aren’t fully removed from their context. Instead, they share a bounding box with the immediate surrounding. Within that box, the model is confident that there is a person or a truck, but hasn’t identified the silhouette of that object. Interestingly, the identified region is rectangular, echoing the underlying order of pixels from which objects are identified.
Technicals
The objects are extracted from a series of stills taken from a live video feed of Times Square, discussed in previous posts for Routine Grid (034) and Routine Difference (035).
A previous post discuss the Yolo model. By modifying the source code, the coordinates of each bounding box were saved to a JSON file. The original images are then cropped and looped through using p5.js and the get() function on the original images.
Next Steps
- Should all images be the same height, despite needing to upsample the pixels?
- Explore all images of the same class being shown in rows