Are you aware of the fact that some of us can hear colors? Synesthesia, which is a neurological condition causing the stimulation of one sense to produce experiences in a different one, affects around 4% of the population. Thanks to the work of scientists from the Lodz University of Technology, soon there will be a way for the rest of us to experience a similar sensation too.
The “Hearing Art” project is an innovative system which enables its users to read digital images through the use of sound. Using motion sensor technology to map out the placement of each color in the chosen image, the app generates a sound suitable for each color. In order to do so, the system analyses the hue, saturation, and value of a color and uses its placement on the color wheel to emit an appropriate sound.
While the system could serve as an additional auditory sensation for just about any art lover, the researchers point towards the revolutionary potential of the technology for visually impaired people. Once implemented, the innovation could make art galleries much more accessible by enabling blind people to experience visual art in a completely different way. The project’s developers admit that the technology would not work for people who were completely blind from birth – it is however suitable for people whose visual impairment is partial, or acquired later in life.
So how does it work in practice? “The user aims at a specific point of a painting. As they start moving their arm in its vicinity, they begin hearing a melody, as each color, and each pixel of the image is assigned a different tone” – explains one of the system’s creators, Damian Jóźwiak.
While the project takes advantage of the fact that visually impaired people often experience a heightened sense of hearing, and are thus more sensitive to sound stimuli, the technology has been adapted for sighted participants as well. The motion sensor is marked with a red dot on the system’s display, which shows its placement on the original painting. By tracking the placement of the sensor on the image, users are able to associate each color with a sound.
As for its accuracy, preliminary tests conducted in the Lodz chapter of the Polish Association of the Blind have revealed an 80% success rate in “reading” colors with the use of the technology. The biggest shock came when one of the participants has attempted calling one of their visually impaired friends – the recipient was able to recognize 100% of the sounds and identify each color assigned to it – adds Jóźwiak.
So what’s in store for the project in the near future? The creators of the “Hearing Art” see its future in museums, where it could be experience by visually impaired and sighted patrons alike. If you feel like hearing colors could be an interesting experience, feel free to plan a trip to the Museum of Lodz in the near future, as the application will find its first home there.
