Aina Ono, Seiya Aoki, Yusuke Yamada, Santa Naruse, Reo Anzai: Holarchy

Summary

Holarchy is an online installation that eliminates the arbitrariness of the artist, creating a new audio-visual work by comparing in the same space the feature of the music and the feature of a number of images selected by the artist beforehand and selecting them automatically.

Abstract

This work is an online installation that creates a new audio-visual using automatic video selection with deep learning. Video expression in the audio-visual and DJ+VJ, where sound and images coexist together, has been based on sampling methods by combining clips that already exist, generative methods that are computed in real time by computer, and the use of the sound of the phenomenon and the situation itself.Its visual effects have extended music and given it new meanings. However, in all of these methods, the selection of the video and the program itself was premised on the artist’s arbitrary decision to match the music. This work is an online installation that eliminates the arbitrariness of the artist, creating a new audio-visual work by comparing in the same space the feature of the music and the feature of a number of images selected by the artist beforehand and selecting them automatically. In this work, the sounds of the youtube video selected by the viewer are separated every few seconds, and the closest video is selected by comparing these features with the features of countless short clips of movies and videos prepared in advance in the same space. This video selection method reconstructs the mapping relationship that artists have constructed so far between video and sound using deep learning, and suggests the possibility of possible correspondences. In addition, unconnected scenes from different films and images that have never been connected before become a single image, and emerge as a whole, and the viewer finds a story in the relationship between them. With this work, audio-visual and DJ+VJ expression is freed from the arbitrary decision, and a new perspective is given to the artists.

The creative acts of artworks, their production and performance have been limited by the intentions of the creators of the artworks and the production environment (software). Particularly in interactive art, despite the name “interactive,” the experience is only possible within the rules set by the artist, and is unidirectional and passive. The author utilizes deep learning to create the rules themselves to address these issues, seeking ways to deviate from the rules and algorithmic mappings created by the creators, and to allow the audience and users of the software to become more active and creative.

The distinctive feature of my work is that I have constructed a system that allows the viewer to spontaneously experience creativity from within. Generally, it can be said that the act of matching images to sound is an act of creativity solely on the part of the artist. However, as mentioned above, arbitrary decisions about the way we experience things are planned and not interactive. Therefore, I decided to create a machine learning audio-visual online installation to avoid the author’s construction of the rules of experience. In this work, a multimodal machine learning model is used to search for features of sound and video in the same space, and short clips of video that are similar to the features of the sound separated by a few seconds are created.

This work was created as an online installation, influenced in part by COVID-19. The difference between an experience on a website being a mere demonstration and being viewed as a work of art is largely dependent on the look and feel of the website and the design of the experience. On the other hand, if you create a work on a website that is difficult to understand, there is no artist or curator to answer your questions, unlike with onsite work. In that sense, I took care of the design so that it would be easy to understand and not look like a demonstration.

Inspiration Behind the Project

I have been working as an artist, creating visual performances using real-time rendered graphics and installations. In this context, while some forms of computerized art and performance may be impossible to create in other mediums, it can be said that even digital expression is not limitless as creations must fit into the framework of the development environment and software developers. As an approach to transcend these limitations in visual performance, I play the image like an acoustic instrument. By treating the image as physical light, refraction, reflection and projection, freeing expression from pixels and frame rate. In my laser installations, daily necessities are used to create a visual experience in which physical light is reflected and refracted onto the space, reminding the viewer of nostalgia and allowing them to spontaneously form their own unique experience. 

Apart from such an approach, holarchy is a piece created within the audiovisual framework I have been working within, which addresses the question of whether creators and experiences can ever completely escape these limitations and become fully creative in post-computational works.

Key Takeaways for the Audience

This work uses AI to select and reconfigure images in response to sound, which has been done in the past as an aesthetic and creative act. Through the experience of that choice, we can feel the difference between the two, and we can think about how the correspondence we humans have been thinking about is a matter of subjective human-centered senses. In this way, I want viewers to feel the possibility of extending interaction through audio-visuals.