2023 INTERACTIVE INSTALLATION

Are You Leaving?

"Are you leaving?" is an interactive installation that explores the topics of language barriers, identity struggles and emotional ties that work through distances. I meant to show the fragility, and emotions that are closely entitled to humanity with machine learning, sensors, visuals and sounds.

As someone born and bred in Beijing and has lived in California and Several European cities, I sometimes struggle with my cultural identity and reconciling multiple language systems.In April of 2023, I returned to my mother's hometown Chongqing (China). It was the farewell moment that I had with my grandmother when I realised the bizarre language barrier between me and her since I couldn't speak her dialect while she couldn't speak Mandarin. At that very moment, I realised that we are so related yet so alienated from each other. She asked me again and again " Are you leaving?", making sure that she understood me correctly, hiding away emotions and making sure I could understand her correctly by just repeating the same sentence. Those words echo powerful and fragile. I recorded that moment and cloned her voice to speak the same things in Mandarin and English, the AI-generated sound gave me creeps and a weird sense of familiarity and comfort since never have I understood my grandma without translation.

Driving back to my grandma's house on mountain roads, not knowing the landscape gave me great challenges. The painful long drive almost works as a metaphor that I am an "outsider" of that town where part of my identity belongs to. I used audio interpolations to generate a sound that gradually changes in between Mandarin, English and the dialect. On the hazy road, I am trying to make sense of and find my way out of the confusing gibberish sound, so that my identity could come out loud and clear.

Are you leaving from Millieon Hu on Vimeo.

Technical Implementation

The technical elements of this piece include machine learning ( voice cloning and text-to-speech), and audio interpolation. I took audio files of my grandmother's voice, which is a dataset of approximately 3 minutes of recording, and then separated them into clips of 10 seconds and selected the ones of the best quality, constituting my training set.

Afterwards, I've used tortoise-tts 1to generate a version of my grandma's voice speaking things in English, however, because of its pretrained voice model, the results turned out not ideal. I then switched to retrieval-based voice conversion2, I recorded my voice saying things in Mandarin and English, and my friend speaking in Chongqing dialect as the original file, then conversed our voice to my grandma's voice after the process of feature extraction and pitch and tone learning.

Afterwards, I did the audio interpolation of the three generated audio files, to create the effect of the audio gradually changing from the dialect, towards Mandarin then English.