How the Hong Kong Baptist University is creating an AI that can sing, compose and dance with Huawei’s support
By Huawei
A university in Hong Kong has launched the world’s first ‘Turing AI Orchestra’ – an AI that creates performance art, with the support of Huawei’s technologies.
Professor Guo Yike, Vice-President (Research and Development) at Hong Kong Baptist University, spoke at Huawei Connect 2022 about the Turing AI Orchestra project. Image: Huawei
This may sound like just another clichéd motivational quote, but the development of AI is now making that a reality. By simply providing three hours of recorded speech data, AI can now transform human speech into any song – all while retaining the individual’s unique voice, tone, and even accent.
One university in Hong Kong is hoping to leverage this capability to bring the arts to the next level. The Hong Kong Baptist University (HKBU) announced the launch of what they termed the ‘Turing AI Orchestra’ (TAIO) back in August, the world’s first AI ensemble.
Beyond singing, the AI ensemble is also able to create other forms of performance arts. It can mimic human movement and dancing patterns to choreograph movement, compose music with the sounds of different instruments, and even create animation simply with text prompters.
Perhaps most notable is the AI’s ability to create art otherwise impossible for humans, says Professor Guo Yike, Vice-President (Research and Development) of HKBU. “When a machine creates things, it’s not restricted to humans,” he adds.
The AI can, for instance, choreograph virtual avatars without a human form, such as jellyfish or trees. Another possibility is to create voices that can sing notes outside the vocal range of humans, Guo explains.
New frontiers of performance art
The TAIO project is part of a larger drive towards art technology research in China. The university received about HK$53 million (US$6.77 million) from Hong Kong’s Research Grants Council in 2021 to further this research.
“The university’s focus is to create new art forms through AI,” Guo says. “It's quite different from how traditional AI works, where you largely build a machine to mimic human creations. We want to liberate ourselves from this kind of thinking, for machines to freely create artwork we can appreciate.”
HKBU hopes to do so through a performance-oriented approach. This means first devising a vision of what they hope to perform on stage, before pursuing the research needed to make it happen. For instance, HKBU put up a concert dubbed ‘A Lovers’ Reunion’ at its annual gala concert for the HKBU Symphony Orchestra.
In another performance during the concert, virtual avatars took to the stage to perform an intricate choreography inspired by the movements of a newly-discovered species of box jellyfish in Hong Kong.
In preparation for this concert, the university had to bring together a team of computer scientists to train the 320 voice models that made up the AI choir.
Additionally, professional dancers from the Hong Kong Dance Company helped to interpret and label underlying emotional and aesthetic values between the music and dance, allowing the AI programme to better choreograph movements that evoked the required emotions and visuals.
A human-machine symphony
When asked if machines will one day render human artists irrelevant, Guo’s answer was immediate: “Never”.
Rather than replacing human artists, the goal of using AI is to push the boundaries of artistic creation by humans, he says.
“All artwork is human and machine co-creation. Machines will never be able to do this alone,” he elaborates. Rather, he believes that technology can actually help more individuals become artists by lowering the barriers to entry.
Guo believes that the most important thing in integrating AI with the arts is the need to ensure that the two can communicate. This will result in a scenario where machines can understand what is appreciated as beauty by humans, while humans can understand and appreciate the presentation done by machines, he explains.
It is only when this human-machine interaction happens that the two can truly work together to co-create art. Then, audiences can expect to see feats like virtual avatars dancing in sync with humans, or machines composing music that will accompany a human choir.
“Today’s AI machines and humans are separate. That is not the right way,” Guo says. In fact, Guo even hopes to train AI to be able to convey emotions through the arts like humans can. AI today can sing quite well, but lacks emotions, according to Guo. “When people sing, it’s not just about following the melody. It’s about expressing feelings,” he says.
To address this, HKBU is trying to train the programme to recognise how different sounds and melodies evoke different emotions. They are doing so by having music experts or singers label the emotions attached to certain sounds. The AI programme can then be prompted to make music which evokes this specific emotion, and humans can provide feedback where necessary.
“We treat the machine like a student,” Guo explains. By training the programme over time, the hope is that one day, AI will be able to master the expression of human emotions through the arts.
Supported by Huawei
Given the vast amount of data the TAIO requires for voice modelling, HKBU works closely with Chinese tech giant Huawei to do the work that they do. The AI orchestra is supported by Huawei’s Ascend AI computing platform.
This platform integrates Huawei’s Atlas 800 Deep Learning System, an AI simulator with cutting-edge computing capabilities. It is more energy-efficient than the industry standard, and can reduce lag time by approximately 30 per cent. This can help to streamline the AI training process and enable the creation of art through its massive database and algorithm training.
Guo also highlighted Huawei’s Pangu Model as a great help to his work, stating that the model’s massive database greatly helped in the training of their AI algorithms. The model is able to derive information from text and pictures efficiently and accurately, and can generalise the information across other mediums. This has helped the TAIO in its art generation, such as by creating visuals from lyrics.
Artificial intelligence technologies are changing rapidly, and AI-powered industry solutions, from physical computing to the humanities and arts, are growing by the day. “Only efficient, reliable, and stable AI computing power can support all these applications and create unlimited future possibilities,” said a Huawei spokesperson. "By always providing better and more stable products, Huawei continues to accelerate AI on the road to promote its popularity and application, ultimately empowering the industry and promoting industrial upgrading.”