is the knowledge-sharing platform for the 

. Explore the experiments and bold ideas behind the Academy's Fellowship programme, delve into the science behind the projects of our HIDA researchers, and explore the results of international collaborations.

is a platform for discovery and documentation. Each 

showcases a unique artistic-technological research project, with topics as diverse as the use of AI in theatrical design, choreography for and with robotics, and the possibilities for theatre in virtual spaces. Each 

 offers the chance to learn more about the members of our artistic research network. By pooling our knowledge in such a way, creative research into theatre, performance and digital technologies can become more accessible, sustainable, and artistically innovative.

Funded by the Federal Cultural Foundation, the Academy will award a total of 54 fellowships between 2019 and 2022, each lasting for five months. The 

 are designed for people interested in artistic-technical research which explores the possibilities for digital narratives in theatrical spaces, as well as their practical implementation. Fellows have researched a variety of digital methods (motion capture, coding, XR, VR, AR, sensor technologies, robotics, and more) across artistic fields including opera, dance, orchestra and drama. 

Helmholtz Information & Data Science Academy

(HIDA), the Academy for Theatre and Digitality has offered joint research fellowships at the interface of digital art and cutting edge scientific research since 2021. The HIDA fellowships are kindly supported by the 

Artificial Intelligence

Installation

VR (Virtual Reality)

Puppetry

Interactive

Stage Design

Dance

Sound Design

Theater Dortmund
Theater Karree 1-3
44137 Dortmund
Germany
Fon: +49 (0) 231 226 14330

Address and location of the Academy for Theater and Digitality:
Speicherstrasse 17
44147 Dortmund
Germany

E-Mail: akademie@theater.digital
Web: theater.digital

Tax number: 314/5702/9367
Sales tax identification number: DE 124643894

Management Team:

Executive Director
Tobias Ehinger

Administrative Director
Martin Lizan

Director of the Opera
Heribert Germeshausen

Director of the Ballet
Xin Peng Wang

General Music Director
Gabriel Feltz

Artistic Director of the Schauspiel
Julia Wissert

Director of the Children's and Youth Theater
Andreas Gruhn

Director of the Academy for Theatre and Digitality
Marcus Lobbes

Please also note our privacy policy.

 as the sixth division of Theater Dortmund (hereinafter referred to as "Theater", information about Theater Dortmund can be found 

), is pleased that you are visiting our website. Data protection and data security when using our website are very important to us. We would therefore like to inform you at this point, in accordance with Article 13 of the EU General Data Protection Regulation (DSGVO) and Section 13 of the German Telemedia Act (TMG), which of your personal data we collect when you visit our website and for what purposes it is used. Since changes in the law or changes in our internal company processes may make it necessary to adapt this data protection declaration, we ask you to read through this data protection declaration regularly. The data protection declaration can be accessed, saved and printed out here at any time.


§ 1 Scope of application

This data protection declaration applies to the Internet offering of Theater Dortmund, which can be accessed under the domain www.theaterdo.de and the various subdomains (hereinafter referred to as "our website").


§ 2 What is personal data?

Personal data is individual information about personal or factual circumstances of a specific or identifiable natural person (data subject). This includes, for example, information such as your name, address, telephone number, date of birth or e-mail address. Information for which we cannot establish a reference to your person, e.g. by anonymizing the information, is not personal data.


§ 3 What personal data do we collect/process and use?

Access data without personal reference is stored, such as the name of your Internet service provider, the page from which you visit us, the names of the requested files and their retrieval date. These data are evaluated exclusively for the improvement of our offer and do not allow any conclusions to be drawn about your person. 

If you wish to make use of services offered by us on our website, such as purchasing tickets via the ticket store or ordering a newsletter, etc., it is necessary for you to provide further data. This is the data that is required for the respective processing, such as your payment information when purchasing tickets or your e-mail address when ordering a newsletter. 

We process the data we collect about you for the provision of the services we offer, to answer your questions and to operate and improve our website and applications. 

We process your personal data exclusively to provide you with a comprehensive service via our website or, if you use our contact form or, to process your ticket purchase or, to process your application or, to provide you with requested information about our offers.

Fellowship

HIDA

discover cutting-edge digital research in theatre and performance 
meet our research fellows and collaborators in the 

Portal: Academy for Theatre and Digitality

Humane Methods (2019-present) is a project consisting of happenings and stage productions where dancetheatre, bio-art, interactive music, adaptive lighting, and AI technology integrate into living ecosystems. Each work in the series acts as a branch of a rhizome: each dives into different facets of the same theme and world. The works created so far – 

 (2020) and the forthcoming ΣXHALE (2021-) – combine in different ways performing arts with technology and the human with the non-human to reveal algorithmic violence through sensory, physical and ahuman strategies. It is an aesthetic research into how a synthesis of symbols, movement, music and AI can enable posthuman forms of empathy.

℧R is a solo performance for a body, AI algorithms, lights and the vibrational force of sound. 

Embodied in Donnarumma’s musical instrument XTH Sense, an interactive, machine learning algorithm transforms the inner sound of muscles, blood, and bones into a real-time choreography of movement, sounds and light. Drawing on Donnarumma’s more recent movement research, the piece combines multi-sensorial stimulation with human-computer interaction and somatic practices to test the seeming impossibilities of bodily articulation and repetition. 

℧R is a walk in the dark. A thin body walks while a dim light shines. A path covered in soil is between them. Gazeless, the figure walks and prays, walks and prays, walks and, entranced, pries into the darkness. Its red cloth sanctifies it. Its missing face alienates it. The walk is a loop, a prayer, an aggression, an annihilation and a transformation. An AI computer vision system watches over the body, attempting to distinguish the body from the ‘noise’ surrounding it. The AI wants to represents what it sees, but the system works against itself in a paradoxical loop; the more the algorithm learns to see, the brighter the light becomes. In the eyes of the algorithm, the body is eaten by the storm of light.

The latest chapter in the Humane Methods series is ΣXHALE (2021-). A poetic reflection on ethics in algorithmic societies, ΣXHALE is a procedural and participatory piece lasting several days.

It develops in loops, performed simultaneously by 6 performers* and an AI computer-vision system. As it watches the performers repeat a proto-prayer ritual, the AI learns their movements in the space and then begins searching for anomalies, something that deviates from what it expects to see. The learning process is accurately transformed in real time in music and lighting, influencing thus the events in the form of audiovisual stimuli. With each repetition, each variation and each anomaly, the loops mutate into stories of violence and empathy and the audience is asked to choose how they want to participate in these stories. The interaction between audience, performers and AI gives life to an endless sequence of micro-narratives.

AI in theatre and performance: Strategy & Questions

Technical Infrastructure

The Autoencoder

A Performing Machine

How To: Technological Infrastructure

Andrea Familari

Marco Donnarumma

Humane Methods (2019-present) is an ongoing project by Fronte Vacuo (Marco Donnarumma, Margherita Pevere, Andrea Familari) that reflects on the new forms of violence emerging in algorithmic societies. The technological research for ℧R and ΣXHALE have been conducted at the Academy for Theater and Digitality and this page describes our research questions, workflow and findings in relation to these projects. Please note that this page does not discuss the theatre pieces we created in their entirety, but focuses only on the technological research that underpins those pieces. For further information about the aesthetic, conceptual and dramaturgical research behind these works please visit our website.

Humane Methods

Performance

This is a sample description for artificial intelligence. 

AV (Audiovisual Design)

Lighting Design

2020

How to depict violence in theatre?           

Without reproducing violence?
Without aestheticizing it?
Without trivializing it?
Without inflicting violence on others while trying to find answers?

>>>>>>> Thoughts and insights from this search <<<<<<<


In the beginning of 2023 there is this vague vision for a setup which gives me the freedom to experiment with different aspects of the depiction of violence:

a human-sized puppet | its digital face-double | a green screen studio

___________________________________________________________________________________________________


I knew right from the start that I want to work with a puppet.              Firstly, because I think that this is a logical approach to the question of how to depict violence in theatre without inflicting violence on others while trying to find answers. The puppet itself feels nothing. And since I am going to animate the puppet myself, I am autonomous - and bypass putting others in situations of violence.
But above all, I choose the puppet because I am obsessed with the effect it has on the viewer.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

RELATIONSHIP BETWEEN SPECTATOR AND PUPPET

THE SPECTATOR`S (LIFE-BRINGING) GAZE
Although the puppet is obviously artificial (manufactured & dead), the subconscious draws a line from the viewer's body to the puppet's body and brings it to life. 

The uncanniness inherent in the puppet is partly due to its resemblance to humans.

THE PUPPET AS 'DESTROYER' OF THE BODY ORDER
This process creates a kind of feedback: we are thrown back on ourselves when looking at the puppet - the sight of it destroys the illusion of the perfect, complete body.

>>> The act itself is already a violent one.

>>> The phenomenon of the puppet lies in the bodily consternation it triggers in us.

What is the puppet's digital face-double?

I am driven by the idea to combine two components - analog theater work and digital innovation - to create a character that serves as a test subject for me in an attempt to research violence.
I asked myself how I could enhance the strange liveliness of the puppet.
Then I had the idea: what if the puppet can speak for itself?
What if I 3D-scanned the puppet's face and then animated it, let it talk?

The original goal at the time was to merge these two components - and create a hybrid character.
An analogue puppet with a digital face - united in one body within a video work.


By working in the green screen studio, I can put the puppet into different local contexts which gives me a endless freedom.
I´m wearing a green morphsuit so I can erase myself from the material.
The puppet moves as if by itself.

In the end, I want to have a character which is half analog and half digital puppet. It moves and speaks by itself, acts in different "worlds" - which consist of videos, images, different sounds and noises that I collect.
_____________________________________________________________________________________________________

What should the puppet I will be researching with look like?
Do I want to set a clear frame with its appearance? Prescribe interpretations?
What type of material do I want to use?
What must the puppet be able to do - how should it move?

//collaboration with puppeteer and puppet maker Emilien Truche


Since I've never built a human-sized puppet that can move like a human and be animated by just one person, I worked with a professional. Emilien was there to advise me (I had the ambition to really build the puppet myself). He was recommended to me by a friend who works at the 

.
His first questions were mainly about what postures the puppet should adopt and what expressions it should master. In order to be able to communicate more clearly with each other, I looked for sculptures, paintings, and photographs from the news and social media that tell a story of violence in their posture. 

//studying Sicilian puppet theatre - opera dei pupi


Before I applied for the fellowship, I had planned a short trip to Sicily in October. To study and research Sicilian puppet theater there.                   It was important to me to first study a traditional form of puppet theater intensively in order to incorporate what I had learned into my work. What do I take away from what I have seen? What do I explicitly want to do differently?                                                                                               The Sicilian tradition is one that I can experience and explore in life - although of course it is by no means the oldest form of puppetry (as it developed in the first half of the 19th century). A major component of the Opera dei Pupi is the violence: swordfights, tournaments and battles with monsters, sorceresses and enemies.
This research trip was really insightful and inspiring for my work on „Researching Violence“ - and perfect for this point in the work process. I had the chance to dive deep into the portrayal of violence through (traditional) puppet animation and to get involved with puppet making and puppetry.

I had the opportunity to see several performances, paying particular attention to the study of the gestures and postures of violence.

In addition, I was lucky to visit various studios. I was able to talk to the puppet makers themselves, ask questions and even try out the puppets myself and study them up close.
For example, when a character strikes with his sword, the opponent can be split in half, fall apart or lose his head. In the puppet maker's (Italian: oprante) workshop I was able to study the mechanisms and construction of the puppets at close quarters.

// building up my 'archive' (by collecting sounds, texts, interviews, images and video material of places)


From the very first day, I collected material for my archive for the experimental setup.
I was interested in building up a broad spectrum and I mainly used my iPhone for the documentation. Regardless of whether it was video recordings of
- places and spaces that are commonly associated with violence
or as a counterpart to it
- places and spaces that are commonly associated with peace
- I also tried to find 'neutral' places
- as well as places that I personally associate with violence and peace

examples of sound recordings I recorded:  

- Munich Central Station during the Oktoberfest
- bee colony in Düsseldorf
- shopping mall in Dortmund city center
- eerie sounds at night on the way home
- cicadas by the wayside in Palermo                                                             - sounds of the sword fight of the dolls in Sicily


- I was lucky to be able to attend a Blender workshop (by Elisabeth  Drache, 3D artist and a member of the duo The Ruhrgebieterinnen) at the Academy, which proved useful for the next steps in animating the 3D scan.

NOVEMBER+DECEMBER 2023// ANIMATION + COMPOSITION                         

At the beginning of November I took part in a motion and face capture workshop as part of the DIVE FESTIVAL OF IMMERSIVE ARTS (In cooperation with the Figurentheater-Kolleg Bochum). Irina Shutova and Warja Rybakova (both working for DOCKdigital BERLIN) drew my attention to the plug-in Faceit from Blender after I told them what I was planning to do with my puppet. The workshop also gave me a better idea of what I need to pay attention to when animating the handmade puppet in the green screen studio.

//animating the puppet in the green screen studio

3 days of intensive exchange, reviewing the video material, discussing.

WORKING THESES 
a) the puppet itself feels nothing
b) the puppet is not able to play in comparison to the actor or actress
>>> Which means e.g. that when an actor dies on stage, they are only pretending. This can still touch us, but we know that they don't really die. The puppet dies the moment it dies. Right before our eyes.<<<
c) the puppet is familiar to every human being and therefore accessible (in the sense of not elitist)

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++



With this first material of the human-animated puppet in the green screen in front of me, I began to go through my archive (with sounds, videos etc.) and experiment with putting the individual components together.


In order to be able to act as quickly and flexibly as possible, I worked with Final Cut. On the one hand, to be able to cut out the puppet with one click and, on the other, to be able to put my video and sound material together as simply as possible.                                                    At the same time I am working with Blender and the plug-in Faceit to animate the puppet´s face. Since I had hardly any Blender experience, I always used apps (with AI-based animation tools for faceplay and deep fake, like Tokking-Heads or Reface) in between to be able to check ideas quickly.


The composition of the individual parts allows contrasts to be created between the puppet's expression, physical movements, spoken text, sound and setting.
How do expressions of horror that really reach us come about?
When does empathy arise, when does alienation?

We have access to so many images of atrocities that the individual images lose their impact. My working hypothesis is that both a puppet and a completely artificial person (avatar) have the power to sensitize us as viewers to the experience of visual and linguistic violence. The puppet and the avatar feel nothing; we create their suffering, pain and exhaustion solely through our imagination. This creates a direct connection between the viewer and the representation.

// REALIZATION OF THE IMPOSSIBLE
The days of trial and error made me rethink my approach completely.   

                                                                                                                      I had the feeling that - with everything I tried - I was aestheticizing. And thus trivializingviolence.


>> You can't depict violence without reproducing it in some way. <<

I also couldn't get out of the vicious circle of thinking that everything is just my personal view. How do I get to the point where something becomes more universal? To become a little more detached from myself and my feelings?

And who am I to talk about this?
What position do I have within the "narrative of violence"?

These doubts made me take a fresh look at the material I had collected - and think again in a different direction. 

new thoughts: The relationship between puppet and puppeteer is already a violent one in itself. There are clear power structures as to who controls whom.

Where does the relationship between puppet and puppeteer tilt?     

Hanna Rode

How can we tell about human abysses without reproducing or trivializing violence? This question is the starting point for an experiment with a handmade puppet and its digital face double.

Researching Violence

2023

Drama

I am interested in immersion, the "reality" part of Virtual Reality. As CyberRäuber, Marcel and I have tried and created various avenues into immersion. A very short summary of our findings would be that a high degree of immersion is generally not necessary to tell a story or to convey what you want to. And that immersion generally suffers if you aim for too much realism: the more you rely on your target's collaboration, their imagination and established social and cultural norms, the more they will get "into it". Participation / interaction trumps detail, trumps realism.

But of course, the more you let people interact, the more you lose the the ability to steer a narrative. In this dilemma we usually opted for limited realism (high aestheticization), limited interaction, a limited scope of character, space and time, and worldbuilding as a narrative strategy.



 as an idea manifested itself over the course of years. It basically grew on the compost of CyberRäuber's work in theatre, in which we usually didn't have the time, scope and resources - and more importantly - not the framing for such a project.



 aims to be maximally immersive with a given, limited technological effort, following an approach of a minimally detailled realism, maximal imagination and direct, personal contact.

Hundred thousands of years of evolution have led us to develop certain quirks that can help storytellers create compelling narratives, something that can also be ab-used, of course. Think about what we imagine in clouds, in the shadows of a forest lighted by flickering flames of a campfire. What kind of faces we see in objects, landscapes, moving leaves, what voices and sounds we hear in the wind. The less there is to see or hear, the more we augment; little impulses, visual or aural clues are launch pads for our imagination, something every horror movie, every mystery story told on a nightly walk makes use of.



One way of direct, personal contact is the human voice. It can supply content and - with language, dialect, wording, timbre and emotion - context. We ascribe very much to a voice that could only be safely verified by meeting and conversing with the person with this voice.
In this project, the means to deliver the voice will be a recording, a voice message. Why? Because it is a controlled way of communication. For the ordinary person an ordinary conversation is something that can be quickly understood and deciphered, the situation, the conversation partner can be assessed. We would normally aim to reduce mystery, to understand what is happening and only continue to engage with the person and the topic if intrigued and feeling save. A voice message instead can be mysterious, incomplete, poetic, out of the norm. It can also contain additional information, environmental sounds. A voice message can be dilligently tailor-made for a situation and our goals. It doesn't need and anyway couldn't deal with an instant reply. The communication here isn't a realtime dialogue, but an elusive conversation over a longer timespan, similar to an infrequent exchange of letters, including lost and/or overlapping messages.




Nevertheless the recipient has to have the feeling of being heard, of being reacted to. And as in any communication, truth and trust plays an important role...

During the fellowship different areas have to be explored:

Voice - characteristics, difference to simple text

AI - what can be done with artificial neural networks? Story, text to speech, speech to text...

Soundscape - binaural sound, artificial and recorded; AMSR (Autonomous Sensory Meridian Response)

The role of time (length, pace, intervalls,...)

As was to be expected, my research plan didn't survive contact with reality. I thought the main focus of my research would be on "Soundscape" and "Story". Instead, most of my time was spent on voice and specifically the creation of text and voice with neural networks, with timing and the delivery system of the messages (AdaptorEx, Telegram). 

"Soundscape" didn't play the expected role due to the realization that interaction with smartphones, the chosen device, is mainly happening visually, and when people listen to voice messages, they oftentimes don't use higher quality headphones and will be surrounded by other environmental audio. After creating and afterwards checking out some binaural and ASMR material on the phone it didn't seem relevant any more in the scope of this research. It is one of those engineering issues that can take a lot of time and are fun to work on, but ultimately don't deliver for an audience.

"Story" got deprioritized when I realized that neural networks, or more precisely large language models (LLM), would take over a large part of creating the narrative. 
Here maybe some further explanation is in order. As CyberRäuber, we've been working on AI in theatre since 2018. In our so-called "neural theatre"-pieces (Prometheus unbound, Linz 2019; Der Mensch ist ein Anderer, Wiesbaden 2021; Mensch am Draht, Ingolstadt 2022; The Merge, Bangalore 2023) we aim to unleash the creative power of generative AI. The text the actors speak and play is created in real time by LLMs and transmitted to in-ear speakers via text-to-speech. You could also call this a text-to-body-mechanic, the actors repeat and interpret what they hear and thereby embody the AI. The process works amazingly well. While still maintaining parts of improv theatre, this isn't improv because the actors follow the logic and emotional content of the texts. Actors can never lose focus, have to stay "on" all the time and are oftentimes visibly struggling with their task, because the text will always be new and unknown, its twists and turns unforeseen. This is fresh material! The audience on the other hand usually themselves struggle with the uncanniness of the whole set-up, try to follow what's happening on stage, enjoy the actors' struggle and they take on much more responsibility, because the play develops on stage and in their heads.

So, AI is already playing a crucial part in CyberRäuber's artistic methods, and we've been following the latest developments in the field on a daily basis. And with the current generation of LLMs (2023/24) we now actually possess very powerful storytelling tools. The question was: is one person alone capable of creating, editing, curating large amounts of coherent, stylistically appealing, sensible text? Can a network convincingly take on the role of a narrator? To check and ultimately prove this thesis became one of the main isssues of the research at ATD.


This documentation is an ongoing process and will grow as the fellowship progresses. Last entry: 30. November 2023

Björn Lengers

You just received another one of those mysterious audio messages that strangely overlay your surroundings and ask you to perform a seemingly insignificant action: to walk a little slower, to press the pedestrian lights even though you don't actually want to cross the street, something else. A voice wants you to believe that you are decisively changing the future.
- - - Resonance is an interactive, immersive and theatrical narrative, which can be experienced by many people independent of time and place on their smartphones.

Resonance

In 2014, Ian Goodfellow and his team published their ground breaking idea of 

, and since than many exciting works were done with this idea.

A typical GAN consists of a Generator and a Discriminator. A Generator would generate “fake” data while the Discriminator will try to distinguish between “real” and “fake” data, and we could compare that to an art forger and an art historian/expert. The GAN are usually constructed in way so that when we proceed with the training, both the Generator and the Discriminator would get better at their job. Computer scientist has developed different metrics to measure the performance of the network they developed. Once the trained network get to an optimal point, the would stop the training, and the trained Generator could then be used to generate realistic looking data.

 has listed a lot of different projects that people have been exploring with GAN. The myth and the promise to create something that doesn't exist before is very intriguing, that's how we decided to start our exploration with the GAN network. And we decided to start with a very popular network called the “Stylegan”.

 is first proposed and published by a team of researchers working in Nvidia in 2018. Since then the team continues to make improvement and created stylegan2, whereas there are official implementation created by Nvidia to support training in 

, which is a popular AI software library researchers use these days. The latest version is 

. For all these years Stylegan remains to be the state of the art network for generating realistic looking images.

Since the GAN are said to be able to create something that don't exist, we are wondering what would happen if we train the network with a theatrical images. The idea is to start with some time lapse pictures of a single production and see if it could generate a “look” that never happen in that production.

So we have the idea, but how should we implement that? How do we use those networks? What kind of hardware is needed? Do we train the network locally or online? These are all issues that take us sometime to figure out.

Stylegan3 is not yet published when we started the fellowship in September 2021, so we actually began with Stylegan2. If we look at the requirements on 

, it said this: “1–8 high-end NVIDIA GPUs with at least 12 GB of memory. We have done all testing and development using NVIDIA DGX-1 with 8 Tesla V100 GPUs”. While this sound pretty normal for big tech company like Nvidia, for other institutions this might not be something that to be accessed easily. And the 

 time is rather long. For example training a picture of 1024 X 1024 up to 25000 kimg will take 46days with only one V100 GPU. Usually the training happens locally with powerful computer and graphics card, but before we committed to such local training and request the related resources, we've explored some training online.

Below are the training results using Runway ML. The picture datasets are the time lapse photos from 3 different productions. There are different training settings and we have used transferred training provided by the Runway ML.

Twelfth Night/15000 Steps/pretrained “landscape” model)

Death of Yazdgerd/10000 Steps/pretrained “illustration” model

The Seagull/10000 Steps/pretrained “illustration” model

From a distance, these generated videos look pretty “real”. But when we look closer, they are of course missing a lot of details in comparison to those original “real” pictures. We discovered that no matter what size pictures we upload, Runway ML will 

. And this leaves us no choice but to explore the option of local training, so that we have more control over the training and can see if that would benefit the network we wish to train.

We started our local training on a window laptop with an external GPU (Nvidia RTX 2080ti), which only has 11GB video memory. At that time styglegan3, which gives you more option to control how much video memory you use, was not available.

We managed to install the software needed on the laptop and start the training. However, the training will stop randomly whenever the computer running out of video memory. So for a while we have to check the status of training everyday, and when it stops we would just have to restart the training. We discovered that this phenomenon is mostly related to the process of metrics calculation that needs more memory. Once we have disabled that the training become more stable, but it will still stop from time to time. This is why we also started to explore other online options for more stable training.

After some research on the internet, we discovered that many people are doing their experiment on 

Colab is an online linux & python-based programming platform that meant to be used interactively, meaning that originally it wasn't designed to be used for long time training. Compared to running things locally, it saves time and perhaps hard drive space to create many virtual environment, and you could test your code quickly with a Chrome Browser. Of course there are 

 of such a service. With the free version you could only run the code when your browser is active, and even for paid Pro+ version your code will only run for 24 hours from the first execution.

The benefit of training on Colab is however you will get GPU that are more than 12GB, which means you will be able to run the training without any memory issues. There are many colab notebook that researchers share online, and it's quite straight forward to use them once you copy them into your own google drive. An example is the 

 made by Derrick Schultz, where he has also modified the code for his own interests. We did some training on various dataset with the notebook, below is the results of the “Twelfth Night” dataset.


Since we are now running the software on appropriate GPU and with 1024 X 1024 resolution, we are getting better results. But if we look closer a lot of details are still lost. This is probably because there are too few pixels for the network to understand the features of the performer/costume fully. One thing to notice is this production is almost a single scenery production, and the ceiling is always there. Apparently the network are catching the features of the ceiling and other fixed elements pretty well.

The software package also allows us to create interpolation video:

The drawback of using online training is that we have to restart the training everyday. We've also subscribed to Google Colab Pro+, which allows us to access better graphics card. The services of the Colab did change a little bit during the period of the fellowship, and since November 2021 a google Colab Pro+ user will get NVidia V100 16GB GPU the minimum and sometimes get NVidia A100 40 GB GPU, which are much faster and stabler compare to what was available before.

Colab will continues to be a great resources for independent research who might not have the resources to build their own deep learning machine, especially in the time when both the materials to make computer chips and the graphics card itself are in shortage.

Since training a new network takes a long time, we try to utilize any resources we have even in some computer we need to reduce the the resolution to 512 X 512. Below is the results of the Stylegan2 training of Theater Dortmund Archive, which contains over 21,000 pictures of Theater Dortmund's productions in the past 10 years:

This training was done in the local linux desktop computer with 2 X NVidia RTX 2080 Super, we have to set that to 512 X 512 so that the training works with the 8GB memory we have. Since we are able to use 2 GPUs at the same time the local training is faster but it only works in 512 X 512 resolution.

By doing things in parallel we are able to explore different datasets simultaneously, although it still takes a long training time and needs to be taken care of daily.

On October 28 2021, NVidia share their official implementation of 

 on github. This development allowed us to training more efficiently both offline and online.

Because the Stylegan3 network is using a different network architecture to understand the datasets, so we have to retrain all our network to see the latest results. The software package would still let you train in Stylegan2, and there are two option of Stylegan3: Stylegan3-R and Stylegan3-T. However these training configuration are not exchangeable, which means once you started the training you cannot change the configuration it uses.

We trained a new dataset, which contains around 24,000 pictures from 15 productions, on a local linux desktop with 2 X RTX 2080ti GPUs with Stylegan3-T configuration, and the results are below:

We now have an AI that help us to generate 2D images, but our target is to have something in 3D, that's why we have look into 3D reconstruction tools that are available.

, a pretrained depth detection network that works with any images without the requirement for the image to be trained with the network. This network allowed us to complete the pipeline of 2D image generation to depth image generation.

With the 2D image and the depth image, we could then move on to 3D reconstruction. We've use instancing technique in TouchDesigner, but other software that could utilize a colour and depth map in 3D creation should be able to create similar results.

Throughout the fellowship we almost never stopped the Stylegan3 training, both online and offline. Below represent the results combine with 3D reconstruction:

Time lapse photos of 15 productions on the left; around 1,100 pictures of theatrical design on the middle; and Theater Dortmund Archive on the right.

 We discovered with all these trainings that the network can actually recreate the dataset pretty well. In the case of the 15 productions datasets, it is almost always able to generate pictures that look close to the original production.

The pictures that generate from the trained network, however, can't really create a “Theatrical Design” that doesn't exist. We think there are a few reasons why that's the case:

1. The first reason is related to the limitation of resolution and hardware.

 The Stylegan network excels in learning and generating human and animal faces, which are usually close-up photos, or generated landscapes which are quite abstract. The theatrical design images we feed the network are mostly wide shots that cover the whole stage, which usually contains a lot of details. While the network is able to learn the overall arrangement of the “stage”, it failed to recreate the costumes and the faces accurately. We've mentioned that we trained the network with 1024 X 1024 images, one could expect that if we train with even higher resolution say 2048 X 2048 or even 4096 X 4096, we could get better results. However, equipment that could train such a network might be difficult to access at this point for researchers outside those big tech companies. Still, we could expect soon 4k or higher training for images will be available and it will be worth then training the network again.

2. The second reason is related to our perception and understanding of theatre. 

The features of a real human or animal is definite: you have brown eyes, black hair and tall nose, and these features probably won't change much. So when the researcher claims that the network created a person that doesn't exist, what they means is it will be hard to find a person with the exact same features in real life. In the case of theatrical design or theatrical images, we however have more tolerance. Small changes in the position of a performer doesn't really create a “new” design. Theatre is imaginary in nature, and theatrical design requires a certain degree of specificities that the Stylegan is not designed for.

3. The third reason is related to the handling of the dataset.

 All the images are currently treated as one big pool of data, which means the network has to learn common features from some very different pictures and it is beyond the design of the algorithm. We might be able to get better results by dividing the picture into different categories. This is called conditional training and will however requires a larger dataset. We did try to use conditional with the 15 productions dataset, but the network failed to converge to usable results.

With all these limitations we found with applying stylegan3 in theatrical design, we then started to explore other ways to use the network. One of the directions is an exploration of using the tools with the subject “Lilith”.

While investigating the possibilities that AI and Deep Learning could offer to the technicalities of the design process in theater, we also decided to see if in some ways it could provide research material for the creation of content during the design process.

Could we use the outcome of the learnings, based on the specific subjects of the play or the main theme of the play, to gather ideas about color patterns, textures, and shapes to use as a resource during the design process? Could the outcome of the deep learning be inspiring to the design process and provide new ideas, providing a new point of view to the storytelling?

We choose a subject that could be of interest in the theatrical matter.

Lilith is a research work looking for the lost myth of devouring femininity. Lilith is the first wife of Adam, who at the time refuses to condescend in the sexual relationship with the Edenic father, transforms herself into a demon of the night and because of that she leaves of course paradise and is erased from the pages of the Bible. Lilith did not like to live as the subservient wife of Adam and hence went to the deserts, desiring solitude and freedom. It is the story of a woman who threatens to shake the very foundations of male-centered faith and discover a new self. Lilith's name means 'night monster' and is derived from the name of the Babylonian female demon Lilit whose male counterpart is Lilu.

From there on unfolds a mixture of legends, tales, magic texts, and cabalistic evocations. When three angels tried to force Lilith's return, everything was vain. Lilith is also described as a vampire child-killer like her classical counterpart Lamia. The evil threatened by her against children was said to be counteracted by wearing an amulet, which bore the name of the three angels. Really, Lilith is the last link between the Jewish culture and the ancient God, the giver of life and death, of which the first biblical woman only inherits the negative aspect of the giver of death.

Lilith is might expelled from the Bible, but that didn't avoid her to be at the center of the conversation and the artistic production. From Michelangelo to Primo Levi, Marina Abramovich and Monica Grycho, she keeps inspiring the artistic work throughout history and the relationship between the feminine and sacred, the role of mother and woman, and so on.

In principle, we started using VQGAN + CLIP to generate images starting from a caption or a prompt. Google Colab is the online programming environment we approached to run the algorithm. In short: VQGAN is a generative adversarial neural network that is good at generating images that look similar to others (but not from a prompt), and CLIP is another neural network that is able to determine how well a caption (or prompt) matches an image.

https://creator.nightcafe.studio/text-to-image-art?utm_source=google&utm_medium=cpc&utm_campaign=text-to-image&utm_id=text&utm_content=vqgan-clip

https://github.com/nerdyrodent/VQGAN-CLIP

https://colab.research.google.com/drive/1ZAus_gn2RhTZWzOWUpPERNC0Q8OhZRTZ

Even though we tried running the algorithm with several different combinations of words in the caption, the presence of the name Lilith was driving the outcome to a very similar color pattern [dark-red was the predominant color], texture, and final image. We can identify the presence of feminine sexual organs and a cartoonish texture of the image.

Many Deep Learning Networks are able to learn the features from a lot of images and recognize the similarities among large datasets. Like we are affected in our understanding by the world and the context that surrounds us, the same happens for those deep learning networks.

The dataset is the world for the AI to understand and from it to find common features that will be collected and reshaped at each step of the learning process, therefore it must be carefully created.

Here below there are two videos. The first is the quick carousel of the images chosen from the network to train the algorithm, the database, the input to the learning process. The second are some of the outcomes during the learning process, the images shaped by the algorithm in color, texture, and shape.

Samuel Chan

Elena Tilli

From text to music, from picture to video, different tools of Artificial Intelligence have already been used in art making. Could AI also be applied to theatre making, particularly theatrical design, which is a more complex “problem” for the machine?

Coming from a theatrical design background, the research team (Elena Tilli and Samuel Chan) would like to explore if these AI tools could “assist” in theatrical design.

AI Assisted Theatrical Design

2021

We have two correlated creative impulses when creating this piece: 

1. To simulate a post-truth digital media environment in which a wealth of content exhausts the attention of its recipients.

2. To channel the kind of unbounded and unguarded generative energy of digital media into our performance, and to create a performative experience of real-time, algorithm-driven, and rewards-optimized content generation.

With what digital theatrical instruments can we use to build up a performance piece, the process of which is “generated” in real-time through the work of algorithms and human improvisations?

How do we bring the audience into the content generation process and, in so doing, play with the logic of media engagement?

How do we work with GPT-3, the OpenAI language generator, and how do we build applications on the GPT-3 platform for the performance piece?

How do we use other applications (e.g., VUO) to create real-time generative theatrics?

Our project, entitled FXKE, is a performance about the post-truth digital environment active in our current attention economy, an economy that stimulates and prioritizes eyeball-grabbing content to keep users engaged. Through FXKE, we want to reflect upon the compulsive undermining of the infrastructure of truth (a post-truth symptom), and our modern-day condition of information overload. 

RESEARCH PLATFORM

Academy for Theatre and Digitality

HAMLET.AI

Michael Rau Michael Yates Crowley Wolf 359

Artificial Intelligence 2024
activeFellowship

Similar Projects

HAMLET.AI

Michael RauMichael Yates CrowleyWolf 359

Artificial Intelligence2024activeFellowship

Similar Projects

Michael Rau Michael Yates Crowley Wolf 359

Artificial Intelligence 2024
activeFellowship