is the knowledge-sharing platform for the 

. Explore the experiments and bold ideas behind the Academy's Fellowship programme, delve into the science behind the projects of our HIDA researchers, and explore the results of international collaborations.

is a platform for discovery and documentation. Each 

showcases a unique artistic-technological research project, with topics as diverse as the use of AI in theatrical design, choreography for and with robotics, and the possibilities for theatre in virtual spaces. Each 

 offers the chance to learn more about the members of our artistic research network. By pooling our knowledge in such a way, creative research into theatre, performance and digital technologies can become more accessible, sustainable, and artistically innovative.

Funded by the Federal Cultural Foundation, the Academy will award a total of 54 fellowships between 2019 and 2022, each lasting for five months. The 

 are designed for people interested in artistic-technical research which explores the possibilities for digital narratives in theatrical spaces, as well as their practical implementation. Fellows have researched a variety of digital methods (motion capture, coding, XR, VR, AR, sensor technologies, robotics, and more) across artistic fields including opera, dance, orchestra and drama. 

Helmholtz Information & Data Science Academy

(HIDA), the Academy for Theatre and Digitality has offered joint research fellowships at the interface of digital art and cutting edge scientific research since 2021. The HIDA fellowships are kindly supported by the 

Artificial Intelligence

VR (Virtual Reality)

Interactive

Sound Design

Projection/Mapping

3D Printing

XR (Extended Reality)

AV (Audiovisual Design)

Theater Dortmund
Theater Karree 1-3
44137 Dortmund
Germany
Fon: +49 (0) 231 226 14330

Address and location of the Academy for Theater and Digitality:
Speicherstrasse 17
44147 Dortmund
Germany

E-Mail: akademie@theater.digital
Web: theater.digital

Tax number: 314/5702/9367
Sales tax identification number: DE 124643894

Management Team:

Executive Director
Tobias Ehinger

Administrative Director
Martin Lizan

Director of the Opera
Heribert Germeshausen

Director of the Ballet
Xin Peng Wang

General Music Director
Gabriel Feltz

Artistic Director of the Schauspiel
Julia Wissert

Director of the Children's and Youth Theater
Andreas Gruhn

Director of the Academy for Theatre and Digitality
Marcus Lobbes

Please also note our privacy policy.

 as the sixth division of Theater Dortmund (hereinafter referred to as "Theater", information about Theater Dortmund can be found 

), is pleased that you are visiting our website. Data protection and data security when using our website are very important to us. We would therefore like to inform you at this point, in accordance with Article 13 of the EU General Data Protection Regulation (DSGVO) and Section 13 of the German Telemedia Act (TMG), which of your personal data we collect when you visit our website and for what purposes it is used. Since changes in the law or changes in our internal company processes may make it necessary to adapt this data protection declaration, we ask you to read through this data protection declaration regularly. The data protection declaration can be accessed, saved and printed out here at any time.


§ 1 Scope of application

This data protection declaration applies to the Internet offering of Theater Dortmund, which can be accessed under the domain www.theaterdo.de and the various subdomains (hereinafter referred to as "our website").


§ 2 What is personal data?

Personal data is individual information about personal or factual circumstances of a specific or identifiable natural person (data subject). This includes, for example, information such as your name, address, telephone number, date of birth or e-mail address. Information for which we cannot establish a reference to your person, e.g. by anonymizing the information, is not personal data.


§ 3 What personal data do we collect/process and use?

Access data without personal reference is stored, such as the name of your Internet service provider, the page from which you visit us, the names of the requested files and their retrieval date. These data are evaluated exclusively for the improvement of our offer and do not allow any conclusions to be drawn about your person. 

If you wish to make use of services offered by us on our website, such as purchasing tickets via the ticket store or ordering a newsletter, etc., it is necessary for you to provide further data. This is the data that is required for the respective processing, such as your payment information when purchasing tickets or your e-mail address when ordering a newsletter. 

We process the data we collect about you for the provision of the services we offer, to answer your questions and to operate and improve our website and applications. 

We process your personal data exclusively to provide you with a comprehensive service via our website or, if you use our contact form or, to process your ticket purchase or, to process your application or, to provide you with requested information about our offers.

Fellowship

HIDA

PlayOn!

discover cutting-edge digital research in theatre and performance 
meet our research fellows and collaborators in the 

Portal: Academy for Theatre and Digitality

Christiane is working on a framework of how to prototype and evaluate social protocols and society building methods in play and theatre context with (inter)active audiences. Markus is curious about playing around/hacking different technologies. A shared interest that we figured out is smartwatches as social devices.

For interesting group interactions with smartwatches as devices (that do not require mobile phones), it would be interesting to have them interconnected, being able to sense (the distance and more) of each other, to measure the carrier's body-information (heart-rate, movement-pattern) and to be able to function as a communication device (with each other e.g. to push and receive game elements or to receive global push notifications).

Here are the specifications of the hardware we need:

how to organize subgroups (tribes) of an instant society in a way that their actions have meaningful impact and can be evaluated afterwards

how to prototype new kinds of social protocols with intuitive instructions/means

how to model a kind of “social sense” (to feel closeness and distance, to feel attention and aversion)

how to model abstract resources (as time) in an intuitive way

how to make the interface as intuitive as possible

build a network where the watches communicate with one another

build a server which saves game state and communicates with the smartwatches

2 x bangle.js *All the Espruinos!* edition

(Since some months we are watching the following watch  makers 

. Unfortunately through the outbreak of Corona the supply chain of the makers is interrupted, so we do not know if we can receive the devices (that were promised not before March 2020) in time. We started looking for alternatives now.)

a simple notebook which functions as a server

bluetooth hubs and maybe wiFi access points

Christiane Hütter

Markus Schubert

We like to see technology as possible space, or making possible space, for participatory processes. During this fellowship, we will use theatre as a living space for encounters, with the research questions: How can we construct society together? And how can we make this process visible for audiences? 
We want to develop a sustainable format for collaborative discourse, in which people can together experiment with different forms of organisation for society. This could start on a small scale, with a family, and can become quite big: How to organise a country?

******

Performance

Networked Objects

Sensors

2020

Besides the aesthetic implementation, we are interested in two main elements:

1. How can non-verbal communication between space (built from sound and scenography) and spectators imply a plot line or narrative?

2. How can the environment's own characteristics play a role, so that they can allow conclusions to be drawn about motivations and correlations? 

Jakob Lorenz

Stella Lennert

'PALIMPSEST' engages with virtual space as an interactive stage, and explores narration through interaction instead of representation. For this, the project takes theatrical literature as an anchor and combines it with elements of gaming. The dramatic texts act as a common ground, a guideline through unfamiliar virtual environments that follow strange rules. PALIMPSEST is experienced via VR goggles and headphones. 

PALIMPSEST

Drama

Stage Design

2022

Humane Methods (2019-present) is a project consisting of happenings and stage productions where dancetheatre, bio-art, interactive music, adaptive lighting, and AI technology integrate into living ecosystems. Each work in the series acts as a branch of a rhizome: each dives into different facets of the same theme and world. The works created so far – 

 (2020) and the forthcoming ΣXHALE (2021-) – combine in different ways performing arts with technology and the human with the non-human to reveal algorithmic violence through sensory, physical and ahuman strategies. It is an aesthetic research into how a synthesis of symbols, movement, music and AI can enable posthuman forms of empathy.

℧R is a solo performance for a body, AI algorithms, lights and the vibrational force of sound. 

Embodied in Donnarumma’s musical instrument XTH Sense, an interactive, machine learning algorithm transforms the inner sound of muscles, blood, and bones into a real-time choreography of movement, sounds and light. Drawing on Donnarumma’s more recent movement research, the piece combines multi-sensorial stimulation with human-computer interaction and somatic practices to test the seeming impossibilities of bodily articulation and repetition. 

℧R is a walk in the dark. A thin body walks while a dim light shines. A path covered in soil is between them. Gazeless, the figure walks and prays, walks and prays, walks and, entranced, pries into the darkness. Its red cloth sanctifies it. Its missing face alienates it. The walk is a loop, a prayer, an aggression, an annihilation and a transformation. An AI computer vision system watches over the body, attempting to distinguish the body from the ‘noise’ surrounding it. The AI wants to represents what it sees, but the system works against itself in a paradoxical loop; the more the algorithm learns to see, the brighter the light becomes. In the eyes of the algorithm, the body is eaten by the storm of light.

The latest chapter in the Humane Methods series is ΣXHALE (2021-). A poetic reflection on ethics in algorithmic societies, ΣXHALE is a procedural and participatory piece lasting several days.

It develops in loops, performed simultaneously by 6 performers* and an AI computer-vision system. As it watches the performers repeat a proto-prayer ritual, the AI learns their movements in the space and then begins searching for anomalies, something that deviates from what it expects to see. The learning process is accurately transformed in real time in music and lighting, influencing thus the events in the form of audiovisual stimuli. With each repetition, each variation and each anomaly, the loops mutate into stories of violence and empathy and the audience is asked to choose how they want to participate in these stories. The interaction between audience, performers and AI gives life to an endless sequence of micro-narratives.

AI in theatre and performance: Strategy & Questions

Technical Infrastructure

The Autoencoder

A Performing Machine

How To: Technological Infrastructure

Andrea Familari

Marco Donnarumma

Humane Methods (2019-present) is an ongoing project by Fronte Vacuo (Marco Donnarumma, Margherita Pevere, Andrea Familari) that reflects on the new forms of violence emerging in algorithmic societies. The technological research for ℧R and ΣXHALE have been conducted at the Academy for Theater and Digitality and this page describes our research questions, workflow and findings in relation to these projects. Please note that this page does not discuss the theatre pieces we created in their entirety, but focuses only on the technological research that underpins those pieces. For further information about the aesthetic, conceptual and dramaturgical research behind these works please visit our website.

Humane Methods

Lighting Design

The human being is not generally thought of as a staple food source.
Observing the anatomy and skeleton, one can see that the animal is neither built nor bred for its meat, and as such will not provide nearly as much flesh as a pig or cow.
The large central pelvis and broad shoulder blades also interfere with achieving perfect cuts.
There are advantages to this however, especially due to the fact that the typical specimen will weigh between 100-200 pounds, easily manipulated by one person with sufficient leverage. 

'Butchering the Human Carcass for Human Consumption'

In March 2001, 39yo German IT professional FRANKY and 43yo German engineer CATOR made an appointment on the internet. After meeting at Kassel main station for the first time in “reality” after their extensive online communication, FRANKY drove the two men to his house, where he cut off CATOR’s penis - which proved too chewy for the both of them to eat -, proceeded to kill him with a stab to the throat and then stored the pieces of meat from CATOR’s cut-up body in the freezer. Both had previously agreed to this, for while FRANKY longed to fully internalize another human being by eating it, CATOR wished to be fully disintegrated and disappear. The two men recorded parts of their meeting, wherein they were constantly forced to synchronize their quite different imaginations of the encounter with the reality of their shared physical and social situation.

15 years later, Taiwanese electronics producer HTC cooperated with US company Valve for launching a virtual reality headset they had christened “Vive”. Together with similar products, it marked the beginning of a slow mainstreaming of VR applications - a technology long envisioned with a moniker first coined by French playwright Antonin Artaud, the proponent of a “theatre of cruelty”. The “Vive” headset enabled its users to move in a small space within a computer-generated surrounding and interact with it. VR experiences proved to be highly immersive - and for many people at first physically quite disorienting, because they entailed being present in two realities at the same time. The headset was originally meant to be called “reVive” - but the name was changed before it hit the market; the notion of a new wondrous device transcending death or mortality thus obscured.

Subject Matter: Interviews, Autonarrations, Artistic Production

“Virtually incomprehensible - but he was happy, sitting there in his own blood.” In an interview, FRANKY thus comments on the sight of castrated CASTOR sitting in his bath tub. The desire for understanding, imaginations of fulfilment and redemption and their conflict with reality are recurring themes of the Rotenburg cannibal case. “Whatever you imagine, in reality it is basically always different,” says FRANKY later in the same interview, formulating a core experience of artistic work - only to dismiss it at once, chastened by his own experience: “Why should I do something, knowing it will never be like I imagined?”

Interviews and autonarrations by FRANKY form part of this project’s material as well as protocols of chat conversations by the two involved. Research also covers cases of other confirmed sexual cannibals (Jeffrey Dahmer, Issei Sagawa, Arthur Shawcross, Albert Fish…) as well as an extensive collection of images, videos and writings by people from the internet community of cannibalism fetishists - i.e. the artistic production of the scene itself. Here, not the shock value - the 

 - is of interest, but rather the genre’s recurrent figurations and formal rules. What are the conventions for depicting the not only unconventional, but outright taboo? An additional testament to the fundamental discrepancy between imagination and reality are documents of the thousands-of-years-long history of insinuations and mise-en-scènes of cannibalistic desire, which 

: From Saturn’s cruelty and St. Erasmus’ suffering, collective colonialist projections on the “other” and the monistic ecstasy of Kleist’s Penthesilea to Hannibal Lecter’s unsettling pop appeal. These may also hint at the function of said discrepancy.

The question remains: Why imagine something, knowing I will never fully understand it?

Working with “cases” that are in one way or the other “radical” or “extreme” is always in danger of giving in to sensationalism in representation and deprecation, belittling or denial in reception. Not surprisingly, the cannibalism case of 2001 has already undergone wholesale journalistic and artistic processing - not least, it inspired numerous films and theatre plays. Its spectacular value has been cannibalised for good. Unlike real flesh though, memories and fantasies are not destroyed through consumption. Today, rather will they be externalized digitally and reproduced forever.

The subject matter of sexual cannibalism is especially suited for performative VR application because it reformulates fundamental problems of the medium itself: the experience of being physically absent and present simultaneously, the relationship of a technologized world (both men came from technical professions) to the “flesh” we inhabit this world with, not least the desire for encounter and singular experience that is not only the basis of 

 performance and of the arts’ hunger for transgression, but for so many of today’s practices and services. VR as a medium - and XR/MR as its follow-up - performs such transgression by first and foremost creating the experience of being alive and present with one body in two distinct spaces at once - the “virtual” and the “real”. It is a lucid challenge for our consciousness, comparable to the first cinematic experiments - and a challenge for theatre, too.

The fellowship time will be used for research and technical try-outs regarding the setting, 3D visualization und interaction of the planned piece. Self-education on Unity 3D's shader graph, re-creating the physical performance space in VR & avatar creation and live control will also be part of the work. In the unlikely case of a global pandemic interfering with schedules, efforts will be made to boil down plans in a way so as to still achieve a presentable outline of the project involving VR elements.

Development will focus on cable connected higher end consumer HMDs, specifically on HTC Vive / Steam VR. Besides a Vive/Vive Pro headset with base stations and controllers, at least two Vive trackers will be part of the setup (more if used for performer body mocap). Wireless microphones for performer(s) and a computer being able to run the VR elements are also necessary.

The piece will reproduce a model of the actual physical performance space in VR, so knowledge of it and development time to produce the appropriate design beforehand as well as enough setup time to ensure correct alignment of virtual and real space are necessary.

As of now development aims for a ca. 30 minute 1-on-1 performance alternating between real and virtual space in the course of its narration.

Arne Vogelgesang

A guided, interactive VR performance which explores transgression, sensationalism and spectacle through the subject matter of sexual cannibalism. This topic is especially suited for performative VR application because it reformulates fundamental problems of the medium itself: the experience of being physically absent and present simultaneously, the relationship of a technologized world to the "flesh" we inhabit this world with, and the desire for encounter and singular experience that is the basis of one-to-one performance and of the arts' hunger for transgression. It is a lucid challenge for our consciousness, comparable to the first cinematic experiments - and a challenge for theatre, too. 

New Flesh

Avatars

Motion Capture/Tracking

How can we (as a field) make a dance in a virtual environment so that the audience “feels” the presence of the digital body?

What is needed for the audience to find new aesthetics when dance is mediated with digital environments like VR headsets?

, which explores the performative possibilities of VR/XR technologies through dance. It questions the concept of a 'stage' when dance occurs simultaneously upon hybrid stages (for instance, in both virtual and physical realities) and asks: how can this kind of content be shared with an audience?

 Virtual reality creates an experience where introspection can be achieved. This opens up possibilities for somatic and other movement practices. Working with the body and technology presents the opportunity to explore rigid computer systems through an ontological approach (praxis/doing), and to explore the medium's own possibilities. Often, dancers who work with technology forget their own embodied language (the 'why/how') and get confused and carried away by the pragmatic reality of technology, focusing instead on the content (the 'what'). They forget their own embodied power. The result is a superfluous object.

To tackle this I propose a praxis which privileges the process over the result. I work with avatars and sigils as a ritual practice with a non-anthropocentric approach, and the key aspect is to shift the focus in choreographic strategies. Instead of asking "what to 

This project continues my research into dance in VR that started in 2022 at 

Motion Bank - University of Applied sciences Mainz

 (Project originally founded by choreographer William Forsythe). 

Using a markerless motion capture system I created, I explored and documented simple visual representations of the body (avatars) in VR. 

. I achieved this using a non-anthropocentric approach. In the documentation it was important to use an inclusive, clear language that addressed both dancers and technologists. In the 1st cycle at Motion Bank, the focus was on the dancer.

In the 2nd cycle at the Academy I want to add a layer where I introduce the stage, thus exploring the relation between audience and dancer.

The new performance rather offers different entry points to the audience so that every audience member can have a different interpretation. Nevertheless, the core of the performance remains the same; we investigate a theatrical narrative while having virtual reality as an actor at the center of the stage. In the first performance, we used a linear narrative around VR to convey how we conduct a research. In the second performance, we used a less rigid structure, namely structured improvisation, to show how we develop a performance. It can be seen as a making-of where Jorge runs around the stage and prepares each technical element for a “real” performance although there is no such a “real” performance that comes out of the making-of.

Though our intention is to show behind-the-scenes of developing a VR-performance, we do not want to limit the scope to “struggle of working with technology in theater” because this theme may be too specific in the industry. We try to add embodied elements and add breaks in between to show humanness of a theater maker. The audience can relate this process to everyday technology such as smartphones, smart devices and potentially VR headsets, which are becoming a commodity.

For the overall structure, we divided the performance into small skits so that each skit features one element (e.g, video of eyes, VR meditation app). As a structure, we might think it follows an idea of Fluxus or it is minimalist; nonetheless, in the actual performance, we see that each element is not separable from each other and we can notice the interconnection between the selected element and other actors on the stage that are not mentioned (Jorge’s body, desk, floor, …). This is because working with technology is inherently a complex process.

Between Information and Noise :: a Practice on dance and VR

: To explore different perceptions of realities when dance happens within Hybrid stages - (VR and on-site)

Missing conceptual context to apply technical tools.

The natural empathic and embodied relation between dancer audience is partially lost when using VR headsets. This research aims to help the audience to find new aesthetics (

mediated experience of time created between the artwork, the artist, the interface and the audience

), to find engagement, new meanings and purpose when dance is mediated with digital environments like VR headsets In hope to 

create a visceral sense of unfolding or revelation, but also a feeling of immediacy and loss.

Exercises on subjective perception of time and space. The aim with this series of videos is in questioning, What is happening in real time.

. An action of a person drinking water (action 1) is recorded with 2 different camera perspectives. A setup in OBS is set to switch between 3 different perspectives of the camera; the 2 recordings and a 3rd live feed. A person comes into the space. There is a projection that switches between the 3 perspectives while a person is in the space doing the action of drinking water. This action is recorded. 


Sigil

Pathfinder

Choreographer, Media and Interaction Designer, Researcher

Jorge Guevara

The natural empathic-embodied relation between dancer - audience is partially lost when using VR headsets. This research aims to help the audience to find new aesthetics, new engagement, new meanings and purpose when dance is mediated with digital environments. Through two live performances, I created a visceral sense of unfolding or revelation, but also a feeling of immediacy and loss. 


Sigil :: bringing the inner world out

2023

Dance

Choreography

. This machine enables a haptic and participatory theatre experience that can be delivered directly to the audience’s home regardless of quarantine or lock down policies. Without human performers, the machine can talk to the audience, record responses, take photos and videos, and show images printed on tapes which are moved by mechanical scrolls, while respecting all public health and safety regulations.

In our research residency, we continued to develop the existing prototype of the machine and expand its basic functions regarding presentation techniques to permit more user interaction. This includes experimentation with new graphical, mechanical and light-based animation techniques, such as RGB colour filters and moiré effects. In doing so, we hope to bring new spatial and expressive dimensions to the performance in the pizza box by mechanical means. On the software side, we developed an abstraction language for the presentation modes of the box, which allows the scripting of complex storyboards, including the possibility for “choose your own adventure” style dramaturgies.

The goal was to develop the prototype into a reproducible open-source toolkit that can be used and replicated by anyone with access to modern rapid prototyping facilities (i.e. FabLabs) and filled with personalised content. Our research is informed by the intense dialogue between technology and dramaturgy, as we ask ourselves how dramaturgical building blocks can be implemented mechatronically and combined to make the performance machine as stable and versatile as possible. The result is a new performance format which can be filled with different contents for different contexts and thus reach new audiences directly in their homes. Our work aims at opening the established format of food delivery for artistic and dramatic content to make performance as easily accessible as ordering pizza.

In the course of the residency we also explore the potential of the technologies we employed on a small scale as techno-dramaturgical building blocks for more conventional theatre spaces and interactive stage machinery. In November/December 2022 we were able to test the new model of the performance machine in. the pizza box site-specifically in Berlin with 

 single board computer (SBC) model 3 B or 3 B+, which controls the interactions with the audience. The main user interface are four buttons which trigger actions of the machine, e.g. start, advance through or repeat sections of the dramaturgy. The box speaks to the audience by playing back prerecorded sound files through speakers or headphones. Images on scrolls moved by motors and illuminated by LED strips illustrate the narrations. Furthermore, the machine can record sound and video with a built-in microphone and camera. The latter is installed in the lid pointing down onto the surface of the box to capture what audience members are drawing or showing on the box. Thereby the box can also become a stage for the audience to actively participate in.

We started our research based on a first prototype model No. 1 IGL which we had successfully used to deliver 

to audiences. it fits into custom 5cm high pizza boxes. This model is battery powered and has one red and one blue button for advancing and repeating storyboard sections. It is able to record audio and video, play-back sound through internal speakers and show black and white illustrations on a single tape of backlight foil which is moved vertically by DC motors. The running time of the performance is 30-40 minutes and most of the construction parts are based on 3D-printed technology.

Only one box was produced, which still works as of January 2022 but was never replicated.

This model was a hybrid model which was replicated twice but never worked well due to technical problems. It contains a shortened version of 

 which takes a more abstract approach and is therefore also accessible to audiences who have no connection to Innsbruck. Rather than hearing and being asked about Innsbruck, this box uses the generic term “your city”. The running time is 10-15 minutes.

The containing pizza box is also 5cm high. The electronic controls utilize a 

 microcontroller in addition to the Raspberry Pi SBC. Being conceptualised for gallery exhibition and marquee-style presentations, it does not have a battery but works with a AC adaptor. It uses headphones instead of speakers and has a separate start button. Two perpendicular tapes are moved by DC motors, overlaying 2 images printed on backlight foil using an LED lighting system which can provide front- and backlight. It can be seen as a work-in-progress phase between the first prototype 

 since it combines solutions from the first model as well as newly developed elements. Despite two attempts of rebuilding it, technical issues due to electromagnetic interference between motors and sensors prevented it from working in a stable fashion.

 pizza boxes. The mechanical construction is optimized for laser-cutting, only using 3D printed parts where it is absolutely necessary in order to save time in the manufacturing. It can be used with an AC adaptor or battery power from an internal LiIon-cell pack. Sound playback is possible through speakers or headphones. The electronic controls use a 

 microcontroller with an advanced sensor configuration for the two perpendicular scrolls. This allows the scroll movement direction to be recognized independently from the motor movements, as well as the ability to detect rips and other scroll-related malfunctions. The “stage light” LEDs are full color RGB+White and the audience has more interaction options with four buttons that allow non-linear “choose your own adventure” style narratives. The model RI is currently in active development. The plans and software code are open source and available via our git repositories for reproduction. The system provides a stable basic set-up that can be altered, expanded and developed further by the maker community.

 in our project description, the research process cannot be separated so clearly into three specific models. Rather, they merely represent more or less arbitrary snapshots of our continuing experimentation. Another approach to describe our work in this residency can be taken by answering our research questions. 

The first two questions - "HOW CAN WE EXPAND MODES OF INTERACTION WITH THE BOX?" and "HOW CAN THIS FORMAT BE TURNED INTO A REPRODUCIBLE OPEN SOURCE TOOLKIT?" - deal with the technical design of the box in software and hardware. We will explain our technical experiments and the redesign process in answer to the first research question, regarding all modules of the machine. The second question will be used to reflect on the ideas, process and documentation of the toolkit.

The other questions concern the context of the performance machine, which we researched alongside the technical development of the box. Question 3 ("HOW CAN INVISIBLE CITIES BE TURNED INTO AN ECOLOGICALLY TOURABLE PRODUCTION FOR OTHER CITIES?") focuses specifically on the practical application of our showcase performance project 

, and is concerned with the format of that project, how it can be realised in different contexts / cities and therefore reach new communities and spread the application of the performance machine in the pizza box. 

The final question, "HOW CAN THE PIZZA BOX TECHNOLOGY BE EXTENDED INTO A ROBOTIC PERFORMANCE SPACE?", concerns itself with the application of the technologies we are using for our pizza box for the theatre as a whole. It reflects on different spaces and the use of robotic or mechatronical systems for stages.

1. HOW CAN WE EXPAND MODES OF INTERACTION WITH THE BOX?

In order to expand the image world of the box, we wanted to extend the singular tape of the first prototype, IGL, with more tapes filled with images which can be overlayed on each other.

However, it soon became apparent that the optimal number and arrangement of scrolls for our purposes is two perpendicular tapes which can be moved in horizontal and vertical direction. Since we set ourselves a size limit for the performance machine to fit inside a pizza box, any scrolls in another orientation would further reduce the space that is available for viewing images. While theoretically it would be an option to split the horizontal or vertical scroll into two independent ones, this would also create an image free zone in the centre of the view port. Therefore we settled for two scrolls and addressing the challenge of even light distribution and the possibility of fading out the image on the back scroll. It also became apparent that the scrolls need to lie on top of each other as close as possible, so as to avoid the back image being blurred. Even fractions of a millimetre matter in this respect.

The scrolls are the largest 3D printed parts in the construction. They are printed in two halves down the central axis, which are held together by laser cut acrylic gears and rings. They are mounted with regular skateboard wheel bearings. The split construction also allows for an easy and fault-tolerant mounting of the image tapes, which are merely inserted between the two scroll halves. In case a motor does not stop when the band reaches its end, the tape just slips out of the scroll without ripping.

From the experience with our prototype model IGL, it was clear to us that moving the images on the scrolls accurately and reliably is one of the biggest challenges. We therefore prioritised this right at the beginning of our residency, not expecting that the problem would accompany us all the way to the end. Reliable and precise motor control is not an easy endeavour, and despite knowing this from the onset we still underestimated the problems in the implementation.

We chose geared, back-drivable DC motors because they are cheaper, smaller, lighter and much less noisy than stepper motors. In the first model, the scrolls were driven directly by the motors. However, this turned out to be less than optimal, as the motor axle tended to tear out of it’s mount in the 3D printed scroll. We therefore decided to implement a simple gear train between the motor and the scroll. This proved to be a great improvement, as the gears double as a cuff which holds the scroll halves together.

 due to their small size and nominal speed of 135 rpm. First test where conducted with 350 rpm motors, but despite a further speed reduction from the acrylic gearing, this turned out to be too fast. Another advantage of slower speeds is that the available torque is increased. At first we underestimated the force exerted by the tightly wrapped tapes around the scrolls. In the original design, the motors powering each scroll were attached in reverse to the same motor controller, with a diode on each motor lead so that inverting polarity on the controller only powered one motor. While this required fewer components and reduced complexity in the controller, we realized that the torque exerted by one motor was barely enough to pull the tape. This made the motor control unreliable, as it was always necessary to give the motors full power in order to move the tape. The solution was to control power for each motor individually, which allows us to run the secondary motor at slightly slower speed, which greatly reduces the tension on the tape and thus also the torque the driving motor needs to exert.

Another important aspect about the motor control is precise position control, i.e. how far the tape has travelled and when it needs to be stopped. This means that the device which reads the position encoding from the sensors needs to be able to operate in “realtime”. Our experience with the IGL prototype showed that the Raspberry Pi SBC is not a good choice for such tasks, which is why we decided to include a microcontroller for time critical hardware control.

Another problem caused by motors is electromagnetic interference. In model SIBI, we experienced issues with the motor control because one motor interfered with the position control sensors, causing the positions to be misrepresented. After narrowing down the problem from software to hardware and finally localising the motor as the source of the error, the mitigation was to equip the motors with dampening capacitors, and use shielded cables for the motor’s power supply and the sensor wiring.

The prototype IGL relied heavily on the use of 3D printed parts. This created two problems: First, the fragility of the parts made frequent replacements necessary, especially in the initial testing phase. Second, the parts were relatively large, and thus required substantial time for printing replacements. Therefore, we decided to switch to a construction that can be mostly laser-cut from plywood (or MDF - see below). The four main parts can be slotted into each other and the base plate to form a sturdy base. Increased stability is reached by connecting all parts including the outer walls with screws.

All screwed couplings are made using square nuts which are inserted into slits in the receiving part with perpendicular holes to guide the screws. These holes need to be drilled manually after laser cutting. To streamline this process, we built a small drilling jig using a 3D printed base and brass tubing, which centers the drill bit and ensures perpendicularity. This allows all holes to be drilled with a hand-held machine, further speeding up the process.

The supports are the central structure of the box, holding both the motors and sensors (see below), as well as forming an enclosure for the light box. This efficient design stood at the beginning of our experimentation process, and was refined iteratively to also feature mounts for holding all cables in place safely.

Initially, we used 4mm and 8mm poplar plywood for prototyping on the laser cutter. However we noticed that the wood was of very variable quality, which impacted the structural integrity of the parts. We therefore switched to 3mm and 8mm MDF board for the final model RI. As this material is harder for the laser to penetrate, it prolonged cutting time substantially. The change also increases the weight of the box in comparison to it’s predecessors, however the increased stability of the resulting parts made the change worthwhile. For future iterations, we would recommend considering redesigning the parts from 8mm to 6mm wood, in order to reduce the weight, increase the space within the box (every mm counts) as well as reducing cutting time and burns of the laser cutting process.

Since the prototype model 1 IGL we have been using optical sensors in combination with encoding bars on the backlight foil to determine the tape position. Since then, we have updated the sensor configuration and bar layout multiple times, but the electronic components for sensing are still the same as in the first device. In the course of prototyping experimentation we considered using sensors with a smaller form factor, however we noticed that the tape has a considerable lateral motion due to manufacturing tolerances in the entire scroll-mount system. Therefore our original sensor models turned out to be the best for the job. The sensors use optical transistors as a switching element, which need to be paired with different resistors depending on the material used for the tapes. Initial electronic prototypes therefore needed to provide means for exchanging the resistors depending on the tape material.

Finding the optimal tape material is an ongoing investigation which has still not shown a clear winner. In the first prototype we used backlight foil with good results. Being made of sturdy plastic, it is virtually indestructible. However, that can prove to be a disadvantage if the tape is affixed to the scrolls permanently, which resulted in the damage to the scroll-motor coupling in model IGL. Also, the foil is covered with a white coating which is a base for the print, but rubs off easily, especially along the cutting edges when the tapes are trimmed to size. Also, we realized in Dortmund that the base material comes in different thicknesses, but it can be hard to determine which thickness is used before receiving a sample, as most printers do not provide this information to their customers. The backlight foils we obtained from the local print shop turned out to be too stiff for our needs, as well as having long production times which were not conducive to our rapid prototyping process.

We therefore switched the material to regular tracing paper, which is also available in rolls, and can be printed by most copy shops with a much shorter turnaround time. The greatest disadvantage of this material is that it tears easily, as well as also being available in multiple thicknesses, each of which has its own set of problems. Nevertheless, we have decided to stick with it for now, but are still on the lookout for better alternatives.

To be able to create transition effects between the images on the tapes, the new models can illuminate the “stage area” from the top as well as with backlight. Top light is provided by a side-lit screen of transparent acrylic which is mounted above the light box. The light itself comes from LED strips, which need to be mounted on a heat sink to avoid damage. The backlight is therefore mounted on 2mm sheet metal, while for the top light we used 4 x 10mm brass.

The first two models use white light only, which could be dimmed using the same motor controller board we also used for the motors. For the third model we switched to full colour 

 strips, which also allow the control of each LED individually through a one-wire communication protocol. While this is not (yet) used in our models, it allows many more possibilities in the present and future models.

The first prototype model No. 1 IGL used two consumer-grade power banks, one providing 5V via USB to the Raspberry Pi and one with 12V which we used for the motors and lights. They needed to be removed from the box for charging, which made box maintenance cumbersome. We therefore decided to switch to a single LiIon battery pack (3S2P) with separate converters for each module (i.e. lights, motors, sound and controllers). The box is delivered in standby mode and activated by opening the lid. In the latest construction, the battery is permanently fixed in the box and can be charged through an external charging port.

One remaining issue is that the power consumption from the four motors and lights is considerably higher than for the prototype model IGL. We estimate that with the current power consumption, our 6800 mAh battery is enough to sustain a one hour performance including delivery time. As motors tend to consume more power depending on the torque they exert, some power issues could be mitigated by fine tuning the motor speeds. In general the energy consumption of the entire system remains a field with high optimization potential. In future models, the box should monitor the power level autonomously, and also have a sleep mode for delivery.

Sound playback and recording is performed by the Raspberry Pi using a standard miniature USB sound card. This is necessary because the Raspberry Pi lacks a microphone input. 

An advantage of this setup is that the wiring for the sound can be soldered on to the jacks of the card directly, which saves the space that would be needed for a plugged connection. For the microphone we use a passive table top stereo microphone which has a high recording quality. The internal speakers are driven by two miniature 3 W amplifier modules. The lid switch also controls the standby mode of the amplifiers.

The laser cut base for the microphone doubles as the cage for the battery, which makes excellent use of the cramped space conditions in the box.

Video recording is also managed by the Raspberry Pi using a 8 MP 

 supports recording in full HD resolution with 30 fps. However, the default file format is raw .h264 video, which makes post-processing necessary. Therefore the latest version of the software includes a routine which converts the videos to MP4 format after the performance playback has been completed. All recorded files are saved to a USB stick for easy access after the box has powered off. Files from every performance session are stored in a separate folder with a unique identifier.

We expanded the two buttons of the first model IGL (one for forward and one for backward) to four buttons in order to allow the performance machine to support more complex stories and non-linear dramaturgies. To indicate when the buttons accept user input, we decided to use illuminated buttons. A side effect of this is that we also have a possibility of informing the audience of the current state of the performance. For example, the red button can indicate that the audio recording is active, and blink when only 5 seconds of recording time remain.

However, for unknown reasons it was impossible for us to find a German supplier for four illuminated buttons in different colours which also fit into the 40 mm height of the box. We therefore used coloured buttons and added an LED of the same colour to show the audience when the button can be pressed. We hope to change this in future versions of the box, although we do find that the seperate LEDs give the box a certain 1980s flair.

We designed a custom PCB to hold the microcontroller and connect it to the peripherals (sensors, motor drivers, buttons etc.). This also is necessary to be able to build the performance machine in a workshop setting, as we found out during construction of the first model IGL that soldering a custom wiring interface on a prototyping board can be very challenging for unexperienced makers.

Assembling the electronics is a lot more failure resistant when connections are protected against mistakenly being plugged in the wrong way round. We therefore use standardized connectors with mechanical reverse polarity protection for all peripherals. Using crimped 

 also makes it a lot easier to assemble the many wires that are needed to connect the sensors.

SOFTWARE FOR RASPBERRY PI AND MICROCONTROLLER

Already in the first model IGL we began writing a Python package which can be easily deployed and run as a service on the Raspberry Pi. We use the 

 to create a command line interface which can be parametrized to be used for testing the assembly and to run the performance on delivery. As stated above, the original software was intended to control the motors and lights as well as play and record audio. While the direct control of the hardware turned out to require a separate microcontroller, large portions of the original Python software could be reused for the modular design of the architecture.

Our first choice for a microcontroller was the Raspberry Pi Pico, due to its support for the Python programming language (specifically 

). As the main code running on the Raspberry Pi SBC is also written in Python, it seemed attractive to be able to reuse the control algorithms already written in the same language. However, it turned out that despite claiming otherwise, MicroPython is not suitable for critical realtime applications. Therefore, we replaced the Pico with a 

 microcontroller, which is compatible to the 

 and is programmed in C++. We also switched the programming environment (IDE) to 

. This proved to be a much more stable experience on the whole. Due to the high clock frequency of the CPU, we are now able to control the motors and the lights simultaneously without losing control over the tape position.

The Raspberry Pi and the microcontroller communicate via a serial interface, using a simple custom protocol. We use hardware based serial ports on both the Pi and Teensy. On the microcontroller side we found the 

 to cover all our needs, while the Python communication side was custom built using the 

All code and documentation can be found on our GitHub repository.

It is not only possible to use the full range of colour in the LED lights but also the illustrations printed on the scrolls can be in full colour. However, for overlaying the images on the two layers of scrolls, care needs to be taken in the design process not to use too many deep colours on the top layers, so that the bottom layer can still shine through. We advise using colours sparingly for visual effects, as described below.

From the outset, our goal was to expand the possibilities of the image world of the box through movement. We realised that there are more ways of creating an animated moving image other than physically moving the illustration. One aspect we explored was using changing Red, Green and Blue light to make elements printed in similar colours appear and disappear. We found that prior work by 

 has been done in this direction, and conducted some tests based on this principle. We could show that it is possible with the performance machine, although rapidly flashing lights are a seizure hazard and therefore care must be taken when using them in an uncontrollable environment. However, we also found that this technique can be used for more gentle animations, as described below.

As a result of the RGB animation effect tests, we realised that it may be more appealing for visual storytelling to make a coloured elements appear or disappear through changing the lighting. For instance yellow lines on an image can be made to “disappear” when illuminated in red, but appear black when the light color changes to blue. This mirrors the visual effect that can be achieved through changing between backlight and front light, without the need of an additional scroll layer. As different light colours are also used to change the “mood” of a classical theatre set, we are excited to see this possibility for expanding the abilities of our performance machine even further.

Invisible Cities: Performance Delivery & Robotic Performance Space is a project which develops a prototype of an interactive performance machine that can fit inside a pizza box. During the fellowship, Peter Lorenz and Johannes Bereiter-Payr refined its functionality, developed an open-source toolkit, and explored the "techno-dramaturgical" potential of this new performance format. Their work aims to make performance as easily accessible as ordering pizza. 

RESEARCH PLATFORM

Academy for Theatre and Digitality

multiVRse

Alexander Nham Gloria Schulz

VR (Virtual Reality)App/UI Design Narrative Design 2020
completedFellowship

Research Questions

Personnel needed

Equipment needed

Similar Projects