Skip to main content

Degree Project - Week 29

 

Semester B: Week 29
Student: Tai Ser Yeet (22064351)
Programme: BA (Hons)(SW) Digital Media Design


WEEK 29

Adding Sound

I wanted to get some audio going for the dance installation, so I did some digging and found a tutorial (linked below) that showed me how to actually handle sound inside TouchDesigner. Steve and I got the hardware side sorted by hooking the laptop up to the amplifier with an aux cable, and we finally got the sound playing through the speakers. Right now, it’s just playing through YouTube Music, and not really part of the system yet. 

My next big task is to figure out how to route that music directly into TouchDesigner so I can control the tracks through my movement, making the sound just as reactive as the fluid visuals.

Parameters Settings for Audio Dev In CHOP

Parameters Settings for Audio Dev Out CHOP


Tutorial With Mafalda

  • Reduce the sensitivity of the body tracking
  • Get the fluid to stick to your body more
  • Play around with the parameters of the Kinect CHOP using Google Gemini (e.g. get it to generate the code) 
Selected Parameters To Experiment
  • Circle back to what kind of visuals complement your dancing
  • Have two plans: 1. Front Projection Setup: Current setup with the light shining on me dancing at the back
  • 2. Rear Projection Setup: If you can't get the lighting to work, then dance in front of the screen. The Kinect will be in front of the audience, propped on top of a stand. The projector will be behind the screen. The visuals will complement my dancing.
  • The purple fluid is better than the blue and orange ones
  • change the colour of the fluid simulation 
  • Try particles as background, so overlay that with the fluid simulation
  • You can connect two Kinects to the same TD file (maybe the Kinect TOP and Kinect CHOP operator and then composite them together)
  • The black background is plain
  • Submit TD file, a promotional video (get friends to come and watch, then have someone film it from both sides)
Tutorial With Steve
  • Get another projector to illuminate from the back (the side where I will be dancing)

After Class

    Link to the sample video: Click Here



    The VGA to Type C cable is too short, and the short-throw projector does not support Bluetooth connection. Wireless connection requires a YW-40 dongle, which is nowhere to be found.

    22/4/26

    On the Vectors page of your Splat GLSL TOP:

    Add a uniform: uSplatBias.

    Set it to a very small number like 0.01 or 0.05.

    Why? Because this shader runs 60 times a second. If you set it to 1.0, the silhouette will turn solid, blinding white almost instantly. A small value makes the fluid "accumulate" or "bleed" off your body gracefully as you move.

    Makes it more opaque

    23/4/26

    uSplatBias - increase the value to make the subject brighter
    uVorticity - 1.0 or less makes the outline of the subject less jagged
    BlurTOP - reduce outline of silhouette and makes it more fluid

    To get the right weight for the performance, I had to stop treating the fluid like a visual effect and start treating it like a physical substance. By tweaking the damping and opacity in the feedback loops, I managed to create a sense of drag where the fluid doesn't just follow me, it feels like it's being pulled through the air by the force of my movement.

    I want it to spread some more, and Gemini suggested adding a Transform TOP and playing around with the scale. It is getting there.


    Here are videos of all the attempts I did, from full body fluid to a combination of lines and pre-recorded video.




    Using the particle lines network, I combined the data from the Kinect player index (live movement) and data from the pre-recorded dance video using a composite TOP to create an illusion that I am dancing with another person. To differentiate between the two, I changed the colour of the pre-recorded dancer from white to yellow.

    I tried crossfading two angles of a dance video; medium shot and a close-up shot, then compositing with the live visuals. It turned out really well, but the live dancer did not go well with the pre-recorded one because they were not interacting with each other. It would be better if the live movement controlled the transition of the pre-recorded one.

    I then explored the intersection of human gesture and cinematic timing by using Kinect skeletal data to drive a generative video sequence (the two videos earlier). By routing specific coordinate channels, specifically the hands and spine, through a custom logic network, I was switching between close-up and long-shot perspectives. I discovered that big movements weren't effective in conducting the crossfade transition, but the most successful and intentional interaction was from a slow clap. By clamping the hands together upon impact, I was able to create a singular, high-velocity peak in the data while simultaneously preventing the "jitter" that often happens with depth sensors when limbs overlap.
    Using Kinect Data to Drive Crossfade Transition

    I wanted to replace the black background with something that is moving. I stumbled upon this tutorial teaching viewers how to create flowing brush streaks in TouchDesigner.

    I took the final network and plugged that into the composite TOP with the other networks for the line visuals.
    Flowing Brush Streaks Network

    Snapshot of the Final Outcome

    At this point, my laptop was starting to overheat and the preview in Touch Designer was becoming laggy. I need to open this file using the i9 machines.
    Snapshot of the Overall Network

    This was how it turned out.






    Labels: