Immersive Research & Practice – Medical, Clinical and Mental Health Applications

This blog will now follow academic/industry research and detail practical experiments that explore the design, development and production of immersive XR (/AR/VR/MR) projects.

I am interested in how screen-based, site-specific installation, geolocated or device assisted experiences can be designed and developed to support the Medical, Clinical and Mental Health fields.

POD
VR App Prototype

To begin this new route, I will re-examine POD, an immersive clinical training application on which I began development a year ago.

I will conduct academic/industry and deploy practice research tools to engage with discourse on how virtual reality can be effective for clinical instruction.

Next Steps:

Review POD Development
Conduct Literature Sweep (2015 – 2020) XR (VR/AR/MR) Medical applications

 

Lunarium Review :

I reviewed the Lunarium project and its progress so far.

I have decided that Phase 3 of the Lunarium development will continue on my commercial blog, sarrahornby.blogspot.com as it becomes more commercially focused because this will link more closely with the marketing and distribution network. The development project boards will continue to be updated into phase 3 and will still contain all development and links to relevant aspects of the project (so will still be of interest to new indie developers out there!)

Lunarium 2 : Trello Development Board

Lunarium 2 : New Blog 

Mixamo Blocker 2019.4

Although the violin playing skeleton animation is still working, I’m having difficulty importing my fuse model through Mixamo.

I’ve tried several approaches which includes importing in a T-Pose without animations and the. Creating an animator with existing and imported Mixamo animations.. to no avail.

I’ve had a dig around and occasionally Mixamo glitches in unity so I’m going to find a fix somehow. I’ve created a new RingMaster character and auto rigged it without facial and finger rigging to try to match the bones with unities auto rig detection.

Once rigged in unity, the animation should work okay.

This is how it all connects:

Unity Animation and Cinematics: Experiments

I am working on th next phase of development for Lunarium. I want to set up Lunarium as a Rift VR experience and test this on my quest via link so that after christmas I can start to test for quest. 

To experiment with getting more narrative engagement and player interaction, I have decided to further develop the Top Tent animation scene. 

To do this, I will need to include multiple animated characters and will include a ringmaster, alienated death and flying bats and add further player interaction. 

The object of this scene is to watch a cut scene and then the player must solve a puzzle to get past the alienated death character and jump into the large hole beneath. 


I need to: 

1) Create a bat object with wing animation 
2) Set looped motion to move the bat around the toptent ceiling.
3) Create new RingMaster character with walk cycle, idle and turning options. 
4) Animate the Alienated Death Character so that when it breaths in (The body inhales) the air is forced out of one of the 3 tubes which flipps the eye and plays a note. 
5) The player must play the notes in the correct order, to close the eyes and enter the hole beneath the alienated death character. 


I am following a Unity Tutorial which covers all aspects of animation, cinemachine and timeline. 



Coding Motion:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;


public class Mover : Monobehaviour
{
private Transform ThisTransform = null;
public float Speed = 1f;
public Animationcurve SplineCurve;

//Start is called before first frame update
voide Awake()
{
ThisTransform = GetComponent<Transform>();
}

//Update is called once per frame
void Update()
{

ThisTransform.position += Speed * ThisTransform.forward * Time.deltaTime;
float Y = SplineCurve.Evaluate(Mathf.PingPong(Time.time,1f));
ThisTransform.position =  new Vector3(ThisTransform.position.x,
Y,ThisTransform.position.z);
}
}

This animates and object in the direction it is looking with variable speed and motion curves. 
Y axis will bounce the object whereas z axis will slow down and speed up the object.  

Editor Motion:
Animation Window

Set simple animation in timeline using keyframes and curves it is possible to set looped motion.

I can also connect motion with an animation event using the animation tab in the timeline. The following is a short peice of code which is set at the point where the character jumps off the ground.. they shout ‘Yippeee’!

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class PrintMessage : MonoBehaviour
{
    public void SayMessage()
    {
        Debug.Log(“Yipeeee!”);
    }
}

Next I created a character model in fuse ‘RingMaster3’ saved to CC and then exported to Mixamo.

Mixamo autorigs the character and offers two additional rigging options. Firstly it offers facial rigging and secondly, finger rigging. I chose both options as I am trying to create better character-player interactions as well as adding more hand interaction in quest later on.

The options to export from Mixamo are :

Download: Maya and 3ds max as FBX files, Blender DAE or Modo FBX,

Animate: Game Engines Unity, Unreal and others. This is where Mixamo can save time in animating walk cycles etc. 

I chose to:

Download FBX for Unity saved to my Google Drive file and CC as a rigged T Pose.

Download DAE to further edit in Blender

Animate multiple animations in mixamo which includes walk and idle animations.

To Do:
  
1) Create a bat object with wing animation 

2) Set looped motion to move the bat around the toptent ceiling.

Done – Create new RingMaster character with walk cycle, idle and turning options. 

4) Animate the Alienated Death Character so that when it breaths in (The body inhales) the air is forced out of one of the 3 tubes which flipps the eye and plays a note. 

5) The player must play the notes in the correct order, to close the yes and be able to enter the hole beneath the alienated death character. 

Fledgling project : H

In between testing the affordances and constraints of the new oculus quest, I have been working on a new fledgling project within a small team.

As I come to the end of the first sprint, I begin to think about the sprint review, recommendations and the next sprint (If required). One particular standout experience is how the waterfall approach can be embedded within an agile methodology. Working with others where both approaches are natural parts of others working process means that some might be used to the agile process and others the waterfall. I have adapted by meeting key milestones as if they are waterfall deliverables while adopting a flexible approach.

Oculus Quest : Quill and Unity Production.

I connected Quest to my PC and downloaded Quill to test that I am able to design in the app and I am happy to say that it works pretty well. At least it worked once disabling my local firewall (I re-enabled it afterwards!)
I checked out various short animated films and illustrations in the Oculus Theatre and this prompted me to think about how I might use quill in Lunarium .. and later my next project Fizhogg which will lend itself to this illustrated style. 
Thinking about how to work between quill and unity I understand that I can export work from quill as an .fbx File while also importing .fbx models into quill. I’m not sure how to use it yet but it could be a really interesting process to experiment with. 

Oculus Quest Games : Quill

Acquiring an Oculus Quest and link to PCVR rift games Not only means that I can learn more about HD VR for PC but also that I can experiment with illustrated VR Animation using Quill.

To begin with I went through the Quest tutorial app which demonstrated the headsets capabilities. This was really useful in that, while playing, I could think about which new interactions I could develop in Lunarium. 

To begin with I could develop one of the rooms within the Labyrinth as an escape room experience. This would allow for more hand interaction. 

Hello… Oculus Quest :)

New Oculus Quest (with Link for rift) 
I received an Oculus Quest and have updated to Unity 2019.4 LTS which means that I can now develop for oculus Quest. This brings with it new modes of interaction, including many hand control options. This also means that I can develop my experiences for mobile and pcvr (for rift) which is where I was trying to get to but was limited by access to tech and software.
Not any more.. 
As I am registered as an oculus developer I now understand that I can continue to develop Lunarium for rift until Christmas when I should be able to launch on Oculus Quest for testing. 
This is the plan…

VR Narrative: Cinematic VR Films

Cinematic VR Films 

Invasion (Baobab)

Asteroids (Baobab)

Baba Yaga (Baobab)
Coming Soon…

Rainbow Crow (Baobab)

Bonfire (Baobab)

Jack (Baobab)

Coco Film 

Coco VR Experience (oculus)
The Line
The Key 
Gloomy eyes

Dear Angelica
Pearl
Cycles (Disney)

Myth : A frozen tale 

A kites tale (Disney)

Melita (AI)
Gymnasia
Wolves in the wall (Fable Studio)
Drop in the Ocean

Age of Sail

Bounty Maid

Jack of All Trades

The Dream Collector
Battlescar
Nothing Happens
The Last Goodbye
Zero days
Hue

Read – Around …

Baobab Studios
https://www.oculus.com/blog/vr-visionaries-baobab-studios/?locale=en_GB

Oculus  : Exploring the VR landscape
https://creator.oculus.com/blog/oculus-at-sundance/

Oculus : Quill (rift)
https://www.oculus.com/blog/quill-launches-vr-animation-and-storytelling-tools/?locale=en_GB

CartoonBrew: Quill
https://www.cartoonbrew.com/how-to/how-to-make-a-short-film-using-quill-an-animators-guide-to-the-vr-software-190901.html

Cartoon Brew: VR
https://www.cartoonbrew.com/vr

Cartoon Brew: Annecy VR
https://www.cartoonbrew.com/festivals/annecy-preview-6-highlights-from-this-years-virtual-program-193087.html

Unity
https://unity.com/solutions/cinematicvr

Mechanical Souls (Thriller VR & Immersive Theatre)
http://digital-rise.com/portfolio/mechanical-souls-training-session/

Caminades VR Demo: Caminades VR Demo animation
Other Worldly Theme Ride: Other Worldly Theme Ride VR animation

And finally…

Sing along now… 🙂