I’m taking a little detour from my usual Unity Gaming Tutorial Series, to give you the top 3 most important changes/updates (in my opinion) that have happened for the Universal Windows Platform (UWP).
Relative Panel
Split View
State Triggers
TL;DR: Code is all on my GitHub, here http://bit.ly/UWP-UI-github
Relative Panel
Now this is a super cool new feature that really comes in handy with Microsoft’s new UWP style of development. What do I mean by this? UWP allows for one build to run immediately on phone and desktop! But by doing this developers have to consider adaptive UI that can change for different views/devices (phones, Xbox, desktops, tablets, HoloLens).
So what is Relative Panel. Well it’s “a style of layout that is defined by the relationships between its child elements” – (read more here). When implementing this style of layout, normally there will be a child object acting as an anchor for the other children derive their location.
Getting Started
So I’m using the new Visual Studio 2015 RC. Interesting fact: You Do NOT need Windows 10 to build for Windows 10!
When you open Visual Studio you should see this interface.
If you are familiar with Visual Studio not much has changed.
Create a new UWP project.
A new project should be generated.
If you look at the Solution Explorer you should notice that there is no longer a shared folder, or two separate projects in the same solution for phone and store apps, like in 8.1.
Relative Panel Dev
Now I’m going to show you how Relative Panels work. Double click the Main.xaml file if it’s not already open.
I’ve circle two important things to notice when designing your app. The orientation and the screen that you are dev-ing on at the moment. Also notice the Grid tag? This is where most of our XAML code will be written. Let’s continue.
Third: The following squares will now be placed in the Panel relative to that initial Square. Let’s also make sure that we can still view everything though when the width changes. We can do this with adaptive triggers, which I’ll talk about later, but for now let’s add a scroller.
Now this is where the new Universal Windows Platform really shines!
Click the run on local machine.
The code runs in a Windows 10 Window much like we would suspect.
LOCAL MACHINE:
Now, click the arrow next to local machine. You should see a list of emulators you can build on.
Select one of the phones. **Caveat – To have this work you need to download the mobile SDK if you didn’t when you downloaded Visual Studio. **
Now run again. The emulator will start and run the app!
EMULATOR:
WHAT!? No changes in code or anything! I’m sorry. I’m just really excited about this, and if you do cross platform development you probably share my enthusiasm.
So most of this code is used from the SDK, however I will be going into a detailed explanation about how it all works. As well as how to put it into Unity.
First things First. Create a new Folder inside your Scripts folder Called KinectScripts. Now in that folder create two new scripts; one called BodyManger and the other called BodyView.
BodyManager Script
First we need a BodyManager script. This object will Mange the Kinect sensor connection and read in all the body data coming from the Kinect. Import Kinect Library with:
using Windows.Kinect;
Then we need some fields for the manager to store and the data and get sensor data.
Now that the connection is open and reading in the data we need to store it in the Body array. We will do this every frame of the game, therefore we need to edit the Update() method.
First we check to see if the _Reader has been established and the connection has been completed. If it has we will take the last frame, the reader read in and if it’s not null, we can then check to see if the data is there.
void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
if (_Data == null)
{
}
}
}
}
We still need to get the Body data from the Senor. To do this we will need to create a new Body array with data from the Sensor.BodyFrameSource.BodyCount.
At the end the method should look like this:
void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
if (_Data == null)
{
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}
}
}
}
Then we need to refresh the stream of data from the Reader. By adding the following code to manipulate the frame.
void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
if (_Data == null)
{
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}
frame.GetAndRefreshBodyData(_Data);
frame.Dispose();
frame = null;
}
}
}
The last method in the Body Manager Class is OnApplicationQuit(), which Disposes the Reader, and closes the Sensor stream, sets it to null.
void OnApplicationQuit()
{
if (_Reader != null)
{
_Reader.Dispose();
_Reader = null;
}
if (_Sensor != null)
{
if (_Sensor.IsOpen)
{
_Sensor.Close();
}
_Sensor = null;
}
}
Now onto drawing the skeleton in the scene.
BodyView Script
The next Script to write is one to draw the skeletal structure. We won’t necessarily need to see the skeleton for the game, however, I’ll show you how to show skeletal body tracking. We also need the skeletal data to track the hands, whose state will dictate controller commands.
For this MonoBehavoir class we will need, a material to draw the bone in the Unity scene. A gameobject to store the BodyManger, to control the stream of the Kinect.
public Material BoneMaterial;
public GameObject BodyManager;
We also need a BodyManager object and a Dictionary to store the bodies being tracked.
Now in the Unity Update() method we need to check to see if the Body Manager is not null and that it has data.
void Update()
{
int state = 0;
if (BodyManager == null)
{
return;
}
_BodyManager = BodyManager.GetComponent<BodyManager>();
if (_BodyManager == null)
{
return;
}
Kinect.Body[] data = _BodyManager.GetData();
if (data == null)
{
return;
}
}
Next, while still in the Update() method, we need to get the amount of bodies in the list of tracked bodies. Then delete unknown bodies.
List<ulong> trackedIds = new List<ulong>();
foreach (var body in data)
{
if (body == null)
{
continue;
}
if (body.IsTracked)
{
trackedIds.Add(body.TrackingId);
}
}
List<ulong> knownIds = new List<ulong>(_Bodies.Keys);
// First delete untracked bodies
foreach (ulong trackingId in knownIds)
{
if (!trackedIds.Contains(trackingId))
{
Destroy(_Bodies[trackingId]);
_Bodies.Remove(trackingId);
}
}
Now that we have the keys for tracking the bodies we need to create a body object with that tracking ID key. We need to write two more methods. A CreateBodyObject() method that will take a ulong id and a RefreashBodyObject() method that will take a Kinect.Body object and a GameObject for the body. We will use these methods after we go through the data, and find if bodies inside are being tracked or not. If it is tracked but doesn’t have a TrackingId, then we need to create a body with that TrackingID. If it is being tracked and has a TrackingId then we just need to refresh the drawn body.
foreach (var body in data)
{
if (body == null)
{
continue;
}
if (body.IsTracked)
{
if (!_Bodies.ContainsKey(body.TrackingId))
{
_Bodies[body.TrackingId] = CreateBodyObject(body.TrackingId);
}
RefreshBodyObject(body, _Bodies[body.TrackingId]);
}
}
}
CreateBodyObject()
The CreateBodyObject takes an ID and returns a body gameobject. So we first need to create a gameobject that will store the appropriate data retrieved; then we need a for loop to go through every joint to draw the body.
private GameObject CreateBodyObject(ulong id)
{
GameObject body = new GameObject("Body:" + id);
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
{
}
return body;
}
For every joint in the body we create a cube and add a lineRenderer to that cube. The cube will be drawn at each joint while the line renderer will be drawn to connect the joints.
private GameObject CreateBodyObject(ulong id)
{
GameObject body = new GameObject("Body:" + id);
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
{
GameObject jointObj = GameObject.CreatePrimitive(PrimitiveType.Cube);
LineRenderer lr = jointObj.AddComponent<LineRenderer>();
lr.SetVertexCount(2);
lr.material = BoneMaterial;
lr.SetWidth(0.05f, 0.05f);
jointObj.transform.localScale = new Vector3(0.3f, 0.3f, 0.3f);
jointObj.name = jt.ToString();
jointObj.transform.parent = body.transform;
}
return body;
}
RefreashBodyObject()
Now to write the ResfreshBodyObject method. In this method we need to go through each joint type possible just like we did in the CreateBodyObject method. But this time we are passing in the current body, as well as the appropriate tracking number so we don’t draw the bones for the wrong person.
private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
{
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
{
}
}
Inside this for loop we need to get the key value pairs we made before in the bone loop for each joint.
We also need to update the skeletons position so it’s in the accurate place on the screen. To do this we need to write a method to get the Vetcor3 from the sourceJoint.
Next step in the for-loop is to get the linerenderer from the bodyObject, which was the cube we create for each joint. Then we need to see if target joint has a value. If it does we can then draw a line from the original joint to the target.
Great! So we are almost done with drawing the skeleton. There is a bit more information that will be helpful that the SDK gives you, which is tracking status. There are three states to choose from, Tracked, Inferred, or NotTracked. We can have the line renderer show us the state of tracking by changing it’s color. To do this we need a method that will return a color based on the current state.
private static Color GetColorForState(Kinect.TrackingState state)
{
switch (state)
{
case Kinect.TrackingState.Tracked:
return Color.green;
case Kinect.TrackingState.Inferred:
return Color.red;
default:
return Color.black;
}
}
Now we add one more line to the for-loop of the RefreachBodyObject method and we are done.
Now in Unity I’ve created another scene by going to file new scene (it will prompt you to save your current scene if you haven’t already). This empty scene will make it easier for you to test and see what you’re doing.
In the empty scene, which I have saved as kinectTest, create two empty game objects. Call them Body Manager and Body View.
Attach the Body Manager script to the body manager object. Attach the Body View script to the BodyView object.
Select the Body View object. In the inspector you need to fill in the Body Manager slot and Bone Material slot. Click and drag the BodyManager from the hierarchy to the slot. Then in your materials folder click and drag the Bone Material into the bone material slot.
Now hit run. You should see nothing on your screen at first. This is because the Kinect hasn’t registered your skeleton yet. Stand back about 4 ft, and you should see the skeleton appear.
If you look closely you can tell that the legs of the skeleton are red. This is because the Kinect is interpreting that’s where my legs should be. My desk was blocking the sensor from detecting where my legs actually were.
But wait! You say. We don’t need a skeleton in our infinite runner game, just the hand gestures and states! So was all this a waste? NO! Of course it wasn’t. We just don’t need to draw the whole skeleton in the scene anymore. But we now have the know-how if you need to in the future. YAY!
The next part of this mini series will be getting those hand gestures to work in the game!
Okay so in part 1 I showed you how to get the kinect working on your computer. But how do we get it in a Unity Project you ask? Oh, you didn’t, well I’ll tell you anyway. =)
To get started we will use the steps directly outlined in the Kinect for Windows Unity Package from Microsoft; but slightly edited since we already have a project created.
Expand the .Zip file, and move Kinect.2.0.1410.19000.UnityPackageto a well known <location>
Open UnityPro (you need to have a Pro edition to pull in custom packages and plugins)
Open your Unity Project
Click on the menu item Assets->Import Package->Custom Package…
Navigate to the <location> from step 1
Select the Kinect.2.0.1410.19000.UnityPackage
Click “Open”
Click “Import” in the lower right hand corner of the “Importing Package” Dialog (which Unity will launch after step 7)**Before you do step 8 here is an important thing to note – When Importing notice that the folder is called StandardAssets. This is the same name as the Sample Assets from the Unity Store. If you are using Unity’s Standard Assets package the import manager will embed the new Kinect files into the already existing folder. Be careful! If you don’t keep track of what you’re importing you might lose files within the numerous folders of your project. So, to keep things organized keep note of the files that are not in subfolders and are just in the Standard Assets folder. In this case there are 10:
EventPump.cs
KinectBuffer.cs
KinectSpecialCases.cs
CameraIntrinsics Import Settings
CollectionMap Import Settings
ExecptionHelper Import Settings
INativeWrapper Import Settings
NativeObject Import Settings
SmartGCHandle Import Settings
ThreadSafeDictionary Import Settings
Okay now back to the steps**
If you wish to see the Kinect in action there are two sample scenes available from the zip.
If you wish to use VisualGestureBuilder within Unity, repeat steps 1 through 8 with Kinect.VisualGestureBuilder.2.0.1410.19000.unitypackage
If you wish to use the Face functionality within Unity, repeat steps 1 through 8 with Kinect.Face.2.0.1410.19000.unitypackage
Okay lets stay organized. Create a new Folder in your Assets Folder called KinectPackage. Then, add the 10 files I mentioned above as well as Windows Folder from StandardAssets.
And that’s it for part 2. In part 3 I’ll show you how to track your joints and start scripting as well as get a skeleton in your project! For the Infinite Runner Game we will use the kinect to act as a controller!
Okkie Dokkie. So the next few posts will be about how to get a kinect working in Unity! This is will eventually be integrated into the Infinite Runner game that the Unity Gaming series is about. However this will be a mini series inside of that since the process is kinda confusing, especially if it’s your first time using the Kinect. So let’s get started!
Software Setup
Now before you start just plugging in any Kinect from random Xbox One or Xbox 360, you need the appropriate software so your computer knows what you’re plugging into it. You also can’t use any rando Kinect. This tutorial uses the Kinect v2; that’s the one that is shipped with the Xbox One. However, in order to use the v2 with your computer you need to make sure you have the Windows adapter!
You can also go to the Kinect for Windows main site and go to their Technical documentation and tools. On the Technical Documentation and Tools page they have a list of useful links for documentation and essential downloads, including the Unity Plugin that we will also need.
After you download the SDK, run it. You will be prompted with a Kinect for Windows Setup wizard.
After you install the SDK you should have some new programs installed on your computer.
SDK Browser
Kinect Studio v 2.0
Kinect Gesture Builder
The SDK Browser will be the most useful of the new software, because it contains links to all the other programs/software as well as demos and example code you can use.
Hardware Setup
Setting up the hardware is pretty straight forward, which is great!
**Mac users** you will not be able to work with the Kinect unless you have Bootcamp installed (or some other way to partition your hard drive) with a windows OS, sorry.
Once you Plug in the Kinect to the USB3 port on your computer we can test the connection with the Kinect SDK applications that were automatically downloaded.
Open the SDK Browser for Kinect. The first program in the list should be the Kinect Configuration Verifier. Run it. The Kinect’s light should now turn on if it’s connected properly and the Verifier will open a window, if everything is correct, it should look like this:
If something is wrong you will get a red X identifying the error:
Even though I have a warning about my USB port, my Kinect still runs. Now to see the information collected by the Kinect we will run Kinect Studio. It will be the third item in the SDK browser; or you can open it straight from your applications.
Kinect Studio looks like this:
Don’t worry if you don’t see anything. At startup. Although your Kinect is on and streaming data to your computer Kinect Studio isn’t reading in that data yet.
You need to hit the plug icon at the top left hand corner so you can see anything.
There are 7 streams of data coming from the Kinect:
Body Frame
Body index
Calibration Data
Depth
IR
Title Audio
Uncompressed Color
You can toggle what streams of information you want to receive by deselecting the check boxes on the left. The most important streams of data are the Body Frame/Body Index, Depth, and Infra Red (IR).
You can close Kinect Studio we don’t need it anymore. And we are done with Part 1! There are a bunch of other cool examples you can take a look at in the Kinect SDK Browser. Part 2 coming soon.
