Month: July 2016

Bots, Unity, XR

TL;DR

It works! I managed to get HoloLens inputs working with the LUIS integration I did before in Unity. The project melds phrase recognition with dictation from HoloAcademy’s github example and then pings the LUIS API.

All the code is here: Github/KatVHarris

LUIS + UNITY + JSON post is here

LUIS 101 post is here

Okay so this is a very short post. Most of the harder parts were completed before this in my LUIS post and the Hololen’s Academy code helped a lot. I’ll just mention here some of the pains I went through and how it works.

Phrase Recognition vs. Dictation

HoloLens has 3 main types of input control for users. Gaze, Touch, and Voice. This application focuses on the last one. The voice input uses the Speech library for Windows.

using UnityEngine.Windows.Speech;

This library allows the HoloLens to then use Phrase Recognition to trigger specific actions or commands in your project. Yet that defeats the point of natural Language processing. In order to then interact with LUIS we needed to feed in what the user is saying. So to do this, I integrated the Communicator Class from the HoloLens Example into my project. This class handles the Phrase Recognition of the project but it also handles dictation, enabling natural language to be captured from the user. My Communicator is slightly different than the HoloLens because of the LUIS interactions as well as reworking the code to call for multiple dictation requests.

Now to activate the dictation Phrase Recognition commands are used. So no need to Tap to activate.

Dev Notes

I did have some speech/phrase recognition trouble. The original phrase to activate dictation was “Allie” (the character on the 100 which my original bot project is based off); however, the recognizer doesn’t recognize that spelling of her name. Changing it to “ally” caused the recognizer to trigger. DictationRecognizer is similar to the PhraseRecognizer in that it also doesn’t recognize the spelling of many names; for example, I would say “Tell me about Clarke.” and the dictation recognizer would write “tell me about clark.”. To fix the dictation errors I used Regex to replace the spelling before querying the LUIS API. One can also change their LUIS API to accept the speech recognition spelling but because multiple bots and applications are connected to my LUIS API I couldn’t implement that solution.

private void DictationRecognizer_DictationResult(string text, ConfidenceLevel confidence)
    {
        // Check to see if dictation is spelling the names correctly 
        text = Checknames(text);

        // 3.a: Append textSoFar with latest text
        textSoFar.Append(text + ". ");

        // 3.a: Set DictationDisplay text to be textSoFar
        DictationDisplay.text = textSoFar.ToString();
    }

Anyway that’s all there is to it. All the major code is in the:

Hope this helps, and let me know if you have any questions with it on Twitter @KatVHarris

Happy Hacking

– TheNappingKat

Bots

So for the past month I’ve tried to push myself to code everyday, and git something on Github… See what I did there 😉

Why did I start –

This month has been really chaotic. I was learning about the new Microsoft Bot Framework, I was in an AD, I was working almost every weekend for events and hackathons, and I was also moving. Suffice to say, not pushing something EVERY day would have been fine. However, 5 days into the month I realized my week commit streak was all green. My record in the past had been 10 days, and I thought wow this is a perfect time to break it.

Milestones –

I decided to start small. Aim for a week, then 10 days to tie my record, then 15 days, then 20, then 25, and 30 at the end of June. See, if I, in this crazy month of upheaval in my personal life as well as having one of the busiest work months, could put aside time every day to code my challenge goal would be achieved.

In the first 5 days I was working on Unity Intro project and my Bot Framework Example, and decided that focusing on just those would be best for narrowing the scope as well.

Day 13 – Starting to Waver // “Cheating”

Like I said I was moving and doing a bunch of events and all of a sudden working on code seemed too much of a commitment, but I desperately wanted to continue my streak. The solution, updating the Readme. It’s a simple solution and I felt really guilty about it. How can that count as working on code?

Well on Day 18, the weekend of Holohacks, a weekend long hackathon with the Hololens team at Microsoft in SF, I got to talking to a bunch of developers. Turns out that the team was “strongly encouraged” (lol pretty much forced) to write all their documentation as they were working and make it high quality. Documentation is such an important part of development especially for open source projects where others will want to add to your work.

Documentation Driven Development (DDD)

Now, those one off commits to the Readme didn’t seem like cheating. I was spending time improving a document that was the guideline for my work. Updating links to point to newly implemented documentation, adding to the feature list and directions, creating templates for other devs were all justified.

I didn’t come up with the phrase DDD, but I believe that’s how many projects should be worked on. Why? Well normally when developers write an amazing piece of code, writing documentation about it is the worst, most boring part. Decent documentation is a god send when most of the time it seems to be out of date in our ever-evolving industry. Imagine trying to build Ikea furniture with outdated instructions. Sure you can figure it out, and hopefully it stays together, but having an accurate guide makes life so much easier.

Day 30

After that hackathon I packed, moved to LA, did another Hololens weekend hackathon, and had an important presentation the following week. However, even with all that I managed to push many updates to my code and write up important documentation for it as well. Day 30 hit and my goal now is to see how long I can keep it up. It’s become more of a habit now. Kind of like working out or brushing your teeth.

Just thought I’d share some of the life updates with everyone.

Happy Hacking

-TheNappingKat