Universal Windows App Development with Cortana and the Speech SDK 3

3 Cortana Integration & Voice Commands

3.1 Create the Voice Command Definition (VCD) file

VCD file:

<?xml version="1.0" encoding="utf-8"?>

<!-- Be sure to use the new v1.1 namespace to utilize the new PhraseTopic feature -->
<<strong>VoiceCommands</strong> xmlns="http://schemas.microsoft.com/voicecommands/1.1">
    <!-- The CommandSet Name is used to programmatically access the CommandSet -->
    <<strong>CommandSet</strong> xml:lang="en-us" Name="englishCommands">
        <!-- The CommandPrefix provides an alternative to your full app name for invocation -->
        <<strong>CommandPrefix</strong>> MSDN </CommandPrefix>
        <!-- The CommandSet Example appears in the global help alongside your app name -->
        <<strong>Example</strong>> find 'Windows Phone Voice Commands' </Example>

        <<strong>Command</strong> Name="MSDNSearch">
            <!-- The Command example appears in the drill-down help page for your app -->
            <Example> find 'how to install CommandSets' </Example>

            <!-- ListenFor elements provide ways to say the command, including references to 
            {PhraseLists} and {PhraseTopics} as well as [optional] words -->
            <<strong>ListenFor</strong>> Search </ListenFor>
            <ListenFor> Search [for] {dictatedSearchTerms} </ListenFor>
            <ListenFor> Find <strong>{dictatedSearchTerms}</strong> </ListenFor>
            <ListenFor> Find </ListenFor>

          <!--Feedback provides the displayed and spoken text when your command is triggered -->
            <<strong>Feedback</strong>> Searching MSDN... </Feedback>

            <!-- Navigate specifies the desired page or invocation destination for the Command-->
            <<strong>Navigate</strong> Target="MainPage.xaml" />
        </Command>

        <<strong>Command</strong> Name="MSDNNaturalLanguage">
            <Example> I want to go to the Windows Phone Dev center </Example>
            <ListenFor> {naturalLanguage} </ListenFor>
            <Feedback> Starting MSDN... </Feedback>
            <Navigate Target="MainPage.xaml" />
        </Command>

        <<strong>PhraseTopic</strong> Label="dictatedSearchTerms" Scenario="Search">
            <Subject> MSDN </Subject>
        </PhraseTopic>

        <<strong>PhraseTopic</strong> Label="naturalLanguage" Scenario="Natural Language">
            <Subject> MSDN </Subject>
        </PhraseTopic>

    </CommandSet>
</VoiceCommands>

Command Prefix : Your app's name or a simplified form of the name. Users can use either to address your application. For Cortana to recognize. 

Example : top level of what a user can say

Command : can add one to many commands. Contains: what the user says to Cortana. What Cortana says in response. What action Cortana will perform.

Listen For: ways to say the command

Phrase Topic: Scenario hints to guide accuracy improvements.

Recognized in cloud, not on device.

Significantly increased speech recognition accuracy

FeedBack : Cortana 做出的回应。可自定义。

3.2 Register the VCD file on APP Startup

JavaScript

C#:

private async void RegisterVoiceCommands()
{
Uri uriVoiceCommands=new Uri("ms-appx:///vcd.xml", UriKind.Absolute):

StorageFile file=await StorageFile.GetFileFromApplicationUriAsync(uriVoiceCommands);

await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(file);

}

3.3 Handle Voice Command Activation

//Windows Runtime App on Windows Phone 8.1, inside OnActivated override in App Class

protected override void OnActivated(IActivatedEventArgs args)
{
    // Was the app activated by a voice command?
    if (args.Kind == Windows.ApplicationModel.Activation.ActivationKind.VoiceCommand)
    {
        var commandArgs = args as Windows.ApplicationModel.Activation.VoiceCommandActivatedEventArgs;
        Windows.Media.SpeechRecognition.SpeechRecognitionResult speechRecognitionResult = commandArgs.Result;

        // If so, get the name of the voice command, the actual text spoken, and the value of Command/Navigate@Target.
        string voiceCommandName = speechRecognitionResult.RulePath[0];
        string textSpoken = speechRecognitionResult.Text;
        string navigationTarget = speechRecognitionResult.SemanticInterpretation.Properties["NavigationTarget"][0];

        switch (voiceCommandName)
        {
            case "showWidgets":
                if (textSpoken.Contains("today's specials"))
                {
                    // Code to show today's specials.
                    // To do this, refactor OnLaunched into a method that you can call from both OnLaunched and OnActivated.
                    // The navigation logic will resemble this: rootFrame.Navigate(typeof(<today' specials page class>), e.Arguments);

                }
                else if (textSpoken.Contains("best sellers"))
                {
                    // Code to show the best sellers.
                }
                break;

            // Cases for other voice commands.

            default:
                // There is no match for the voice command name.
                break;
        }

    }
}

if(args.kind == ActivationKind.VoiceCommand)
{
VoiceCommandActivatedEventArgs vcArgs= (VoiceCommandActivatedEventArgs) args;

string voiceCommandName=vcArgs.Result.RulePath.First();//what command lauched the app

switch(voiceCoomandName) //navigate to right page for the voice command

{
case "FindText" :
         rootFrame.Navigate(typeof(MSDN.FindText), vcArgs.Result);
         break;

case "nlpCommand":
          rootFrame.Navigate(typeof(MSDN.NlpCommand), vcArgs.Result);
          break;
}
}

// windows Runtime App on Windows Phone 8.1 , inside OnNavigatedTo in FindText.xaml.cs

protected override void OnNavigatedTo( NavigationEventArgs e)
{
//get recignition result from parameter passed in frame.Navigate call

SpeechRecognitionResult vcResult=e.Parameter as SpeechRecognitionResult;

if (vcResult!=null)
{

string recoText=vcResult.Text;

//store the semantic dictionary for later use
IReadOnlyDictionary<string,IReadOnlyList<string>> sematics=vdResult.SemanticInterpretaion.Properties;

string voiceCommandName=vdResult.RulePath.First();

if( voiceCommandName== ""FindText")
{
	//what did the user say, for named phrase topic or list "slots"// users add some search terms
	if (semantics.ContainsKey("dictatedSearchTerms"))
{
	HandleFindTextWithSearchTerms (semantics["dictatedSearchTerms"][0]);
}
	else
{
	HandleNoSearchTerms();
}
}
//else handle other voice commands
}
navigationHelper.OnNavigatedTo(e);
}

//Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo in NlpCommand.xaml.cs

// free format
protected override void onNavigatedTo (NavigationEventArgs e)
{

base.OnNavigatedTo(e);

//get recognition result from parameter passed in frame.Navigate call
SpeechRecognitionResult vcResult=e.Parameter as SpeechRecognitionResult;

if (vcResult!=null)
{
	string commandMode=vcResult.SemanticInterpretation.Properties["commandMode"][0];
	
	if(commandMode=="voice")
{
		SpeakText(audioPlayer, String.Format("MSDN app heard you say {0}", vcResult.Text));
		HandleNlpCommand(vcResult);
}
else if (commandMode="text")
{
	messageTextBox.Text=string.Format("working on your request \"{0}\"", vcResult.Text);
	HandleNlpCommand(vcResult);
}
}

phrase list :help narrow down the command

3.4 Example Using JavaScript

Command Monkey

website:   http://codefoster.com/commandMonkey

Cortana Voice Command  ----->   Phone -----------> Service ---------> Device

3.5 Localize the App for Relevant Languages

create different command sets

xml:lang="en-us"     

xml:lang="fr-fr"