Microsoft Cognitive Service for Text Analysis vs Google Natural Language Service

August 26, 2017

Tags: AI  azure  Cognitive Services  Google Cloud  Machine Learning  Text Analytics 

Start up

I am trying to evaluate a number of service offering in the context of natural language processing, my starting point here is to compare Microsoft Text Analytics service with its Google counterpart (Google Natural Language service). This post is the first of a series that would cover natural language processing from multiple cloud providers.

Microsoft Cognitive Service & Text Analytics

Microsoft Azure offers a number pre-baked services that could help developers and organisation with anything related to recognising images, classifying, or analysing text. The one we are interested in here is the Azure Text Analytics service, which is currently in Preview. This service offer the following apis:

A full list of the apis and an interactive demo can be found on the MSDN website here

Google Natural Language

Google Natural Language service is very similar to Microsoft Azure’s offering, but seems to be more mature. The list of apis includes:

Implementation

I am trying to evaluate the services using varying data sources and volumes, so I thought of building one Api that would call both services to analyze the data and give me back the summary of responses from all services. Thus, I created a simple web api app that takes text in any format, send it over to Google and Microsoft services, then display the results.

For Microsoft Azure service, there does not seem to be any official Nuget package, so I ended up calling the RESTful apis directly. It’s quite simple, and a sample code can be seen below:

// Microsoft implementation. The payload (dto) objects were implemented as per the docs here: https://westus.dev.cognitive.microsoft.com/docs/services/TextAnalytics.V2.0/operations/56f30ceeeda5650db055a3c7
    using (var client = new HttpClient())
    {
        var payload = new MicrosoftSentimentAnalysisPayload(new List<MicrosoftSentimetAnalysisDocument> { new MicrosoftSentimetAnalysisDocument(request.Input) });
        var serialisedPayload = JsonConvert.SerializeObject(payload);
                
        // sentiment analysis
        var response = await PostHttpRequestToMsServiceAndReturnResult(request, client, MicrosoftCognitiveServicesConstants.SentimentApiUrlSuffix, serialisedPayload);
        var sentimentResponse = JsonConvert.DeserializeObject<MicrosoftSentimentAnalysisResult>(response);

        // keyPhrases analysis
        var keyPhrasesResponse = await PostHttpRequestToMsServiceAndReturnResult(request, client, MicrosoftCognitiveServicesConstants.KeyPhrasesApiUrlSuffix, serialisedPayload);
        var keyPhrases = JsonConvert.DeserializeObject<MicrosoftKeyPhrasesAnalysisResponseDto>(keyPhrasesResponse);
    }

Ironically, Google has already baked a nice support library and its available on nuget :). The library I used is Google.Cloud.Language.V1. Note that the library is in Beta, so you would have to tick the box in nuget before you can see it in your search. The Google cloud team also have another Google Cloud Language nuget package but that one is more experimental, so they are using it for pushing more prototypes rather than production-ready code. The experimental nuget package can be found here.

Once the package installed the sample code is very simple as below.

// calling google cloud natural language service
    LanguageServiceClient client = LanguageServiceClient.Create();
    Document document = Document.FromPlainText(message.Input); 
    var response = await client.AnalyzeSentimentAsync(document);
    var syntaxResult = await client.AnalyzeSyntaxAsync(document);

One thing to remember is to add the file path of your Google Auth service account (json file) in your Environment Variables. I added it in the project settings as you can see below.

Environment variable for Google Cloud Natural Language service auth

Evaluation

For the evaluation, I have collected lots of data from product reviews, hotel reviews, news articles, economic reports, etc. In this post, I will show the hotel reviews and the news article.

Hotel Reviews

I am using these hotel review that we found online. Here we have the review, the rating that the user given to that hotel, and Google and Azure service responses:

Star Rating Review Text Microsoft Response Google Response
Score (0 to 1) Key Phrases Score (-1 to 1) Magnitude
4 Travelled solo. I have been in this hotel twice since last year for my business trips. Good for me as they have safety solutions, room service, kitchen stuff, balcony and the location is perfect. However, the wifi connection is quite weak, could hardly be reached on the 4th floor. 0.997 kitchen stuff, balcony, room service, safety solutions, location, year, wifi connection, business trips 0.100 1.899
2 Travelled Solo. Stayed in a 2 bedroom apartment. Very small. It not cleaned much at all. The walls were dirty and the mattress had a dirty stain on it which could be clearly seen through the sheet. There was also an opened packet of maltesers still on the bench from the previous guests. Definitely wont stay here again. 0.00002 dirty stain, bench, mattress, opened packet of maltesers, previous guests, walls”, bedroom apartment, wont stay, sheet -0.2000 2.2999
3 Travelled as a couple. This was our second stay at the star gate, I have to admit not as good as the first visit, I didn’t realise all the appartments are not the same this one was much smaller than the first, but let me say still on par with other places we have stayed at, it was facing the street so s bit noisy during the day but quiet through the night, the room was clean with a small fridge and microwave, with a small but ample supply of plates and utensils, the staff are as the first time helpful and friendly, all in all I was a bit disappointed in the stay this time but only because I was spoiled the first time! But I was told I could request the room we had on our first trip so I’m sure it will not be the last stay we have at the star gate. 0.1184 stay, room, time helpful, star gate, small fridge, microwave, ample supply of plates, day, utensils, night, staff, street, trip, visit, places, couple, appartments 0.1000 0.3000

Things to note here:

  1. Google Sentiment Score is a number that range between -1 and 1, which represent the sentiment of the text as detected by Google Cloud. The response from Google also includes Magnitude, which is an positive integer to show the strength of the detected sentiment
  2. Microsoft Azure gives a sentiment score that ranges from 0 to 1. Azure does not offer Magnitude value for the sentiment.
  3. Azure is also giving us the Key Phrases in the text that we are passing. This is good to find out the themes, or the main topics of a piece of text. Note that Azure had an api that was call Detect Topics that would do just that, but it was deprecated earlier this month.
  4. Using Google’s syntax analysis api, I was not able to get the key phrases, but what it gave me was the breakdown of the paragraph or documents into smaller sentences. I found this to be very useful as sometimes you would have a large document (say a news article) and giving it all at once to the ML algorithm could mess up with the result. I might try the other apis from Google like analyse entities in a future blog post

As you can see, Google and Azure both detected the sentiment well most of the time. The accuracy goes up when the rating is closer to the edges (1 or 5 stars). Also note that the key phrases here was not as helpful.

News Articles

Natural language processing is fascinating because it is not just what the person says, there is a lot that is inferred in the context. Thus, I wanted to see how the services would perform in other forms of speech, so I chose a paragraph from a news article. Here is the results:

Paragraph Microsoft Response Google Response
Score (0 to 1) Key Phrases Score (-1 to 1) Magnitude
Australia’s unemployment rate is already stubbornly high at 5.7 per cent (around a percentage point above the US) and if none of those 40,000 auto-related workers found new jobs it would rise back above 6 per cent. 0.0310 related workers, new jobs, Australia’s unemployment rate, percentage point

Interestingly, both Azure and Google Cloud sentimental analysis accuracy went terribly down in this case. Azure did try to give a score, although not accurate. However Google Cloud returned 0 for both sentiment and magnitude, which is odd.
On the other hand, the key phrases returned from Azure are more relevant.

Conclusions

Natural language processing is a very complex and fascinating subject. From what I have seen, you cannot just take a service from any vendor and just use it out of the box. All the services I have reviewed so far have accuracy issues. The tricky part is finding the sweet spot by evaluating varying data sources and massaging the data before feeding it to the algorithm. In the next blog post, I will do that and review some of the other available services from IBM and Amazon.

If you have a comment, feedback or a question, I would love to hear from you