Monday, January 16, 2012

Bing Vision API

As part of the Windows Phone 7.5 Mango upgrade, Microsoft added a nifty feature to Bing search called Bing Vision. Bing Vision is capable of scanning barcodes, book covers, album covers and posters. It can also perform OCR very capably. It appears that Bing Vision is Microsoft's response to Google Goggles. Just like Goggle Goggles, Microsoft's Bing Vision does not seem to have a public API. This post is an attempt at reverse-engineering the image search portion of the Bing Vision API. I have not been able to reverse-engineer the OCR portion of the Bing Vision yet, but once I get it I will be the first to post it.

Warning: this API is not publicly released by Microsoft, so a lot of it is subject to change. Use at your own risk.

It is important to note that Bing Vision's image search is limited to searching products. If I were to feed it an image of the Mona Lisa, I would get back a list of books and frames of the Mona Lisa for sale rather than identifying it as a portrait by the Italian artist Leonardo da Vinci.


T
hat said, the beauty in Bing Vision is in its simplicity. You simply feed the API call with an image, and you get back the result in an XML. Sounds simple.. right? Now let's dig deeper.

The API that we are going to examine in this post is Bing Vision's image search. Send a POST request to http://
wp.bingvision.ar.glbdns.microsoft.com/ImageSearchV2.ashx with the following headers:

    Pragma: no-cache
    Content-Type: image/jpeg

with the image you would like to search as the body of the POST request.

The following working code sample illustrates how you could issue your request in C#. Note the HTTP requests here are synchronous just for simplicity.

namespace BingImageSearch.NET
{
    using System;
    using System.Net;
    using System.IO;

    class BingImageSearch
    {
        static void Main(string[] args)
        {
            string path = @"C:\image.jpg";
            byte[] imageByteArray;

            using (FileStream fs = File.Open(path, FileMode.Open))
            {
                FileInfo info = new FileInfo(path);
                imageByteArray = new byte[info.Length];
                fs.Read(imageByteArray, 0, imageByteArray.Length);
            }

            BingImageSearch.BingImageQuery(imageByteArray);
        }

        private static void BingImageQuery(byte[] image)
        {
            HttpWebRequest request =
                (HttpWebRequest)WebRequest.Create("http://wp.bingvision.ar.glbdns.microsoft.com/ImageSearchV2.ashx");

            request.Method = "POST";
            BingImageSearch.AddHeaders(request);

            using (Stream stream = request.GetRequestStream())
            {
                stream.Write(image, 0, image.Length);
                stream.Flush();
            }

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            BingImageSearch.PrintResponse(response);
        }

        private static void AddHeaders(HttpWebRequest request)
        {
            request.ContentType = "image/jpeg";
            request.Headers["Pragma"] = "no-cache";
            request.KeepAlive = true;
        }

        private static void PrintResponse(HttpWebResponse response)
        {
            using (StreamReader reader = new StreamReader(response.GetResponseStream()))
            {
                Console.WriteLine(reader.ReadToEnd());
            }
        }
    }
}

The service is capable of accepting a barcode image, and it would return to the caller the barcode type and its number, as shown below:

  
    
  

And it can also accept the cover of the product. I get back the following XML when I post the cover of this game.


  
    
      EA FIFA Soccer 12
      FIFA Soccer 12 delivers a true soccer experience with authentic club and league licenses, and intelligent gameplay that mirrors real-world soccer. Compete as any one of over 500 officially licensed clubs and experience responsive, intelligent and realistic action. Enjoy turning defenders with sophisticated dribbling and ball control, snapping off precision shots and placing beautifully timed passes with pin point accuracy.
      EA FIFA Soccer 12
      http://bingvision.blob.core.windows.net/thumbnails/655f98db86ca372371078f0abc4016d6.jpg
      E0703D6431D7D0845005
    
    .
    .
    .
  

It is up to you to choose how you will parse the XML response. If you plan to explore this API further with C#, you might want to create proxy objects (via xsd.exe) so that you can easily serialize the XML and enumerate the objects and their properties. I will not be covering that in this post.

11 comments:

  1. Hi Fadi,
    When I am trying to hit the URL that you have mentioned above, I am receving 403 forbidden error.Can you help in this regards.

    ReplyDelete
    Replies
    1. The URL in the code sample was incorrect - I just fixed the code sample. Though, I was not getting 403 with the incorrect URL. How are you hitting the URL? You shouldn't be getting 403.

      Delete
    2. I just made some modification as to make it compatible for working on Windows Phone 7,i.e. instead of GetRequestStream and GetResponseStream , I used BeginGetRequestStream and BeginGetResponseStream respec...On debugging I got an exception stating server not available..so I directly copied the URL and pasted it to the browser..that time I got this error :( which I have mentioned above...Please can you help me in this regards and suggest any effective alternative to implement the same on windows phone 7.Thanks in advance

      Delete
    3. why don't you just post your email - don't worry, only I will be able to see it as I moderate my comments.

      Delete
  2. HI, i success to adapt your code to WP7 but i have it answer only balise
    so i find that i don't send a good image.
    maybe if you help me to transform WriteableBitmap to BitArray, i think that i could succes, thanks again
    Antony

    ReplyDelete
  3. Hi, i succeed to adapt your code on Windowsphone 7.1
    but it answer me with and didn't success to read my image.
    I take a photo and directly send it to webservice
    so if you help me to transform Writeable to bitmap Image, i could succes to obtain good result.
    xqz me for my english but usually speak french

    thanks

    ReplyDelete
  4. Hi Fadi,
    I'm working for a professor at the University of Toronto on Windows Phone development, and I'm current trying to adapt your code on Bing Vision to my app. I've gotten asynchronous web requests to work on other URLs, but the one for the Bing Vision API returns an exception saying the remote server has not been found. I've tried using Fiddler to look at the traffic, and checked that I've gotten the right headers. I can also check that I've uploaded the correct image, so I'm wondering if it's a problem with the server.
    Can you please help me with this problem, and tell me if there's something I'm missing?

    Thank you.
    Ray

    ReplyDelete
    Replies
    1. Did you find the solution ? If so can you please share how you were able to solve it ?

      Delete
  5. Hi Fadi,
    I'm working for a professor at the University of Toronto on Windows Phone development, and I'm current trying to adapt your code on Bing Vision to my app. I've gotten asynchronous web requests to work on other URLs, but the one for the Bing Vision API returns an exception saying the remote server has not been found. I've tried using Fiddler to look at the traffic, and checked that I've gotten the right headers. I can also check that I've uploaded the correct image, so I'm wondering if it's a problem with the server.
    Can you please help me with this problem, and tell me if there's something I'm missing?

    Thank you.
    Ray

    ReplyDelete
  6. Hey, im a Computer Science student from the Netherlands. I have tried to adapt your code in use of a Windows Phone 7.5 app, but the services returns empty xml nodes for both Product and Barcode-matches. I identified the issue as my image info not being JFIF encapsulated, as opposed to the wireshark caputured Bing Vision data. I am having trouble getting this to work, is there anyway you could assist me in this?

    ReplyDelete
  7. Hello Fadi,

    Thank you for your code.

    Can I use this code to read numbers from a picture?

    Thank you and regards,

    ReplyDelete