The Most Important Feature Missing In The Google Search API

About This Blog

This site is for  entrepreneurs.  A full RSS feed to the articles is available.  Please subscribe so we know you're out there.  If you need more convincing, learn more about the site.

Community

Google+

And, you can find me on Google+

Connect on Twitter

Get Articles By Email

Your email:

Google

Blog Navigator

Navigate By : 
[Article Index]

Questions about startups?

If you have questions about startups, you can find me and a bunch of other startup fanatics on the free Q&A website:

Answers.OnStartups.com

Subscribe to Updates

 

30,000+ subscribers can't all be wrong.  Subscribe to the OnStartups.com RSS feed.

Follow me on LinkedIn

OnStartups

Current Articles | RSS Feed RSS Feed

The Most Important Feature Missing In The Google Search API

 


At my startup, HubSpot, we have been working with the Google Search API to implement some of the features we think would help our customers.

The Search API is reasonably robust in that it supports the various features of the Google search engine (finding related links, approximating the number of results, etc.)

But, there is one critical feature that the brainiacs at Google either forgot to include (which is bad) or intentionally left out (which is really bad).

Outside of normal “search” type stuff, I think one of the most common reasons people would use the API is to answer one simple question:

Most common question:  For a particular search phrase, where does my site rank on Google?

The reason this question is common should not be surprising (most webmasters, bloggers and SEO consultants care about this issue).  It’s also difficult to answer this question via the regular search engine (without manually entering the search term, and paging through the results looking for a “match”.  There are web utilities out there (that let you enter your API key and run a query), but they’re just doing a brute-force iteration over the result set too.

Here are some thoughts on the topic:
  1. As it stands, there is no way to answer the above simple question without making repeated calls to the Search API (basically retrieving a page at a time and checking the results until a match is found).

  1. This is even more annoying because Google only allows you to retrieve 10 result items at a time.  So, to figure out if you are in the top 100 hits for a search phrase, you have to hit Google 10 times.

  1. This is made yet more annoying because Google limits the number of calls you can make to their API to 1,000 (with no clear way of increasing this limit – even by paying money).

  1. It seems (at least from my perspective), extremely easy to implement this feature.  All they would have to do is include a separate method call that took a search query and a site name as parameters and returned the position of the first “match”.  This way, I could figure out that when searching for “software startups”, that this site (OnStartups.com) is the #5 hit.


Given how smart the Google folks are and how common this particular need likely is, I have only two theories about why they left this feature out:
  1. Google intentionally left this feature out for some “strategic” reason.

  1. Google doesn’t realize how important this missing feature is.


For the Google API experts out there:  Am I missing something simple?  Is there a work-around to this, or have I stumbled into something that is already widely known and has already been discussed to death?  If you have insight, please leave a comment.  All help is appreciated.

Posted by admin_onstartups.com admin_onstartups.com on Thu, Oct 05, 2006

COMMENTS

I opt for reason # 1, from my understanding of google is they want their results to be good for the USER. If the rate that spammers figured out their system was greater then they could update it would not be a good thing. With that in mind I see google limiting the amount of tools seo people get so they can stay ahead of the spam wave. Any one else know more about googles policys on api's vs spammers ?

posted on Thursday, October 05, 2006 at 11:46 AM by Charles Verge


You can get more than 10 results at a time (up to 100) through normal GUI search, the API should support it as well. To get more than 10 results, add &num=XX to the query URL, or go to Advanced Search and put the number you wan tthere.

posted on Thursday, October 05, 2006 at 12:02 PM by Yoav Shapira


Coincidentally, I started a project last week that will help solve that problem, among other things. Email me if you want to know more.

posted on Thursday, October 05, 2006 at 2:40 PM by Erik Peterson


As I recall, the interactive Google Sitemaps does answer this question in a limited way: "How does my site rank for its most popular search terms?"

Not quite the same question you asked, but useful nonetheless.

posted on Thursday, October 05, 2006 at 3:34 PM by Martial Development


Though not in the API, (which seems to make sense to me since the API is more oriented toward providing end-user functionality, rather than webmaster data) you can get the info you're suggesting by logging into Google Sitemaps, (http://www.google.com/webmasters/sitemaps) where you can either view the info directly online or download.

posted on Thursday, October 05, 2006 at 3:45 PM by Dylan Hunter


Good to see somebody tackling the can of worms.

These guys are smart. There's no way they aren't aware.

This is part of the big plan from the start.

Personalised search is a valid issue, but is it a killer (NO) and does it currently mean very much anyway (DON'T KNOW).

Put yourself into their heads, work through the thinking and you'll see some of what they're up to.

posted on Thursday, October 05, 2006 at 4:34 PM by Anonymous on this


of COURSE google left that out. They don't want people to be able to figure out how to manipulate their search results.

posted on Friday, October 06, 2006 at 12:55 PM by larry d'anna


Try Google Sitemaps. It allows to download the full keyword/position stats in CSV format, as long as it is your own site you are analyzing.

posted on Friday, October 06, 2006 at 4:27 PM by Alex


As per google definitions "The SOAP Search API was created for developers and researchers interested in using Google Search as a resource in their applications." Google never wanted SOAP api to be used as keyword positioning tools for SEO. Hence they kept this restriction. After releasing soap api they come to know it wasnt doing well with restrictions for webmasters. So they stop giving more soap api and now switch to AJAX search api which can give infinite search results but cant be used by SEO as keyword position tool.

posted on Thursday, July 26, 2007 at 4:04 AM by free seo tips guidelines


Erik Peterson I would like to talk with you about you recent post: 00 39 34 0595 3232

Coincidentally, I started a project last week that will help solve that problem, among other things. Email me if you want to know more.
posted on Thursday, October 05, 2006 at 2:40 PM by Erik Peterson


posted on Tuesday, December 11, 2007 at 4:34 AM by Nikolas


Comments have been closed for this article.