Connect with us: facebook twitter rss

Site Diagnosis – are all your pages indexed properly?

Posted by sam on March 30th, 2008 71
Posted in Uncategorized |

I have just released a new feature to Link Diagnosis Firefox Extension that will allow easy diagnostics of the indexed pages on your website.

Couple of weeks ago I was facing a tedious task of finding out which pages out of 100k on the site are indexed and which are not. I knew that some of them could have been marked as duplicate content or that Google simply didn’t indexed them because of the size of the website.

First I installed Google Webmaster Tools hoping that Google will tell me that. Unfortunately, the Indexed Pages tab just points me to use site: command.

I don’t trust site: command. Especially, the count of number of pages is very inaccurate. I know I have 100k pages and Google tells me I have 150k pages indexed.

Also, there is no easy way to see more than 1000 pages (you can play with inurl: commands but it takes ages and you can get banned).

Because of these problems I decided to code a tool which would automate it – Site Diagnosis.

The internal algorithm of the tool works as follows:

1. Go through every URL in XML Sitemap file and do a simple check inurl:http://www.samplesite.com/dir/url1
2. For every url that does not appear on inurl: command there is still a chance that page is indexed but does not appear with inurl
3. For every url in XML Sitemaps I get a title and perform this check site:http://www.samplesite.com sample title
4. If the page does not rank in top 10 for its title within the site then probably something is wrong.

This check is suprisingly accurate and most of the pages that don’t survive this check have some problems like duplicate content, missing titles, missing content or not enough content. These troubled pages usualy don’t appear in the search results if you search for any text on the page – not even when you enclose sentences in quotes.

Obviously, the goal of search engine optimization is to fix these pages so Site Diagnosis will hopefully be essential in identifying them.

This entry was posted on Sunday, March 30th, 2008 at 5:35 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

  • Billy

    Hi, is there a way to private label your tool?

  • Billy

    Hi, is there a way to private label your tool?

  • Janusz

    Billy> I am not planning to do any white-labels sorry. You might try Blogstorm hosted link analysis tool.

  • Janusz

    Billy> I am not planning to do any white-labels sorry. You might try Blogstorm hosted link analysis tool.

  • http://directory.letspromoteit.com Share your link

    Very nice tool. Thanks for sharing.

  • http://directory.letspromoteit.com Share your link

    Very nice tool. Thanks for sharing.

  • http://www.threeaces.de pare

    Thank you for that awesome tool. I´m a frequent user and I have to say that all the features are very usefull and functional.

  • http://www.threeaces.de pare

    Thank you for that awesome tool. I´m a frequent user and I have to say that all the features are very usefull and functional.

  • http://www.thinkpragmatic.net Alex

    It’s really a nice tool: useful and simple!

  • http://www.thinkpragmatic.net Alex

    It’s really a nice tool: useful and simple!

  • http://www.j8seo.nl Jeroen – J8 SEO

    Great tool! Thanks a lot! Very useful!

  • http://www.j8seo.nl Jeroen – J8 SEO

    Great tool! Thanks a lot! Very useful!

  • Randy Moore

    Great Tool!

  • Randy Moore

    Great Tool!

  • Pingback: Google

  • http://www.typicalenglish.nl Typical English

    the best thing since sliced bread.
    - Great stuff

  • http://www.typicalenglish.nl Typical English

    the best thing since sliced bread.
    - Great stuff

  • http://www.alejandroarco.es posicionamiento web

    Very nice tool. Thanks for aplication.

  • http://www.alejandroarco.es posicionamiento web

    Very nice tool. Thanks for aplication.

  • http://znak-u.ru Igor

    VERY USEFUL INSTRUMENT

  • http://znak-u.ru Igor

    VERY USEFUL INSTRUMENT

  • http://www.viking.rimini.it Hotel Rimini

    I love your tool , I have read the there is a not-free version or will be it?

  • http://www.viking.rimini.it Hotel Rimini

    I love your tool , I have read the there is a not-free version or will be it?

  • http://www.typicalenglish.co.uk/index_en.php David Nicholas

    Great tool

  • http://www.typicalenglish.co.uk/index_en.php David Nicholas

    Great tool

  • http://www.businesspark.org Hewolka

    Hej – a great tool! Really love it. And very interesting insights you shre in your articles – so many thanks to you!

  • http://www.businesspark.org Hewolka

    Hej – a great tool! Really love it. And very interesting insights you shre in your articles – so many thanks to you!

  • 4everdesign

    Thanks. FF ext is especially cool.

  • 4everdesign

    Thanks. FF ext is especially cool.

  • http://ajemailmarketing-software.blogspot.com Email Marketing Solution

    Great Tool. Very helpful!!

  • http://ajemailmarketing-software.blogspot.com Email Marketing Solution

    Great Tool. Very helpful!!

  • http://www.guitarpedalreview.com guitar pedal reviews

    Now that’s a useful tool. That’s the problem I always have because of the limitations the search engines give. Nice.

  • http://www.guitarpedalreview.com guitar pedal reviews

    Now that’s a useful tool. That’s the problem I always have because of the limitations the search engines give. Nice.

  • http://wordwolf.com April

    This is actually a feature I had been hoping someone would come up with. As far as I’m concerned, if pages aren’t indexed in Google they are potentially losing me money. So being able to see what pages aren’t indexed I can simply get a few links to them. Thanks.

  • http://wordwolf.com April

    This is actually a feature I had been hoping someone would come up with. As far as I’m concerned, if pages aren’t indexed in Google they are potentially losing me money. So being able to see what pages aren’t indexed I can simply get a few links to them. Thanks.

  • http://www.wikihow.com/Get-Your-Website-Indexed-in-Less-Than-2-Days Expert SEO guides

    That’s new.I hadn’t never thought the evidently simple ways Google worked in. The truth of the issue is that even though Google looks at your page multiple times, it takes a metric tonne of work on your part in order to get a page to become intriguing to Google. This will add to my understanding of search engine optimization.

  • http://www.wikihow.com/Get-Your-Website-Indexed-in-Less-Than-2-Days Expert SEO guides

    That’s new.I hadn’t never thought the evidently simple ways Google worked in. The truth of the issue is that even though Google looks at your page multiple times, it takes a metric tonne of work on your part in order to get a page to become intriguing to Google. This will add to my understanding of search engine optimization.

  • http://www.prosearchplus.com Terry Watson

    I know this is off the topic but I found this site by searching on Yahoo for hotel marketing. How did you optimize your site to rank so high in the search engine results?

  • http://www.prosearchplus.com Terry Watson

    I know this is off the topic but I found this site by searching on Yahoo for hotel marketing. How did you optimize your site to rank so high in the search engine results?

  • http://www.zrce.eu Partyurlaub

    your tool is really great. i like it. its fast and you can see all information direktly after the scan.

  • http://www.zrce.eu Partyurlaub

    your tool is really great. i like it. its fast and you can see all information direktly after the scan.

  • http://www.internetiq.nl Danny InternetiQ

    I searched and found! Excellent tool.

  • http://www.internetiq.nl Danny InternetiQ

    I searched and found! Excellent tool.

  • http://freecouponstoprint.org free coupons to print

    nice points. thanks for sharing.

  • http://freecouponstoprint.org free coupons to print

    nice points. thanks for sharing.

  • http://www.air-jordan-16.com air jordan 16

    I like to come here, I sincerely hope here will be a place of internet communication of travelling amateur.I believe that there are many problemsfor you about travelling.But here can solve all of your problems.So I like it very much,it let me make sme friends here. Recently,I found some fashion websites, there are many things very suitable for you, go and see.The website is Nike Zoom Blur Low

  • Jmk

    this Website is very good. you can find something surprise if click it.http://www.gucciusaoutlet.com/

  • Pingback: LEROY

  • Pingback: JEROME

  • Pingback: JEFFREY

  • Pingback: DARRELL

  • Pingback: SERGIO

  • Pingback: JEROME

  • Pingback: SIDNEY

  • Pingback: CECIL

  • Pingback: DENNIS

  • Pingback: KIRK

  • Pingback: PHILIP

  • Pingback: DANIEL

  • Pingback: RONNIE

  • Pingback: FERNANDO

  • Pingback: ANDRE

  • Pingback: VIRGIL

  • Pingback: TYRONE

  • Pingback: ALFREDO

  • Kllkg9

    The Hollistervivienne westwood cufflinks brand determine gives out jeans, T shirts, outerwear, jackets, vivienne westwood uk sweatshirts, innerwear for grownup males collectively with a

    vivienne westwood bracelets assortment of fragrances, shoes and accessories. Their vivienne westwood online outfits are classified as “Dudes” for grownup males as well as vivienne westwood necklaces the women’s collection is recognised as ‘Betty’s”.

  • Kllkg9

    There are flip http://www.westwoodjewellerystore.com/vivienne-westwood-necklaces.html flops, beach top totes, belts as well as some products http://www.westwoodjewellerystore.com/vivienne-westwood-rings.html of jewellery http://www.westwoodjewellerystore.com/vivienne-westwood-bracelets.html available. They have http://www.westwoodjewellerystore.com/vivienne-westwood-uk.html cosmetics also like lip balms and gloss. The Hollister stores are made to begin looking like cabins as well as the internal rocks with music, http://www.westwoodjewellerystore.com/vivienne-westwood-cufflinks.html the most latest perfume fragrance and dim lighting.

  • http://pulse.yahoo.com/_AC3BUBNZAR26IEFQJAADAH7XBE You You

     

    Ralph
    lauren,Polo ralph lauren,ralph
    laurenpolo,Classic-Fit Polo,ralph lauren Classic-Fit Polo,Custom-Fit Polo, ralphlauren Custom-Fit Polo, ralph lauren Sweaters, ralph lauren Jacket  Outwear,Black Watch ,ralph lauren sport shirts, ralphlauren tee, ralph lauren Han dbags,ralph lauren dress, ralph
    lauren hatcap,Ralph Lauren Children , Ralph Lauren Sport Shirts
    ,ralphlauren Hoodies,ralph lauren shorts,Big Pony Polo,OceanRace
    Polo,Tonal Big Pony

     

    ,Rustic Nicked Polo Banner Big Pony,Dual BigPony,Havana Stripe Polo,Mercerized Stripe,Yacht
    Club Stripe,Pieced BigPony,Tonal
    Big Pony Polo,Mercerized Thin Stripe,Claridge Stripe

     

     

    Shelter Stripe,Island Stripe Polo,Shelter
    Stripe Polo,Stripe ShelterPolo,Beaded Big Pony,Dual Match Polo,Country Flag Polo,Mercerized Polo,Oxford StripePolo,Point
    Stripe Polo,

     

    Polo
    Ralph Lauren (Ralph Lauren) in 1968 registered in New York. Ralph
    Lauren’s design combines the romantic atmosphere, innovation
    and inspiration, classical charm, attention to detail, fabric always gives the
    feeling of comfort, style simple, smooth, the U.S. representative of the brand
    creation, Shengtaibaoluo “has become gentleman today synonymous with
    casual wear. the spirit of the brand is also the most polo.

     

     

    In RALPHLAUREN
    clothing kingdom, men’s, women’s, children’s clothing, sportswear, has been home accessories,
    and everything smelled an free, comfortable atmosphere of the United States,
    all design details are not in a framework of time-out values. Has been focused
    on shaping the minds of the integration of the Wild West, Indian culture, and
    old Hollywood feeling of “American style”, RalphLauren
    is to be representative of the magazine are labeled
    “American classic” designers. In addition to clothing, PalphLauren
    leather accessories as well as the development of the accessories are very
    complete, from head to toe, including eyeglasses, jewelry, purses, belts,
    socks, shoes … and so, pay attention to all is a “lifestyle”, he
    lead trend, but modest fashion, with a kind of wealthy middle-class taste.

  • http://deejay.blog.hr House Music DJ Blog

    That’s a good update to Link Diagnosis.

  • 42room

    　　It is reported that Lee Yong-ho said South 女性のためのuggのブーツKorea should fulfill the 6.15 and 10.4 of the joint declaration.The source said that the U.S. participants at cheap ugg boots sale the meeting that the process of denuclearization progress, the DPRK-US relations can be improved.

  • Bunanun Posri