Yahoo!’s BOSS API will soon include new features from Bing’s own Webmaster tools. The good news to Link Diagnosis users is that the tool will not go away after the merge. If you have a time machine, we encourage you to set the dials back to July 7, 2010 at around 700 GMT to take a peek at when our service “blew up”. That it happened in the middle of our beauty sleep was a most unwelcome surprise. But as it turns out, that was but a premonition of the upcoming bigger one slated to happen sometime in the near future (we’re already adjusting our hibernation cycles and started assigning sentries just for kicks- send us marshmallows for the poor soul who’s by the campfire in those ungodly hours).

As we have previously alluded to in the past, we use Yahoo! BOSS as our main service provider to query for backlinks. But roughly about a year ago, Yahoo! Search decided it likes Bing and the two pledged their undying love for each other. The consummation of that marriage is the birth of a new service. The service could at some point included a monetized version of their currently free and open APIs. We can’t say this has taken us by surprise- we were kinda waiting for the official word, but at the same time kinda hoping it would never come.

Fortunately, we have an excellent backup system in the works, which you will love, and it will likely eventually take over our current service – most importantly we will keep it FREE.

What this means for our users is at the very least, we will be publishing a new FireFox extension. The jury is still out on this one but like you, we would like to keep our doors open forever.

To read more about the upcoming shake-up from the our gracious data providers, follow this link: http://developer.yahoo.net/blog/archives/2010/08/api_updates_and_changes.html

As always, thank you for choosing LinkDiagnosis and thank you for continued support!

So to our dedicated users, thank you for bearing with us. We promise we haven't been vacationing on a beach somewhere. This is only the beginning of roll outs, a partner site to Link Diagnosis is about to come out to, the one which our home page hints at.

As before, here are the top 10 notable features you'll find in our re-release:

10) Remove Unpopular Options Let's face it, we don't always know what you want. So when we came out with the options to disable Anchor Text Detection and Outbound Link Checker, we were fairly confident it was going to be a good thing. But of course you said otherwise and as much as we hate to admit it, it did cause some problems.

9 ) A Brand New Template Everyone likes makeovers, and so do we! Our new template is much easier on the eyes compared to our previous bubble gum pop injected template. This template is a mix between the classic and version 2 and fits in nicely with our network of brands.

8 ) Speaking of brands… Did you notice our footer? It's possible that you may have even been contacted by one of our brands at one point in time. Well that's all thanks to the mothership who pays for our upkeep. But the cool thing is though we work for them (and build them tools of destruction which blows everything else out of the water), they still let us do our own thing and take some of those tools we built for them and let you play with it! Neat!

7 ) Result Sets Upped to 1000! Yeah, you read that right! We have upped the all around limit from 100 to 1000. That means up to 1000(or 1001) indexed pages will be retrieved… multiply that by up to 1000 backlinks for each of those pages… and yes, the theoretical maximum data set now possible is 1,000,000 up from a measly 10,000. That's an incredible 10,000% increase! Expect report generation to increase by as much too!

6 ) Return of the Full Reports In yet another case of thinking we knew what you wanted, we thought you didn't want to be inundated with pages upon pages of links from a single site just because a link back to your site is a permanent fixture on that site's template. Clearly we were wrong, right away we had people letting us know something was wrong. We just wanted to show the relevant links to serve up an intelligent report. So by popular demand, you now have the option to show a complete (now up to 1000!) listing of all your site's backlinks. Keep in mind though, that hundreds or thousands of these backlinks might come from the same URL – so we still recommend using the Optimized report option.

5 ) Cleaning Up and Clearing Out those Pesky Little Bugs If we had a penny for all those little bugs we uncovered/reported to us… it would probably cause our office building(what's this, we're not in a garage somewhere? blasphemy!) to collapse! There are too many to mention but among the notables:
- we had erroneously dropped support for PR 10. Although if you had a PR 10 site linking back to you you probably won't be using our tool, or worse, your site is on a .gov watch list.
- sometimes, the incorrect anchor text was getting pulled especially when multiple backlinks to your site exist on the page.
- improved and lighter flash usage.

4 ) Back to Basics We (now) understand not everyone likes flash. We don't know if its just Apple's influence or… At any rate, we have created a Basic version which is solely javascript powered. The plain vanilla flavor, however, doesn't enjoy all the same goodness the enhanced version does. Likewise, if you find you're having trouble viewing the Enhanced version, you may still reload the same report using the Basic version and vice versa.

3 ) No PageRank Option Double Checker A good number of you complained about not getting PageRank even though the option has been set to "on". We have now packaged in a lower level double check to make sure you wanted to run a report with PageRank set to off. Right before a report starts, it will ask you to confirm if you really wanted to NOT check for PR.

2 ) Extension Manager The extension manager is quite a hefty addition and improvement to our FireFox extension. So much so that we dedicated a separate blog post for it. Link here!

1 ) Save the Trees, Share Reports! Every report you run is automatically saved to the cloud. Consider that running a report is the equivalent of running multiple queries on the search engines, and each search to the SE has the same environmental impact as boiling a kettle of water, sharing reports could very well put an end to global warming… of course not. But we think its a nifty little feature which makes it easier for you, our users, to share results with others. The best part is they don't even have to download our tool or use FireFox in order to view the report you shared! But remember, unless you share it, they won't find it! We are not going to put up a listing of all the reports you ran, only you can share it. Registered users have more powerful ways to share reports which they'll find in the Reports Page. To show you just how awesome it is, here's the backlink report on the 10 most popular Twitter Profiles:

1 ) aplusk – saved report
2 ) britneyspears – saved report
3 ) TheEllenShow – saved report
4 ) ladygaga – saved report
5 ) BarackObama – saved report
6 ) KimKardashian – saved report
7 ) Oprah – saved report
8 ) johncmayer – saved report
9 ) taylorswift13 – saved report
10) twitter – saved report

* How were we able to minimize N/F results in these samples? We piggy-backed on the private iAcquire Technology Solutions. Sometimes, public traffic is allowed in especially when testing platforms and servers but it remains private for about 363 of the 365 calendar days. How will you know when its allowing public traffic? You won't. You'll just notice your PR results will not have as much N/F in them… and you'll just have to check everyday if N/F results are at a minimum.

^{The extension manager can be accessed through the context menu or through a link in the pop-over.}

The extension manager provides a list of all the reports that have run, is running, or is queued. These are differentiated into "steps" as marked under the progress column.

- Step 1 means the report has been successfully recognized by the tool and is currently queued pending the completion of a running report. If there are multiple reports in step 1, then the report that will be tackled next will be picked randomly. If no report is in the step 2 designation, then there shouldn't be any report in step 1.
^{A report is being generated by our FireFox extension.}

^{Another report is requested while a report is still running.}

- Step 2 denotes that report is currently running and is usually accompanied by a progress estimate. Only one report may be in step 2 at any given time.
^{A queued report automatically starts upon completion of the previous report.}

- Step 3 marks all reports that are no longer being worked on, this could either mean its finished or it was cancelled.
^{The manager showing 3 reports in varying steps of completion.}

The one thing you have to remember is that the extension manager is what could be called the controller for our tool which in turn resides within FireFox… it's bound by the same rules set by and for FireFox. Keeping a lot of reports in the tool would naturally mean FireFox will need more memory. Given that this is not an essential part of FireFox, it has low priority so when the internal Garbage Collection routines of FireFox kick in, the reports that you ran that's being kept track of the tool could fall prey to those. In which case, a reload from the extension manager could fail and it would have to be downloaded from our server. The best use is to keep saving reports in our interface, and keep deleting reports on the extension manager to keep our tool's footprint within FireFox on the low side. The delete feature within the extension manager works two ways. Firstly, reports that are in step 1 or step 2 gets cancelled if they are deleted from the manager, they then take on the step 3 designation. When in step 3, deleting the report then removes it from the tool's memory.
^{The extension manager showing a cancelled report.}

Though the extension manager is a spartan tool for our app in FireFox, the potential for more functionalities to getting tacked into the manager is boundless. What would you like to see added within the extension? Speak up in the comments below, or contact us through the usual page.

9) Upgraded Blog
The design and layout of the blog has been reformatted so that users can check up on Link Diagnosis news and updates. Don’t forget to read all the comments of each blog entry as our users always have great input. We read every single comment and do our best to develop based on user feedback.

8 ) PageAge Check
At Link Diagnosis we understand that checking PageAge of a site is a key part of evaluating the quality of a link. We built a PageAge check within the links tab so that users can gather all of the important information they need as quickly as possible.

7) Select Rows Using Shift and Control
Users can now export selected parts of a report to gather specific information. This option was built so users can research a backlink portfolio using targeted criteria.

6) Improved Charts
Some people like charts to get the visual breakdown of statistical data. The new charts image is cleaner and uses a nicer color breakdown.

5) Creating Accounts and Saving Reports for FREE!
This new upgrade almost made #1, as the ability to save a report is one of the best features of the site. We analyzed the reports that were being run and noticed that many sites were being run multiple times in a single day. You can still run as many reports as you like,  but now you can save and load them in a matter seconds.  Best of all it’s FREE!

4) Export Using Selected Data only Now Available
Many of our users run reports with many results. We understand that filtering through all of this information is time consuming. Analyzing a sites backlinks doesn’t have to take hours. Users can now select a group of links and export them in a matter of seconds.

3) Updated Link Data
We provide better data by truncating out multiple links coming from the same URL, so in 100 links from the same URL we take the top page and display it only once (very important for PageRank scraping limitations). We are allowing users to continue using the classic version Link Diagnosis. The updated data allows users to gather information quickly and analyze the sites that are important in your link building campaign. We will also roll out an option to run “full reports”, like Classic Link Diagnosis, vs. “optimized reports” like New Link Diagnosis.

2) Faster Reports
SEO’s and website administrators today have to do more in less time. We have sped up the process so our users can analyze more sites and run more reports in less time.

1) No More Paging!
Number one is without a doubt the elimination of paging. Many reports had hundreds of pages of data; unless you exported the report it was difficult to get the entire picture. Users no longer have to page to see the information and can still export if needed.

Before you dive in and start running reports here are a couple of tips and updates so you are prepared for the new site.

• New Download:  Before you get started make sure that you have downloaded the new tool. Here is the URL to go straight to the download page:

www.linkdiagnosis.com/download.php

• Adobe Flash: The new version of the site is now all in Flash.  This update allows for you to see all of the back links for the site at once. There is no more paging on the new site.  Make sure that you have the latest version of flash.  You can download the latest version of flash here:

http://get.adobe.com/flashplayer/

Both of these updates will prepare you for the new and more advanced Link Diagnosis experience.  The new site has improved on all of the old functionalities and added tons of new features.

The new flash version of Link Diagnosis gives us the ability to sort information faster.  Since we can display more and sort information easier you can now sort and organize your report using different criteria.

• Page Rank:  You can now sort by PageRank from 1-9.

• Anchor Text: Using the columns on the report you can sort your report by anchor text.

• Outbound Links:  By selecting the column you can sort from least to greatest number of outbound links.

The reports page now displays information in a more user friendly layout.  We have had numerous comments requesting better sorting and filtering features.

At the top left of the report page you will see the results from the report displaying the total number of backlinks, total number of unique anchor texts, and the number of pages indexed.  On the top right users can select groups of links using the PageRank filter or the link type filter.  Selecting links is helpful as you can now export specific links to CSV format.

There are 4 buttons below the overview data, which include:

• The Links Tab- This tab enables you to open a URL in a new window, check the page age of a URL, or run a new report.

• The Export Tab – The feature will let you save an entire report or selected URL’s to a CSV file.

• The Charts Tab- This features a brand new graphic summary of a backlink report.

• The Save Tab – The tab allows you to save a report to your account.

Clicking the top of any of the columns will sort the results in ascending or descending order.

The pages indexed and most popular anchors display have also been modified for quick and easy sorting and improved display. See the below screen shots for a better layout of these pages.

Pages Indexed Display:

Most Popular Anchors:

The biggest update that Link Diagnosis has released is the ability to run a save a report and create projects.   Some reports can take 15 minutes or more to complete, and we noticed that many users from the same IP address run the same report multiple times. To improve efficiency for our users we are now allowing users  the ability to save these reports and retrieve them at anytime.

You can create your LinkDiagnosis.com account by following this link:

http://linkdiagnosis.com/login.php

Once you have created your own account you will be able to quickly load saved reports and create projects. Here is a screen shot of the new navigation for Link Diagnosis accounts:

We are constantly working on new features and will be unrolling out new functionality in the coming months.

Please let us know what you think and send over any suggestions.  We take your comments very seriously and read everyone’s feedback.

As always, we appreciate your loyalty, and wanted you to know that we are busy working on an updated free version and paid version of Link Diagnosis. We are not going to take any functionality away from the free tool version.

We need your input though. What else could we be doing? How can we make the results better?

Please provide your insights. Keep in mind that speed is always an issue based on the sheer volume of scrapes we perform. We will soon allow you to modify the results sets so that data can be more quickly reached.

Best,
LinkDiagnosis.com Team

We will also be incorporating a service which allow you to create and store projects online, setup multiple URL’s per project, combine multiple URL results into one big report with sorting, and even private-labeled report exports. Stay tuned folks – more good stuff to come!

Couple of weeks ago I was facing a tedious task of finding out which pages out of 100k on the site are indexed and which are not. I knew that some of them could have been marked as duplicate content or that Google simply didn’t indexed them because of the size of the website.

First I installed Google Webmaster Tools hoping that Google will tell me that. Unfortunately, the Indexed Pages tab just points me to use site: command.

I don’t trust site: command. Especially, the count of number of pages is very inaccurate. I know I have 100k pages and Google tells me I have 150k pages indexed.

Also, there is no easy way to see more than 1000 pages (you can play with inurl: commands but it takes ages and you can get banned).

Because of these problems I decided to code a tool which would automate it – Site Diagnosis.

The internal algorithm of the tool works as follows:

1. Go through every URL in XML Sitemap file and do a simple check inurl:http://www.samplesite.com/dir/url1
2. For every url that does not appear on inurl: command there is still a chance that page is indexed but does not appear with inurl
3. For every url in XML Sitemaps I get a title and perform this check site:http://www.samplesite.com sample title
4. If the page does not rank in top 10 for its title within the site then probably something is wrong.

This check is suprisingly accurate and most of the pages that don’t survive this check have some problems like duplicate content, missing titles, missing content or not enough content. These troubled pages usualy don’t appear in the search results if you search for any text on the page – not even when you enclose sentences in quotes.

Obviously, the goal of search engine optimization is to fix these pages so Site Diagnosis will hopefully be essential in identifying them.

When you use that filter and see the search result pages then you will see that under every listing there is a date which shows you when Google saw that page for the first time. We can then use this information to find how old the page is if we craft the query to include the url of the page we are interested in.

inurl:http://blog.linkdiagnosis.com/?p=16

For that query it returns the date when my blog post about Amazon was created – 15 February 2008 – which is spot on.

Now to make it even easier to get the age of the page I have created a new feature to Link Diagnosis Firefox Extension.

Install the extension and you will be able to find out the age of the page with one click.

See the screenshot below:

Enjoy!

P.S Don’t do many queries at once otherwise Google will ban you (for me after about 70 inurl commands with 1 sec delay). If anybody has any tips how to prevent that from happening, then please let me know and I will code the age of the page to the main report so you can see both metrics – pagerank and age together.

The most important bits from Yahoo patent (and also from Microsoft research paper) are that search engines can and do create a visual model of the page to find out which is the “Most significant element”. Some of the factors that Yahoo confirms in the paper:

• formatting of the text – bold , h1 etc. – GOOD
• tables with data and other grid-like structures – GOOD
• distance from the top and center. The most important ones are near the horizontal center of the page and also above the fold.
• content / links that are at the absolute header/footer of the page – BAD

Microsoft research paper talks more about the fact that Pagerank model is extended where the atomic entity is not the page but page-block. One page can have multiple blocks with each having different semantics and importance.

This information can have serious impact on how the link juice is flowing from the pages and especially the part from Microsoft paper affects that. Most of the bought links are not placed in the “Most Significant Element” of the page – which is the main content bit. The bought links are placed in side bars, footers etc. These research papers say that these links are much less important than the links which are within the content block. This then just rises the value of bought content links of PayPerPost style where they are within the context of the blog post.

Another implication of these research papers confirms that search engines are able to reconstruct the HTML Document Object Model and know which content is important regardless of the position in the source code of the page. This contradicts with many SEO Experts which have been claiming in the past that you should put the most important content first just after the BODY element.

As always, your comments are welcome.