Every once in a while I come across a website that looks great, functions well, and yet Google and Bing seem to be avoiding the site like the plague. There are times when, despite all your best efforts, it just seems impossible for you to get your website ranked by today's standards of optimization.Today's standards of SEO require you to have a lot of new content on your website. It's a tough job to continually create new content. Writing, taking photos, and creating videos take a lot of time and with all that work you hope that the search engines will reward you with a lot of organic traffic. It's more than depressing when that traffic doesn't come, it also cripples your business.
Let me turn my attention now to a basic explanation of how websites are created today compared to how they were created back in the 90's. Back then, we used to program everything by hand (while riding our dinosaurs). It was so labor intensive that I can remember charging customer according to the number of images on a page, the number of hyperlinks, and the number of times we made words bold. Seriously, that's how we did it. Today there are very few websites that are programmed by hand, and those that are, are probably for academic purposes. The web has expanded at meteoric rates due to the ease of website creation using content management systems.
The "content management system," or CMS for short, allows everyday average people to create websites without any programming knowledge. Each CMS has features built-in to create blogs, product catalogs, forms, membership websites... everything. But sometimes even though these systems make your life easy, they also cripple your business.
Each CMS has built in methods of controlling how Google and Bing read the website. There are settings that tell search engines to ignore the website, and there are settings that allow the search engines to read everything. There are usually settings like this that can be applied on a page by page basis. Setting like this can be controlled using a file called robots.txt that you save on the web server or by setting the "meta robots" tag to "noindex.nofollow" for each page of the site.
Other than settings, there are also features that might accidentally hide your website from search engines, like something called an AJAX script.
An AJAX script is a fancy JavaScript that allows parts of your page to load asynchronously, and without needing to reload the page. You've probably seen an AJAX script running on websites with large product catalogs where the images fade in one after the other rather than suddenly appearing all at the same time. You've also seen AJAX running as you scroll down to the bottom of the page in your favorite social network only to have more posts and photos magically appear.
Although AJAX is a great method of creating interaction with users, most of the time it's a killer for your search engine optimization. There's no guarantee that Google or Bing will be able to mimic the user actions required to make an AJAX script run. If your written words, images, and videos are loaded by AJAX you run the risk of making your website invisible to search engines.
Why would you use AJAX?
In 2011 Google started to experiment with measuring page load times on websites. As of today, they use the speed of your website as a small measurement in your ranking. Websites load a lot faster when parts of the page load asynchronously, or with AJAX, and many programmers have embraced the technology just for that reason without carefully monitoring if their version of AJAX is invisible to search.
So we have 3 potential ways to accidentally prevent your website from being indexed by search engines: AJAX, robots.txt, and meta robots tags. If set up incorrectly, you could have a completely un-crawlable websites.
Here's a simple test to see if Google is reading your website... do a search for your domain name using the "site" command. For example, if my website is www.perosijewelers.com I would search for...
"site:perosijewelers.com"
...without the quotes.
When I do that search right now Google tells me that there are about 11,900 pages indexed. The perosijewelers.com website is one of my test sites with a full product catalog, so that number is about right.
When using the "site:" search command for your own website you should immediately realize that something is wrong if the number of indexed pages does not clearly represent the number of pages that are actually on your site. Too many pages is a different problem than too few pages. Having too few pages means your site is un-crawlable. If you have too many pages then you most likely need an upgraded CMS.
Digging a little deeper into the crawlability of your website will require that you look into Google Webmaster Tools account. Inside there you need to look for the page that shows your "Index Status." At the time of this writing you could find that in the left navigation under Google Index >> Index Status. This provides a nice chart showing how Google had read and indexed your website over the last 12 months.
Hopefully the information in today's Daily Golden Nugget has helped you understand what might cause your website to be blocked from search engine ranking and where to look to find out if it is. I didn't explain how to fix the problem because you really will need to show this to your website programmer and SEO agency rather than trying to fix it yourself.
