The Scenario:
The day has finally come when you've decided to upgrade your website to the latest and greatest content management system and website design. It's a big decision to make that leap considering how much time it will take and the cost to do it. It might be months before the new site is live.
You're going to have a lot of things to worry about during the process of setting up the new site. Difficult design issues, photography concerns, new content creation, and of course learning the new CMS. By the time you get to the launch date, you're going to be exhausted by the entire process.
That's when it happens. You switch over to the new site and your ranking in Google search completely tanks!
Google suddenly hates you and you have to wonder why you invested all that time and money in the first place. Your phones have stopped ringing, emails stopped coming in from the website, and even your foot traffic has dropped off all because your Google ranking vaporized.
You wonder what went wrong and someone suggests that you probably should invest in some online advertising until you figure out how to get your organic ranking back.
It all seems like a waste of money and now you need to spend even more.
The Discussion:
This same scenario has played out many times, and I've heard this story from several jewelers who called me in the midst of a panic attack during the week after a new website launch.
Although the situation looks grim, and the financial repercussions certainly feel grim, there is plenty of hope.
The first thing to realize is that Google knew about your old website. They've spent the last few years reading your site over and over again and figuring out what your business was all about and how your code was structured. They also knew how to match your pages to search queries.
Google's index has several complicated levels of importance. I've noticed that one of the most important ranking factors is the uptime of your website. Their spider has the ability to test your web pages extremely rapidly, but there seems to be a huge difference between the Google spider "pinging" a page for a test, versus when the spider is taking the time to read and digest your information.
It does take time for your web pages to be read and digested by Google Search. Exactly how long varies from site to site, and it could be as little as minutes or even seconds for very important sites (like the news) while at the same time it could take months for Google to index a random personal blog website.
However, once indexed and appearing in search, Google wants to make sure that the pages are always valid. Google hates it when you click on a SERP link to a missing or broken page. Broken pages could be the result of a 404 error, an intentional 410 Gone error, or even just a website that's down for a moment. Either way, Google hates it because they want the best possible search results--and broken links always make people grumpy, right?
So Googlebot races around the internet at breakneck speeds to test pages over and over again just to make sure they are still alive. In fact, to illustrate how ridiculous Googlebot can be some times, over the last 90 minutes, I've watched Googlebot ping my most recently published Nugget 59 times. Googlebot always seems a little more rambunctious every morning right after the Nugget is published, but it eventually slows down.
The results of all this pinging are visible in your Google Webmaster Tools reports if you know where to look, which, of course, I'm about to tell you...
In your Webmaster Tools account, navigate over to the Crawl menu on the left, then select Crawl Errors. That page will show your Site Errors and your URL Errors. Every website usually has a few random errors in the list there. Seek help from your web programmer if you have more than a few dozen errors.
Let me get back to the original scenario above. Your Google ranking bottoms out very quickly when a new website does not have the same URL structure as the old website. As soon as Googlebot sees your pages disappear, it sends a proverbial red flag through the search index telling it that your site is down, and that all your results must be removed.
OMG The Sky Is Falling:
If Googlebot was a person I can just imagine them running into the office for Google Search yelling "the sky is falling because that website is down!" As a knee-jerk reaction, Google Search leans over and presses the delete key on their keyboard... Poof you're gone.
I then imagine a staff meeting between Googlebot and Google Search ensues about a week or so later where Googlebot gives Google Search a new list of web pages. All of your new web pages are in that list, and Google Search will eventually get to adding them back into the results.
Holding Up the Sky:
There's a critical step that you could take, way back on that very first day when you decide to upgrade your site. You could ask your web programmer to create a comprehensive list of every URL on your website. This is not an easy task, and your programmer might need to use a special spider program of their own to figure out the full list.
I'm talking about a much bigger list than what you might create for the sitemap.xml that you upload to Webmaster Tools.
This list needs to include all the pages, product URLs, and everything else you might have on the site. Googlebot knows about all your URLs with variables, so make sure you itemize them too in this big list.
Oh, and if a lot of your organic traffic come from Google Image Search then you need to have a comprehensive list of the URLs for the most important image too. Yeah, that's even harder to do because most spider programs don't know how to track images.
Keep the list handy, because that list eventually saves you from a panic attack.
During the few weeks just prior to the new website launch, you need to map out every single old URL to the corresponding new URL. This task is quite tedious, especially if you have hundreds of pages or thousands of products. Your web programmer might be able to write a script to do the mapping for you, but it still requires some double-checking.
With both lists of pages in hand, the old and the new, your webmaster will have to program a 301 redirect for each old URL.
Outsmarting that Googlebot Ping:
My measurement of Googlebot pings on several retail jeweler websites shows a really sluggish ping rate. Sometimes pages are only pinged 1 time in 2 months. But bigger websites appear to be pinged more frequently. Googlebot warns Google Search as soon as a page error is found.
Remember that Googlebot's pinging is very rambunctious. Don't launch your new website until you have completed the setup process for all the 301's from the old pages to the new. When Googlebot sees the 301 redirect it will follow it to make sure the new page exists. Your website then tells Googlebot that everything is okay and to keep sending that organic traffic.
With the 301 redirect file in place, I can imagine that eventually Googlebot and Google Search have another staff meeting where Googlebot give Google Search a new list of website pages, and a list of web page corrections. Your pages will be in that list and they never get removed from the SERPs.
It only took 160 minutes for the Titanic to sink after hitting the iceberg. That's quite fast for such a big ship. Bigger websites, like big ships, will sink much faster to the bottom of SERPs because Googlebot revisits faster, which means it finds your dead pages faster.
Recovery After a Fallen Sky:
Even on smaller sites, you probably only have about 4 or 5 days before Googlebot figures out you have missing pages, and tells Search to remove them. That's when your phones stop ringing and when your organic traffic will fully bottom out.
After that, you have to wait until all your new pages are re-indexed and added back into the search results. That could be weeks or months.
Sadly though, without the 301 redirecting, you will also lose all the off-site SEO that comes to you via social sharing and links from other websites. You can't get that back without the 301 redirecting. You simply need to start over with a clean slate.
Bitter, Bitter, Can't Be Bothered:
As I wrap up this silly Nugget today I hope my anecdotes will help you realize just how important it is to plan ahead when rebuilding your website.
It doesn't matter how small your website is today, or how little you've done with it over the years. Setting up 301 redirects is your way of saving all the goodwill you have.
Unfortunately, there's one tiny hiccup with this entire process. Unless your new programmer is really good at what they do, you probably have to ask the programmer of your old site to help you create the list of website files. In some cases I've know it might be impossible because that person isn't around anymore, or they are out of business.
Sometimes you might dislike your old programmer or you simply don't want them to know you've hired someone else. For those of you who feel this way, I promise you that the panic attack you feel when business drops will make you feel much, much worse. A good business person won't be bitter that they are losing a client.
Lastly, there's probably going to be an extra expense with all this 301 planning and implementation. Don't simply avoid this process because you couldn't be bothered or didn't want to spend the money. The larger your site is, the more of a chance that your sales will slump while Google Search relearns about your new site. For large sites, the cost of the 301 setup will probably be much less than the loss in sales you would otherwise experience.
