Until around August this year the traffic had being growing nicely on MTF and I was getting around 6000 hits a month. Since that point traffic has declined considerably and last month (October) hits had almost halved. Google Analytics in August was indicating 57% of traffic from search engines (mostly Google)which is down to 30% in October. Clearly something is not right so I started some lengthy research to try and track down whats happening.
Now a simple answer would be that no one is searching relevant keywords on my site since August but I think you would agree that is unlikely. So it has to be something else.
During my research I discovered that although Google webmaster tools will indicate a page is indexed on Google this does not mean it will show up in search results (SERPS) believe it or not they appear to have TWO indexes the main and a supplementary(supplemental). I even found the search query to find what is in the two indexes.
The results shocked me but perhaps help explain why suddenly the search traffic is drying up. Out of 518 pages that are indexed by Google for MTF only 41 show up in the main index!
So why I wonder has this happened? Why is MTF mainly in the second eleven. Possible answers would of course include rubbish content-which might well be true- but surely I reason there are a few more pages that are worthy of reference.
Now during my search for answers I have also looked at lots on Search Engine Optimisation(SEO) and again perhaps MTF is just not search engine friendly since I know naff all about SEO. Which brings me round to a possible reason that might be linked to this. My Google Webmasters Tools(GWMT) has recently started showing a single 404 error (page not found).
The URL of the page is as follows: http://www.my-thai-friend.com/ps/rpc_relay.html now I haven't a clue what this missing URL is BUT according to GWMT 100 of my blog pages link to it and of course its missing. Now the 100 pages that link to the missing page, although previously in the main index are now gone!
There is little intelligible information on the web regarding the missing file string but some correspondents suggest it is linked to the followers widget and Google friends connect. As a temporary measure I have removed this widget from MTF to see if a future crawl rectifies things (I doubt it).
BTW if you don't believe me regarding two indexes (Google say they only have one) try this little search in Google.
1. site:www.yoursite.com-this will give you all the pages indexed by Google.
Now just as you are feeling pleased with the return try this search.
2. site:www.yoursite.com/* this gives you your pages in the main index.
Take the total of 2 from 1 and you have the amount of pages in the Google bin oops sorry the Google supplemental index.
So over to you, has anyone got any ideas why I appear to be in the Google bin or how to get out of it or indeed solve the mysterious URL that so many pages link too.
If you enjoyed this post why not subscribe to my RSS feed
22 comments:
Mike, a single 404 isn't going to do this so that can't be the problem.
While there is a supplemental index I don't think it works in such a way. I think the first site: shows all the possible posts and pages and internal links to them in the index and the second site:/* shows the actual number of posts and pages indexed.
Using the regular site search I show 770 indexed but I don;t have 770 posts and pages. Using the second site search it shows 350 which is how many posts and pages I actually have.
A few things could be affecting your search ranks. Google sometimes dances and sometimes that dance lasts a while or you may have been penalized.
I don't know how you do things on the back end or if you are using keywords or not but if you have used too many keywords they might think you are stuffing.
There is also the chance that you have just lost rank for keywords you ranked well for before. Google Analytic s should be able to tell you more specifically where you are ranking for keywords and where you may have lost ground.
Talen thanks for the info. I have been working on this again this morning and when I look at the cached pages from the 100 pages that link to the 404 error I find that in the ones that are cached that the widget for followers/friends connect is reporting as broken on the cached screen shot.
What do you mean by the back end-I have noticed you refer to it before-sorry to be thick?
BTW I get different figures for your site. Am I using the wrong search strings?
Talen I almost forgot can you tell me why the /* only shows 41 pages for MTF? Since I have 518 on the blog. Is it just that 400 odd pages are naff content?
Mike, by the back end I mean when you are behind the scenes in the back of the blog writing your post and setting it up before you post it.
I see 301 pages indexed using site: for your blog and 44 pages using site:/* so if any are in the supplemental it is only 44. But I don't believe that is the supplemental index. When I check my stats the stuff in the site:/* is actually ranking well in most cases.
As for the pages linking to the widget showing a 404 I'm not really sure but those pages are fine so that shouldn't be the problem.
You will get different figures because google is made up of many data centers and depending on which data center you go through the figures might show differently.
Have you checked keywords through Google analytic s? and if so what did that bring up. Analytic s will also tell you a great more in depth information on how you are doing.
Create a page at http://www.my-thai-friend.com/ps/rpc_relay.htm
In it have links to all the pages that are supplemental.
The percentage of pages that go supplementary is usually in direct relationship to your sites overall authority.
But you can usually get them out of the supplementary index by making links to them, therefore raising their importance in googles eyes.
Mike if it is any consolation my site hits dropped around 50% a couple of months back with my most searched posts dropping out of Google searches. I know nothing about SEO, so I did nothing and now BTMJ hits are back up again with the dropped posts ranking high once more. My site:www.thaisabai.org/* search revealed all posts and believe me I get 404 errors as regularly as I get junk mail. I think patience and time are the keywords to solve your problem.
Mike, The more I look at it this might be a duplicate content issue. Google pretty much sends duplicate content, stolen content, pages without content and pages that haven't been linked to into the supplemental bin.
The duplicate content might be stemming from having the post, having the feed of that post and having the category link of that post.
You can stop all the duplicate content through the robots.txt by disallowing certain content from being crawled. Such as:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
I'd have to look at mine to see what else I disallow.
Deano thanks for the advice. I have since discovered the issue is with the followers/friends connect gadget from Blogger. Nor sure you method will cure the root cause.
Martyn this is sort of reassuring but I can't quite get my head round a drop from 150 or so Google search hits to 10/20 almost overnight. Guess I should suck a lemon and forget about it.
Talen thanks for all your input. You may be right but unfortunately with Blogger you cannot alter the standard robot txt file they use. It covers archives/labels and search. I believe you can make meta tags to do the same function.
I don't really understand the ins and outs of your problems, but I do know that Google seems very erratic in how it treats blogs. Mine seems to fluctuate happily between no page rank whatsoever (not even 0) and 4. I have absolutely no idea why. Somebody did once say it could be the supplemental index/duplicate content but I've changed nothing (because I'd no idea how) and at the moment I'm back to PR4.
Hi there, hubby (Ricky) alerted me to your problem. At first look it looks (sorry for the double look ;-)) you have a double content problem. I am a bit tired just know, but I bookmarked your blog and will look into this tomorrow a bit more. As a first measure I gave you a PR4 backlink from my site, that should give you a bit more authority in Google's eyes ;-) BTW, the term is "sandbox" not "bin", try also read up all what you can find about "double content" especially in relation to Google blogs, see ya tomorrow, SY
hospitalera-thank you for taking the time and trouble to help. I really appreciate the link.
I didn't realsie you were Rickys other half if you will forgive the Anglo Saxon pun.
You are the second person to mention duplicate content-very interesting.
I will follow your advice and do a bit of reading. Thank Ricky too please.
Sheila-Hi, funnily enough my PR here has stayed at 3 (having briefly flirted with 4) but I fancy thats because of the incoming links I have from people like yourself and hospitalera
I think both Talen and hospitalera may be right regarding content but I don't understand (yet) how stuff might be seen as duplicate content since all my articles are original.
Replacing the followers widget with the friends connect/followers widget seems to have cured the
http://www.my-thai-friend.com/ps/rpc_relay.htm issue sinec the page is no longer showing 404.
Unfortunately the crawler when it stopped by found 2 more 404's not related to the above!!
Equally unfortunately caring out the same procedure on mythaiphotoblog didn't get rid of the same fault there.
Bugger!
Ok, I am back, have a look at the following two urls from your blog:
http://www.my-thai-friend.com/2009/11/in-google-bin.html
http://www.my-thai-friend.com/search/label/Blogging
Do you see how content overlaps? That is what Talen and I mean by double content. Now, I am not by any stretch of imagination an expert for the blogspot platform, the few niche blogs I have on it are set up far more simple then this blog. So you have to do some research yourself into how to set your labels noindex and how to show Google how you want your site be indexed. I hope you have submitted a sitemap to Google? It might be also a good idea to put a link to your whole sitemap in your top navigation menu and noindex everything else (categories / labels etc) that should get rid of the double content problem. Then take some time and link your blog posts to each other by creating relevant text links like "as I wrote here about TEFL teaching in Thailand" using relevant keywords as an anchor text. Don't overdo it with that but create some deep-links between your blog posts over time.
Bad Neighborhood links:
Check also where are you linking to, you have an awful lot of links / banners in your footer area that go out to directories, some of them might be really good sites, but some of them might be little more then link farms. Google doesn't look kindly if you link to the wrong places. Hope that helps a bit, I will check back and see how it goes, SY
PS Have also a look at http://www.mattcutts.com/blog/ he is the ultimate authority when it comes to things like this ;-)
hospitalera, thank you so much for taking the trouble to help.
Your first point I have covered since by default on Blogger a robot txt file prevents access to labels along with search results.(found that out through Analytics/GWMT).
I have several site maps all of which show indexed pages.
I do internally link but probably could do more so I will work on this.
I agree re links at bottom of page I have already started work on this and will try and weed some out since I guess by inference I share page juice with them too for little or no return.
I have been doing quite a bit of research today and have found many of my blog posts quite highly place in SERPS but they are there because of third parties. OK they link back to MTF but my original article is nowhere to be seen.
Some of these sites have just scraped my content.
I read Matt Cutts but sometimes find it a bit technical for my old brain!
Once again many thanks.
Regards
Mike
Soooo, can i be cheeky and ask that you make a post on Bucketlist Blogs when you figure it all out? You are pretty much my top source when it comes to Blogger!
Mike,
Didn't you previously have a Google Page Rank of 4? I'm sure you did. I'm seeing a 3 now. That does make a difference to traffic.
Hi Gary, yes I briefly flirted with PR 4!
Interestingly I read an article this week about Google dropping PR altogether. It no longer features in GWMTools.
My concern and hence the post is that almost overnight (beginning Sept) I am getting little or no search traffic from Google-used to get at least 100 hits a day this way. PR has remained static.
Fortunately referrals from links on other sites and a solid reader base mean that although I have lost around 50% of my traffic its not too grim.
The main downside is AdSense clicks which inevitably come from search engine rendered hits. Although I have suddenly started to get a few Yahoo hits!
Please keep us up to date how it goes and drop me a comment / email if you need any further help, SY
Emm I have done part of what you requested on Bucket List Blogs. More later!
Hospitalera I tried to mail you but the address on your contact page didn't work. I left a comment on your blog.
Thanks for your continued assistance.
@The TEFL Don
I emailed you back ;-) How are things going, any better? SY
Mike,
If you submit a blog site map, Google only ranks 41 pages.
Yesterday I found out that my rss site maps weren't working at all so I did a bit of research and found this very useful site and since today my posts are indexed.
Good luck and SEO is a never ending story, I am just scratching at the top of an immense iceberg!
Camille, thank you. I have been doing this for a while but to no avail with MTF which is definitely still suffering a Google penalty STILL!!
I even complained to Google who noted my email.
Post a Comment