Welcome to the Q&A Forum

Thanks for the information. How do you filter by exact match?

I'm guessing broad match... I'm familiar with the concept but is there a way to restrict it to one or the other?

Broad would make sense, it's possible that the rank is different for different combinations.

I've been seeing this more and more... a page that ranks well for a great keyword but the traffic is much, much less than one would expect based on information from the Google Keyword Tool. The keyword gets 3600 searches locally, but the site, #9 on the SERP, got 11 visits last month.

Does that sound like a plausible drop-off or is there something else at play?

Thanks!

Theresa,

I'm going to support the two answers here: creating a new account is not a good use of time.

The main reason, in my mind, is just that you wouldn't have anything to compare the new site with. If you have new code, new content, new everything, it's important to compare the changes with the old site to make sure you made positive changes.

The best example I can give of a scenarios where you'd want that historic data is with any page on the site that already ranks well. After the page changed, did traffic increase? Decrease? Time of page? It's good to know how these were changed.

I'll also add, historic analytic data can help you find problems caused by the move. Disallowed robots.txt from the test site? Structure issues? Many of these can be found out from looking at your analytic data and having a baseline is important.

Hope that helps!

I'm seeing the meta tag that's added for the first option:

<meta name="robots" content="index, follow" />

... but I could actually access a file at domain.com/robots.txt that had the content mentioned above. When I logged in via FTP, it wasn't there. I added an actual file there with the correct information and reloaded it to make sure it was showing the correct information.

I tested it on my local install and I'm not seeing a robots file being generated.

Very odd!

Just make sure you don't set that Privacy setting in a live directory. It takes weeks/months to fully recover.

I should mention that this is a WordPress site and, with that, I may have answered my own question. Perhaps WordPress generates a robots.txt dynamically when the setting is active at Settings > Privacy?

So here's a weird one...

Client comes to me for some simple changes, turns out there are some major issues with the site, one of which is that none of the correct content pages are showing up in Google, just ancillary (outdated) ones. Looks like an issue because even the main homepage isn't showing up with a "site:domain.com"

So, I add to Webmaster Tools and, after an hour or so, I get the red bar of doom, "robots.txt is blocking important pages." I check it out in Webmasters and, sure enough, it's a "User agent: * Disallow /" ACK!

But wait... there's no robots.txt to be found on the server. I can go to domain.com/robots.txt and see it but nothing via FTP. I upload a new one and, thankfully, that is now showing but I've never seen that before.

Question is: can a robots.txt file be stored in a way that can't be seen?

Thanks!

Thanks for the insight, that's really helpful. It sounds like it truly is a trade-off, in some cases you're just trading your ranking for money, not what we really want to do.

Great answer, Simon, thanks for taking the time to type that out. This was the direction I was leaning on this and it's good to hear an honest reply.

Paid links, in my mind, are at the lowest end of the "bad things to do" spectrum. It's still on the wrong side of the tracks but you see so many people getting away with so much, you can't help but to think, "hey, it's not that bad."

I think I've made up my mind, thanks for the push in the right direction!

I work with a large group of community sites and we were approached by a company called Text Link Ads to purchase links on our sites for SEO purposes. This means we can't add "nofollow" and the resulting page is crawl-able. This goes directly against Google's policy but I think it would be tough to find out exactly what we're doing. It could be a great revenue source and this company has many, many sites they work with. That said, this sounds like what you say right before something bad happens.

So, two part question: Has anyone worked with this company before? And should these links be avoided or are they not so bad?

Thanks in advance!

I should mention that this is a WordPress site and, with that, I may have answered my own question. Perhaps WordPress generates a robots.txt dynamically when the setting is active at Settings > Privacy?

Theresa,

I'm going to support the two answers here: creating a new account is not a good use of time.

The main reason, in my mind, is just that you wouldn't have anything to compare the new site with. If you have new code, new content, new everything, it's important to compare the changes with the old site to make sure you made positive changes.

The best example I can give of a scenarios where you'd want that historic data is with any page on the site that already ranks well. After the page changed, did traffic increase? Decrease? Time of page? It's good to know how these were changed.

I'll also add, historic analytic data can help you find problems caused by the move. Disallowed robots.txt from the test site? Structure issues? Many of these can be found out from looking at your analytic data and having a baseline is important.

Hope that helps!