Meta Robots "Index, Follow"
-
In my MozBar under "General Attributes" it says "index, follow" next to Meta Roberts for one of our client's websites. I've never seen "index, follow" before. I've seen it say "not found." What does index, follow mean and is that a bad thing? I know the reason should be obvious but this site has had a lot of problems and I'm wondering if this is related.
-
Hello!
Index/Follow means that you are allowing search engines to index your pages and follow links through those pages. If you didn't have this tag on the page, search engines would do it anyway. Basically, this meta tag isn't necessarily a bad thing and the search engines would be doing it even if the tag wasn't there.
You can use meta robots to do good things for your site, such as stop the crawling of blog archive pages to limit duplicate content. There's tons of resources here on MOZ about the robots file and I would suggest looking into it. This is a good one: http://moz.com/learn/seo/robotstxt
What type of problems is your site having?
-
Thank you! That was very helpful!
-
What you're seeing is the page's meta robots tag, which instructs the search engine bots to (in this case) index a specific page and follow links on the page. The opposite, "noindex,nofollow" instructs the search bots to not include this page in its index or follow links on the page. You can also use "noindex, follow" so the search engine keeps the page out of its index, but still follows the links on the page. I personally have never come across an example of "index, nofollow" but I imagine that it's possible - probably more of an outdated practice though related to hoarding links.
If you've never seen this attribute before, it may be because the page doesn't include the tag (in which case the bots, by default, assume "index, follow"). And as for whether or not having it is a bad thing - probably not, unless it's a page you really don't want visible in search results.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
"New" issues not previously found being shown?
I'm not sure what logic Moz is using for its reporting of Site Crawl issues, but it appears to be pretty flawed (unless I'm missing something, which is possible). I've got a client site that has been in Moz for about 6 months now. Every time the crawler runs, the same number of pages are reported as having been crawled. However I'm consistently getting "New Issues" reported that should have been reported during previous crawls. Example: A redirect chain was reported several month ago. The referring URL was the homepage of the website, and we tracked it down to an old link in the header. This was fixed, marked as resolved, and the issue was not shown on the next crawl. Several weeks later, the same issue was reported for a different page on the website - a page which has existed since 2014 and was already crawled many times. Again, we fixed. Fast-forward to the report that just ran on 12/1 and we have the same issue reported, for a different page, which has also existed for years and has been previously crawled. It's very hard to explain to a client "this item you are seeing has been resolved", only to have it continually crop back up in future reports. Note this is not limited to redirect chains - that's just an example. I'm seeing this for other items such as missing canonicals, duplicate titles, etc.
Moz Bar | | RucksackDigital0 -
Page Grader states "includes Canonical Tag" but it's not in the page source at all
I've ran it multiple times and changed other things it picked up on so not sure where it's getting the canonical tag is included even though it isn't?
Moz Bar | | Wana-Ryd0 -
Why doesn't the Keyword Explorer "Explore By Site" work?
Whatever page/domain url I put in here, I get a message coming up saying ""Getting rankings counts failed" Why does his happen? I can't find anything in the Help about this.
Moz Bar | | mfrgolfgti0 -
Fresh Web Explorer: if you put a keyword in quotes, is it case sensitive?
For example, would a saved search for "racor fuel filter" capture mentions of "Racor fuel filter"? I'm really hoping I don't have to duplicate the keywords with the alternative capitalisations!! Many thanks Steve
Moz Bar | | swbaxter0 -
Zendesk robots.txt
Hello! We have a Zendesk support site at support.zspace.com - our Moz crawl report is saying that there is 85 temporary redirect issues, mostly coming from our support site. My question is, does the Moz crawler use robots.txt? We have currently Disallow: /search in our robots.txt, but not /search/* Most of the temporary redirect URL's in the crawl error report look like this... https://support.zspace.com/hc/en-us/search/click?data=BAh7CjoHaWRp So would we need to add "Disallow: /hc/en-us/search/*" to get Moz crawler to ignore these?
Moz Bar | | zspace0 -
Domain.com isn't recognized by on-page-grader, but domain.com/index.php is
I am running a website through On-page-grader, as www.domain.com and scores an "F" for a specific keyword. When it's ran as www.domain.com/index.php, it scores an "A" for that same keyword and has everything checked other than "keyword in the domain name". There are no other files such as index.htm, or index.html that would interfere and can't figure out why this page is not being recognized. I checked, the robots and .htaccess file, but do not see anything that would hinder. Could this be a server issue?
Moz Bar | | werkbot0 -
Change an "Answered" question to "unanswered?"
Some times Moz marks my questions as 'answered" when they are not. I know that if I mark something as a "good answer" or select "answered" (obviously) it will change, but occasionally (like the last question I asked) it gets changed and I'm not sure how to change it back to "unanswered." Anyone know if that's possible? Thanks, Ruben
Moz Bar | | KempRugeLawGroup1 -
How do I find #of RSS Subscribers", "Most Popular Post URL", "What is the most popular post about"
Hi, I am a new user at Moz.com and looking for finding below information for a list of blogs. "#of RSS Subscribers", "Most Popular Post URL", "What is the most popular post about" for a list of blogs URLs? Please response Which tools I should use and how to use tools? -Muhammad
Moz Bar | | mmhossain4580