Meta Robots "Index, Follow"
-
In my MozBar under "General Attributes" it says "index, follow" next to Meta Roberts for one of our client's websites. I've never seen "index, follow" before. I've seen it say "not found." What does index, follow mean and is that a bad thing? I know the reason should be obvious but this site has had a lot of problems and I'm wondering if this is related.
-
Hello!
Index/Follow means that you are allowing search engines to index your pages and follow links through those pages. If you didn't have this tag on the page, search engines would do it anyway. Basically, this meta tag isn't necessarily a bad thing and the search engines would be doing it even if the tag wasn't there.
You can use meta robots to do good things for your site, such as stop the crawling of blog archive pages to limit duplicate content. There's tons of resources here on MOZ about the robots file and I would suggest looking into it. This is a good one: http://moz.com/learn/seo/robotstxt
What type of problems is your site having?
-
Thank you! That was very helpful!
-
What you're seeing is the page's meta robots tag, which instructs the search engine bots to (in this case) index a specific page and follow links on the page. The opposite, "noindex,nofollow" instructs the search bots to not include this page in its index or follow links on the page. You can also use "noindex, follow" so the search engine keeps the page out of its index, but still follows the links on the page. I personally have never come across an example of "index, nofollow" but I imagine that it's possible - probably more of an outdated practice though related to hoarding links.
If you've never seen this attribute before, it may be because the page doesn't include the tag (in which case the bots, by default, assume "index, follow"). And as for whether or not having it is a bad thing - probably not, unless it's a page you really don't want visible in search results.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I pull "Mobile Rankings" in MOZ?
My client wants me to pull mobile-specific rankings through Moz's keyword explorer. Does anyone know if this is possible? Thanks!
Moz Bar | | TaylorAtVelox0 -
What does "Optimal Use of Keywords in Header Tags" really mean?
I get this explanation on the grader: Although using targeted keywords in H1 tags on your page does not directly correlate to high rankings, it does appear to provide some slight value. It's also considered a best practice for accessibility and helps potential visitors determine your page's content, so we recommend it. Over-using keywords, however, can be perceived as keyword stuffing (a form of search engine spam) and can negatively impact rankings, so use keywords in H1 tags two or fewer times. To adhere to best practices in Google News and Bing News, headlines should contain the relevant keyword target and be treated with the same importance as title tags. But I have the keyword in an H1 tag, only once. So why am getting this message on the grader? The URL is https://www.armsecuritysystems.com/business-security-systems/ The keyword is "business security systems" Any insight would be greatly appreciated.
Moz Bar | | bizmarquee1 -
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this: User-Agent: rogerbot Crawl-Delay: 5 Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop. Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
Moz Bar | | AllAboutShapewear2 -
Zendesk robots.txt
Hello! We have a Zendesk support site at support.zspace.com - our Moz crawl report is saying that there is 85 temporary redirect issues, mostly coming from our support site. My question is, does the Moz crawler use robots.txt? We have currently Disallow: /search in our robots.txt, but not /search/* Most of the temporary redirect URL's in the crawl error report look like this... https://support.zspace.com/hc/en-us/search/click?data=BAh7CjoHaWRp So would we need to add "Disallow: /hc/en-us/search/*" to get Moz crawler to ignore these?
Moz Bar | | zspace0 -
On Page Grader - We have keyword in Meta Description But does not show up
The on-page grader is not counting/picking up the keyword in the meta description. We have double checked the meta description and re-uploaded the page, and checked the source code on-line. The keyword is there in meta description, but not showing up in on-page-grader. https://www.trustdeedrealestateinvesting.com/trimarkfunding/buy-first-trust-deeds.html keyword buy first trust deeds
Moz Bar | | Manifestation0 -
Clarify "broad keyword usage in page title"
Hello Page grader has two different grades for page title that I want clarification on. There is "Broad Keyword Usage in Page Title" and "Exact Keyword Usage in Page Title". Googling around about and searching here I have found that "broad" seems to mean the keywords should be used throughout the page, rather than just in the title and header. Which makes sense as this is a kind of check to ensure the page IS about the keywords and not something unrelated. But what is meant by "broad" usage in the page title? This refers specifically to the page title and not the whole document. My best guess got me to this, given the keyword "Visit London Today"; "Come and visit London today" - exact match only "London - visit today" - broad match only "Visit London the city of dreams | visit London today" - matches both That could be complete nonsense, but basically is broad usage the use of keywords scattered in the page title? Thanks.
Moz Bar | | yolkcreative0 -
How does an index page have a higher Authority than the root domain?
So just curious, but on my domain, http://www.bulwarkpestcontrol.com the Page Authority is 59 and the root domain authority 52. That seems odd as it is the same page. Explanation?
Moz Bar | | Thos0030 -
Moz "Crawl Diagnostics" doesn't respect robots.txt
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like: Duplicate content Overly dynamic URLs Duplicate Page Titles The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Moz Bar | | Vitalized
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored): Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/ Many thanks for any info on this issue.0