Robots.txt - blocking JavaScript and CSS, best practice for Magento
-
Hi Mozzers,
I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento.
I'm concerned we are blocking bots from crawling essential information for page rank.
My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not?
You can view our robots.txt file here
Thanks,
Blake
-
As Joost said, you should not block access to files with help in the reading / rendering of the page.
Looking at your Robots file, I would look at the following two exclusions. Do they block anything else that runs on a live page that Google should be seeing?
Disallow: /includes/ Disallow: /scripts/ -Andy
-
Best practice is not to block access to JS / CSS anymore, to allow google to properly understand the website and give determine mobile-friendliness.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best practice recommendations for enabling multiple languages on your site?
I find that the advice for multi-language sites is always tied with multi-region, but what about US only sites that want to be multi-lingual? What are the best practice recommendations there? HREFLANG tags necessary? TLDs? Do you need to purchase yoursite.us , yoursite.sp , etc.. or would yoursite.com/en yoursite.com/sp suffice? Should the extensions be region based even if the language is the only difference?
Intermediate & Advanced SEO | | emilydavidson0 -
Best Practice for setting up expert author contributing to Multiple Sites?
If a single author contributes to multiple sites, should each site have its own author page (tying to the same single gg+ account)? Ex. One author > one gg+ account > multiple author pages (one per site) Or, should all sites publishing his content link to a single author page/bio on a single, main site? Ex. One author > one gg+ account > a single author page on one site (all other sites link to this author page) In this event, where would the 'contributor to' link point for the additional sites he is contributing to, the homepage? Thanks!
Intermediate & Advanced SEO | | seagreen0 -
Eliminate render blocking javascript and css recommendation?
Our site's last Red flag issue is the "eliminate render blocking javascript and css" message. I don't know how to do that, and while I'm not sure if I could spend hours/days cutting and pasting and guessing until I made progress, I'd rather not. Does anyone know of a plugin that will just do this? Or, if not, how much would it cost to get a web developer to do this? Also, if there is not plugin (and it didn't look like there was when I looked) how long do you think this would take someone who knows what they are doing to complete. The site is: www.kempruge.com Thanks for any tips and/or suggestions, Ruben
Intermediate & Advanced SEO | | KempRugeLawGroup0 -
Pluggin to minify CSS
Can somenone please recommend a good pluggin to minify css upload so that my web runs faster as it uploads more than 20 css and i want to placed it all on the same css
Intermediate & Advanced SEO | | maestrosonrisas0 -
Are articles still benificial and how best to promote them?
Hello, I'm trying to promote a new site doing things differently moving forward if needed in order to prevent getting google slapped while being as efficient as possible.. We have a main site which manufacturers materials. we also have a blog on blogger.com every week someone in our office writes an article about something related to our area of work and within the article has a varied keyword or two embedded within the article they are writing... My questions are as follows: -1- should be change our blog site address from oursite.blogger.com to blog.oursite.com?
Intermediate & Advanced SEO | | Robdob2013
-2- would it be beneficial to have a link from our main site to the oursite.blogger.com
-3- We also have a ezine account, would it be beneficial to also post this same article perhaps with some minor changes to our ezine account so that it would start to get more visibility from other sites or is this now possibly a no no?
-4- should we be now usin nofollow links in our articles? if we do use nofollow links aren't we losing the benefit? Any suggestions would be greatly appreciated0 -
About robots.txt for resolve Duplicate content
I have a trouble with Duplicate content and title, i try to many way to resolve them but because of the web code so i am still in problem. I decide to use robots.txt to block contents that are duplicate. The first Question: How do i use command in robots.txt to block all of URL like this: http://vietnamfoodtour.com/foodcourses/Cooking-School/
Intermediate & Advanced SEO | | magician
http://vietnamfoodtour.com/foodcourses/Cooking-Class/ ....... User-agent: * Disallow: /foodcourses ( Is that right? ) And the parameter URL: h
ttp://vietnamfoodtour.com/?mod=vietnamfood&page=2
http://vietnamfoodtour.com/?mod=vietnamfood&page=3
http://vietnamfoodtour.com/?mod=vietnamfood&page=4 User-agent: * Disallow: /?mod=vietnamfood ( Is that right? i have folder contain module, could i use: disallow:/module/*) The 2nd question is: Which is the priority " robots.txt" or " meta robot"? If i use robots.txt to block URL, but in that URL my meta robot is "index, follow"0 -
Temporary Redirects on Magento
I've recently taken over a client who uses the Magento platform and there was definitely a duplicate issue with his homepage. It redirected www to non www, however the canonical tag was setup wrong and pointing to the www version. When I looked at OSE for both versions the non www has only 7 linking domains and a page authority of 32. The www version has 24 linking domains and page authority of 39. As the domain is fairly new, I decided to redirect the non www to www and keep the canonical the same. (I changed the internal linking structure etc). When I run both URLs through this tool: http://www.ragepank.com/redirect-... it's returning a whole bunch of 302, rather than 301 redirects. What's the deal with that? Is that a Magento setting that I can fix or something a little harder? I'm not sure if it's proper etiquette to post the URL of a client, so if that would help and is OK, please let me know. Thanks
Intermediate & Advanced SEO | | bradkrussell1 -
What would be the best domain choice?
Hello I got a website www.keywordCA.com and I'm ranking #1 spot on "keyword" but what I notice if you have the exact match you get more site links and etc. Like this keyword that match with my domain name "keyword CA" The ideal name will be www.keyword.com but is taken and the owner don't want to sell the domain (at least he is not using it, is just parked) and I also got the domain www.keyword.net Do you think www.keyword.net will be much better than KeywordCA.com in order to get more exposure and google will generate more site links?
Intermediate & Advanced SEO | | jpgprinting0