Robots.txt - blocking JavaScript and CSS, best practice for Magento
-
Hi Mozzers,
I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento.
I'm concerned we are blocking bots from crawling essential information for page rank.
My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not?
You can view our robots.txt file here
Thanks,
Blake
-
As Joost said, you should not block access to files with help in the reading / rendering of the page.
Looking at your Robots file, I would look at the following two exclusions. Do they block anything else that runs on a live page that Google should be seeing?
Disallow: /includes/ Disallow: /scripts/ -Andy
-
Best practice is not to block access to JS / CSS anymore, to allow google to properly understand the website and give determine mobile-friendliness.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Linking to other peoples you tube videos related to our "How to do " articles on our website. Is there Best practices ?
Hi All, We are currently writing some "How to do" articles on our tool hire website and as there is alot of DIY related you tube videos out there, we thought It would be good to link to some of these at the bottom of our articles. From an SEO perspective, is there any do's and don'ts with regards how we should implement this. We are unable to do our videos so linking to others would be our preferred option. Does anyone know if this would give an SEO ranking benefit even though it's an outbound link to someone's video etc. thanks Pete
Intermediate & Advanced SEO | | PeteC120 -
URL Formatting - Magento
Hi, We are working with a client on Mangento who URLs are formatting Google friendly eg; productname.html - as seen in site search in Google) but when you click the link to the site it is adding on #.VEWKQxbc754 (or similar) The site is also having some page indexing problems as well Thoughts? specific settings/Add on in magento?
Intermediate & Advanced SEO | | Pure-SEO0 -
Best practice for expandable content
We are in the middle of having new pages added to our website. On our website we will have a information section containing various details about a product, this information will be several paragraphs long. we were wanting to show the first paragraph and have a read more button to show the rest of the content that is hidden. Whats googles view on this, is this bad for seo?
Intermediate & Advanced SEO | | Alexogilvie0 -
Title Tag Best Practices
In light of all the Google updates in 2013, have you updated/changed your title tag best practices? Is the format of (Keyword | Brand) still working well for your optimization efforts or have you started incorporating an approach similar to this format . (Keyword in a Sentence | Brand) Thanks in advance for your opinions.
Intermediate & Advanced SEO | | SEO5Team0 -
Massive URL blockage by robots.txt
Hello people, In May there has been a dramatic increase in blocked URLs by robots.txt, even though we don't have so many URLs or crawl errors. You can view the attachment to see how it went up. The thing is the company hasn't touched the text file since 2012. What might be causing the problem? Can this result any penalties? Can indexation be lowered because of this? ?di=1113766463681
Intermediate & Advanced SEO | | moneywise_test0 -
What is best practice SEO approach to re structuring a website with multiple domains and associated search engine rankings for each domain?
Hello Mozzers, I'm trying to improve and establish rankings for my website which has never really been optimised. I've inherited what seems to be a mess and have a challenge for you! The website currently has 3 different www domains all pointing to the one website, two are .com domains and one is a .com.au - the business is located in Australia and the website is primarily targeting Australian traffic. In addition to this there are a number of other non www domains for the same addresses pointing to the website in the CMS which is Adobe Business Catalyst. When I check Google each of the www domains for the website has the following number of pages indexed: www.Domain1,com 5,190 pages
Intermediate & Advanced SEO | | JimmyFlorida
www.Domain2.com 1,520 pages
www,Domain3.com.au 149 pages What is best practice approach from an SEO perspective to re organising this current domain structure? 1. Do I need to use the .com.au as the primary domain given that we are in this market and targeting traffic here? Thats what I have been advised and it seems to be backed up by what I have read here. 2. Do we re direct all domains to the primary .com.au domain? This is easily done in the Adobe Business Catalyst CMS however is this the same as a 301 redirect which is the best approach from an SEO perspective? 3. How do we consolidate all of the current separate domain rankings for the 3 different domains into the one domain rankings within Google to ensure improved rankings and a best practice approach? The website is currently receiving very little organic search traffic so if its simpler and faster to start again fresh rather than go through a complicated migration or re structure and you have a suggestion here please feel free to let me know your ideas! Thank you!0 -
What are best practices for multi-language seo?
We are looking for best practices for multi-language seo. We are interested in any books, message boards or blogs that you can recommend. Thanks.
Intermediate & Advanced SEO | | FareCompare0 -
Website Siloing..best practice?
Hi all I am doing some research this week on the effects of siloing a Magento site. We have about 1,654 pages with approx 1,400 products. We want to silo the website in order to address the internal linking issues and to also focus the customer journey in a more organised way. I need to report all of the possible angles and effects that this will have on the site, prior to implementing it. Does anyone have info on best practice for siloing? I'd appreciate any help... Thanks Nick
Intermediate & Advanced SEO | | Total_Displays0