Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Simple question: I wonder if I'm over internally linking?
Hi, What's you guys' policy on how much to internally link. I do it a lot - whenever it makes sense, but hold off if I just linked to the same page in the last paragraph, for instance. Would like to know your thoughts to see if I'm overdoing it. This is for Ecommerce blog posts, category descriptions, and product descriptions if that matters. Thanks.
On-Page Optimization | | BobGW0 -
Duplicate content question
Hi I have a site that is run off one CMS system but has 3 different web addresses. One is a comic shop, one is a toy shop and one is a game shop. Now due to the nature of what we are selling some of the products we are selling on both or all 3 of the sites. I was wondering as to whether this would affect my ability to rank in google and if i would be penalised for any duplicate content? Thanks in advance
On-Page Optimization | | TheZenAgency0 -
Usability-question on 2 of my websites
Hi all, It has been a while since I have been in this forum, but that's normally a great sign 🙂 Anyway these days I'm work on a "face-lift" on a couple of my sites (www.krydsord.dk, www.bingoforum.dk and www.alutagrender.dk), but also this face-lift should be more than just that. Hopefully it will help the usability to be better, and as a result of that - I hope it will make it easier for my visitors to easily find what they are looking for. The question is: I would like to ad, say, 25 words of description (mouse-over) in my top menu buttons. Why? Because, I believe that it's a possible problem that the menu just says "krydsord" (crosswords) in stead of a describing text saying something about "Push this button and you will see a complete list of all the cross words on Krydsord.dk). This way, I get the chance to add more text (for the Search Engines) and to guide my visitors in a way that I'm not capable of with the limitations my top menus give me as they are at this moment. Naturally, my visitors should be able to easily decline this "extra information", once they have got it once. Is this completely way off, or do you like the idea? More specifically: Will this be annoying for the visitors, will it have any positive effect on SEO matters? Actually I got partly inspired during a SEOmoz webinar (http://www.seomoz.org/webinars/designing-for-seo), even though it's not exactly what they are talking about in thwe webinar. For anybody who are curious (or just need to kill some time), you can have a look from 19:00 and watch it for about two minutes. Thanks a lot! Nicolai
On-Page Optimization | | MPO0 -
Internal Linking Question(s)
Is it unwise to link internally to a page more than once on the homepage. I am reading that it is considered spammy. I am also reading that it passes PR twice to the internal page instead of just once... Which is it? Is there a way to stop passing PR to the "contact us" page. I watched an older video that Matt Cutts suggested a nofollow. Now I read that this strategy is a no no? Which is it? Thanks! 🙂
On-Page Optimization | | JML11790 -
Rel canonical tag question
Im trying to to fix my duplicate content problem with my catagory pages in my shopping cart. I have read about adding a rel canonical tag to the page so it links back to the main catagory page. So if I add a rel canonical tag to the main catagory page it will show up on every other page for that catagory like page 1 page 2 and so on and it will have the tag linking back to the main cat. That should fix it it right? Now that being said I cant seem to add the tag invetween the head tags. I can add it to the body where I can add content. Will the rel canonical tag work outside the head tags? Any other ideas on this fix? I contacted my people that host the cart to see if they have any features to help this will see what they say.
On-Page Optimization | | Dataken0 -
Question about URLs
Hello! I have a client that wants to upload an URL like this: www.example.com/keyword/page-name.html The main problem is that www.example.com/keyword/ doesn't exist and gives a 404 error so I'd prefer not doing that...... What do you think about this? And if the client wants to go ahead, is there any solution? A 301 to the final page would help? Thank you in advance!
On-Page Optimization | | Juandbbam0 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0