Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
Magento Duplicate Content Question - HELP!
In Magento, when entering product information, does the short description have to be different than the meta description? If they are both the same is this considered duplicate content? Thanks for the help!!!
On-Page Optimization | | LeapOfBelief0 -
Home page optimisation question - Expanding box
Hi guys, I was wandering if anyone can help me on how google looks at expanding boxes now? What I am referring to is on our home page orderblinds.co.uk we have an article written which shows a taster of the information about the company, the user then has to click read more to expand the box and see the rest of the content. Is this bad for seo as when you view the html all the content is there but I'm sure google can work out that this text isn't visible until you click read more? Any feedback on this subject would be great,
On-Page Optimization | | OrderBlinds0 -
Internal Link Title and Anchor Text Question
So, I am kind of a newbie with all of this stuff, so please bear with me! If you have a chance to look into the domain it's www.noip.com I have checked the internal followed and 301 links. It seems like it is pulling in all of our navigation a funky way. all of those link have no data for the title and the anchor text is Home, Contact Us, Forgot Password?, ETC. Is this correct? I have tried to look into the way other websites are doing it and it seems wrong. I tried to talk to my developer, but he is not so sure. Do we simply need to make more copy on our site that has internal links within the copy with anchor text that is for keywords that I would like to be ranking for? Like Managed DNS? and then that page goes to the managed DNS page? Or what? Sorry if i am not being totally clear. Any help and advice is 110% welcomed!!! Natalie
On-Page Optimization | | vwnatalie0 -
Using Transcription Service For Videos - Have Question Around Search and Spiders
Hi All, So I have put together a weekly video series on security topics. I have read an SEOmoz post around how you can boost SEO by adding the transcription to the page, which makes perfectly good sense. My question is, can I include the first couple of paragraphs and then have a "read the full transcription" so when the user clicks, the rest of the content appears? Do the search engine spiders only crawl the first two paragraphs in this instance or do they crawl the whole thing even though the entire content is not on the page? I dont mind making the page longer and including the entire transcription if it is easier for SEO but if there is no difference, than I think the first option would be the best user experience. Thanks for the help Pat
On-Page Optimization | | PatBausemer0 -
Can duplicate content issues be solved with a noindex robot metatag?
Hi all I have a number of duplicate content issues arising from a recent crawl diagnostics report. Would using a robots meta tag (like below) on the pages I don't necessarily mind not being indexed be an effective way to solve the problem? Thanks for any / all replies
On-Page Optimization | | joeprice0 -
This is a real dumb question about on page optimization
I've been reading the docs for using this site and am puzzled by the on page optimization reports, and the tactics for creating the reports. In no place can i find that you first must use the keywords you have identified to track in the campaign on pages on your web site. Am i missing something? From just reading how to do your campaign, it would appear that you never have to touch your pages with the keywords you want to track, all 500 of them for one page, if that's what you want to do. Not knowing how the application works, it is possible to imagine anything, including not needing to do the obvious steps of writing a well optimized page first. I want to presume that first you optimize your pages with the keywords you want to track, then go ahead and run your campaign. Although this is the obvious thing, I don't see that part of the seo puzzle being mentioned in any of the docs. Where do i hunt for that part of the equation?
On-Page Optimization | | highersourcesites0 -
Url question for seo
Would it be beneficial to use url that is a match for my keyword to help with seo, then have my currently url forward to that one so I don't have to change any marketing materials? I was one of the feedback that I got when doing the on page keyword optimization tool on seo moz. Thanks J
On-Page Optimization | | fertilityhealth0