Using Robots.txt
-
I want to Block or prevent pages being accessed or indexed by googlebot. Please tell me if googlebot will NOT Access any URL that begins with my domain name, followed by a question mark,followed by any string by using Robots.txt below. Sample URL http://mydomain.com/?example User-agent: Googlebot Disallow: /?
-
Not sure if that would work, but you can test by changing your robots.txt and running a test in GWT > Health > Blocked URLs
You might also be interested in specifying specific URL paraments (e.g. /?sort=name&order=asc > can block sort and order parameters) from within GWT (Configuration > URL Parameters)
Learn more about parameters - https://support.google.com/webmasters/bin/answer.py?hl=en&answer=1235687
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Correct use of schema for online store and physical stores
I have been getting conflicting advice on the best way to implement schema for the following scenario. There is a central e-commerce store that is registered to it's own unique address which is "head office". There are a few physical shops each of which has their own location and address. Each shop has its own landing page within /our-stores/. So each page on the website has the Organisation schema for the central 'organisation', something like: Then on each physical store landing page is something like the following as well as the Organisation schema: Is this correct? If it is should I extend LocalBusiness with store URL and sameAs for GMB listing and maybe Companies House registration? It's also been suggested that we should use LocalBusiness for the head office of the company, then Departmentwith the typeStore. But i'm not sure on that?
Technical SEO | | MickEdwards0 -
Robots.txt Disallow: / in Search Console
Two days ago I found out through search console that my website's Robots.txt has changed to User-agent: *
Technical SEO | | RAN_SEO
Disallow: / When I check the robots.txt in the website it looks fine - I see its blocked just in search console( in the robots.txt tester). when I try to do fetch as google to the homepage I see its blocked. Any ideas why would robots.txt block my website? it was fine until the weekend. before that, in the last 3 months I saw I had blocked resources in the website and I brought back pages with fetch as google. Any ideas?0 -
Robots.txt and Magento
HI, I am working on getting my robots.txt up and running and I'm having lots of problems with the robots.txt my developers generated. www.plasticplace.com/robots.txt I ran the robots.txt through a syntax checking tool (http://www.sxw.org.uk/computing/robots/check.html) This is what the tool came back with: http://www.dcs.ed.ac.uk/cgi/sxw/parserobots.pl?site=plasticplace.com There seems to be many errors on the file. Additionally, I looked at our robots.txt in the WMT and they said the crawl was postponed because the robots.txt is inaccessible. What does that mean? A few questions: 1. Is there a need for all the lines of code that have the “#” before it? I don’t think it’s necessary but correct me if I'm wrong. 2. Furthermore, why are we blocking so many things on our website? The robots can’t get past anything that requires a password to access anyhow but again correct me if I'm wrong. 3. Is there a reason Why can't it just look like this: User-agent: * Disallow: /onepagecheckout/ Disallow: /checkout/cart/ I do understand that Magento has certain folders that you don't want crawled, but is this necessary and why are there so many errors?
Technical SEO | | EcomLkwd0 -
Have you used the Addthis Widget and had your rankings drop
Just wondering what experience people have of this widget? I added it to a site I look after late last week and we took an noise dive over the weekend. Any thoughts / comments are appreciated!
Technical SEO | | RodneyRiley0 -
Linkedin how to use it to promote your business
Hi i have been told about linkedin and how good it is but every time i look at it, it puzzles me and i am not sure if it is worth joining or not. I am looking to promote in the UK and not abroad and would like to know how good it is. Even my accountant uses it but each time i look at it i cannot get my head around it and how to use it to promote my business. Can anyone please let me know how much it cost to join and if it will have any benefits for me to promote my business and my sites. I look forward to hearing from you
Technical SEO | | ClaireH-1848860 -
Using symbols in the html title of a webpage
If you a symbol in the title of a webpage will this dilute the keywords in the title
Technical SEO | | mickey11
thus making it rank worse in search engines here is an example <title><br /> Black Shoe Polish<br /></title> versus <title><br /> ▶ Black Shoe Polish<br /></title> will the extra symbols count as words and thus the dilute the effectiveness of the Black Shoe Polish keyword. sort of making like 4 words instead 3. By the way, The reason to use a symbol is to make it stand on in the search engine results0 -
Is anyone using Media Temple?
I'm looking to move 5 of my sites from Hostgator's shared servers to Media Temple's dedicated virtual servers. Anyone have experience with (mt)? I'm planning on adding a few more sites this year and several things they offer are attractive to me: A (virtually) dedicated environment: Faster websites, better user experience, plus I like having some control over my site's resources Scalability: I can add more resources easily (although not super cheap) Unique control panels for each site: More control for my tech savvy clients. Unique IPs for $1 a month: More linkjuice between my related sites. $50/month is a big jump from my $12/month Hostgator account but I'm thinking it will be worth it. Am I on the right track or is this a fool's errand?
Technical SEO | | AaronParrish0 -
Using Thesis as blog platform vs. Tumblr
I read a lot of advantages by using Thesis as a platform for blogging, but I like the themes and other plugins from Tumblr. Are there equivalents at Tumblr to the Thesis benefits so I can go a head and go with Tumblr?
Technical SEO | | HyperOffice0