How to fix and issue with robot.txt ?
-
I am receiving the following error message through webmaster tools
http://www.sourcemarketingdirect.com/: Googlebot can't access your site
Oct 26, 2012
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.The site has dropped out of Google search.
-
Hi Stacey
What plugins do you have running - any caching plugins such the W3 Total Cache plugin?
Are you able to access your servers error logs to see if you can see anything there?
-
Thanks for your answer.
I have received this message from Google
**http://www.sourcemarketingdirect.com/ **using the Meta tag method (less than a minute ago). Your site's home page returns a status of 500 (Internal server error) instead of 200 (OK)
It looks like the permalink structure has changed but I'm not sure how.
-
I've seen several people ask this very same question over the last week in different forums. I am wondering if the major outages with hurricane Sandy have affected several hosts or DNS's.
Your robots.txt looks fine to me.
I'm guessing that you will completely recover once Google has a chance to fully crawl the site again.
-
just a quick check you have got wordpress visible to search engines set in the admin area? if not it will be set to disallow googlebot to crawl it.
it is in admin - options - privacy and select appropriate box - default is no index, no follow.
-
Thanks Matt.
There is no robots.txt as far as I can see. Is there a plugin I can use for wordpress?
The site was down for 2 days last month while hte original host transfered the site over to me.
Right now a site search says their are 13 pages indexed.
Just concerned that this site has always ranked number 1 for a company name search and now they are not on the first 10 pages in Google.
-
have you made sure your robots.txt is loading in your browser by adding robots.txt after your domain same as a normal page and can you see contents? has your site been down in this period? have you changed the contents of the file just before this issue? are you sure googlebot hasnt come back since that date - whats your analytics say? do an index site: search for your domain to see if it is in google.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a similar CMS fabric/theme used by 2 colleagues, cause SEO issues?
Does using the same CMS fabric/theme with substantially different content for each website but the same business address (on google places and the websites) cause SEO issues? For example: 2 colleagues with somewhat similar services located at the same business addresses desire to use the same CMS fabric /theme but will have different content on the sites. Will this hurt their SEO / should they use a different website skin/theme?
Web Design | | toti5880 -
Curious why site isn't ranking, rather seems like being penalized for duplicate content but no issues via Google Webmaster...
So we have a site ThePowerBoard.com and it has some pretty impressive links pointing back to it. It is obviously optimized for the keyword "Powerboard", but in no way is it even in the top 10 pages of Google ranking. If you site:thepowerboard.com the site, and/or Google just the URL thepowerboard.com you will see that it populates in the search results. However if you quote search just the title of the home page, you will see oddly that the domain doesn't show up rather at the bottom of the results you will see where Google places "In order to show you the most relevant results, we have omitted some entries very similar to the 7 already displayed". If you click on the link below that, then the site shows up toward the bottom of those results. Is this the case of duplicate content? Also from the developer that built the site said the following: "The domain name is www.thepowerboard.com and it is on a shared server in a folder named thehoverboard.com. This has caused issues trying to ssh into the server which forces us to ssh into it via it’s ip address rather than by domain name. So I think it may also be causing your search bot indexing problem. Again, I am only speculating at this point. The folder name difference is the only thing different between this site and any other site that we have set up." (Would this be the culprit? Looking for some expert advice as it makes no sense to us why this domain isn't ranking?
Web Design | | izepper0 -
301 Redirect Issue for URL with # and !
Hi All, We had a WIX website and now moved to Wordpress. I m having issue while doing redirecting from old URL to new URL. Example: Old Url: http://www.firsttraffic.com.au/#!traffic-management/ccfn New Url: http://www.firsttraffic.com.au/our-services/traffic-management/ I tried different wordpress plugin but nothing works. I m thinking its due to the # . But How can I to redirection for URL like this . Thanks
Web Design | | emarketexperts0 -
Redirects Not Working / Issue with Duplicate Page Titles
Hi all We are being penalised on Webmaster Tools and Crawl Diagnostics for duplicate page titles and I'm not sure how to fix it.We recently switched from HTTP to HTTPS, but when we first switched over, we accidentally set a permanent redirect from HTTPS to HTTP for a week or so(!).We now have a permanent redirect going the other way, HTTP to HTTPS, and we also have canonical tags in place to redirect to HTTPS.Unfortunately, it seems that because of this short time with the permanent redirect the wrong way round, Google is confused as sees our http and https sites as duplicate content.Is there any way to get Google to recognise this new (correct) permanent redirect and completely forget the old (incorrect) one?Any ideas welcome!
Web Design | | HireSpace0 -
W3C My site has 157 Errors, 146 warning(s) Is it an issue?
Is having this number of W3C errors & warnings an issue and will be impacting my site's performance? When the site was built 6 months ago my developers told me that it "was nothing to worry about", but I have read that any errors aren't good, let alone the huge number my site has? Your advice please Thanks Ash
Web Design | | AshShep10 -
New site or fix the old one
I have a delima. Basically the main business product I used to offer is not going to be offered anymore. The types of sales events we conducted for auto dealerships are not able to be insured any longer forcing the change. So I am pivoting to just offering direct mail and I plan on going into digital probably social, landing pages, content marketing and not sure what else. I was able to register http:www.roiautos.com and www.roidirectmail.com both variations of www.roiautosolutions.com withc was the original site. Also that is the closest to the actual name of the business. My question is whether to build a site focusing on direct mail using the direct mail dot com, or just to redo the current site. The current site doesn't have much rank if any because the old product was not something that was searched for. As a mater of fact 99% of my business came from referrals and word or mouth so I just never really bothered. My thoughts are that ROI Direct Mail will work better for search and I am even going to use that as a DBA and TM. But I am unsure of what to do for search. One thing that has to happen is that all references to offering staffed sales events have to be removed from any site per my insurance company. Any advice?
Web Design | | roiautos0 -
Breadcrumbs show for some section and dont come for some. What could be the issue
Hello, We have implemented breadcrumbs - which comes up nicely for Blog and Forum in Google Search However, when we look at our 2 other section - New Car Product and Latest News - it does not comes up, is there any issue which can foresee Sample URL of New Car - http://www.mycarhelpline.com/index.php?option=com_newcar&view=product&Itemid=2&id=8&vid=188 Sample URL of News - http://www.mycarhelpline.com/index.php?option=com_latestnews&view=detail&n_id=474&Itemid=10 Its been over 1 month - since breadcrumbs been introduced in both New Car and News. is there any issue like in them as why is is not showing in search. Any recommendations Many Thanks !!
Web Design | | Modi0 -
Duplicate content issue
I have recently built a site that has a main page intended to rank for national coverage. This site also has a number of pages targeted at local searches, these pages are slight variations of each other with town specific keywords. Does anyone know if google will see this as spam and quarantine my site from ranking? Thanks
Web Design | | stebutty0