First Crawl Report

[iainmoran]

Just joined SEOMoz today and am slightly overwhelmed, but excited about learning loads from it.

I've just received my Crawl Report and there is a
404 : UserPreemptionError:
http://www.iainmoran.com/comments/feed/

This is a WordPress site and I've no idea what the best course of action to take. I've done some searching on Google and a couple of sites suggest removing that url from within the robots.txt file. I'm using the Yoast Plugin which apparently creates a robots.txt file, but I can't see any way to edit it. Is there another solution for resolving the 404 error?

Many thanks,

Iain.

[iainmoran]

John, thank you so much for your help with this. It's very much appreciated.

That does explain why I couldn't find the robots.txt file.

Is there another way to resolve the 404 error warning that SEOMoz is telling me about?

404 : UserPreemptionError:
http://www.iainmoran.com/comments/feed/

I understand that if I disable comments on my WP site, that would fix the issue, but don't really want to do that if it can be avoided.

Thanks again,

Iain.

[Johnny4B]

Actually, on further investigation, blocking the includes folder does not effect ranking, as the content is loaded server side (PHP)

I don't imagine that the robot.txt file is your issue, as its just stopping people accessing your include and admin folder, which you want

[Johnny4B]

Iain

I dont use WP, but just Googled this

"I believe WordPress itself creates the robots.txt file and it is a virtual file, meaning there won't be a hard copy on your server for you to edit. You could create one yourself and upload it to your server and I think search engines will use that one instead, or you can use a plugin like this one, that will let you edit your robots.txt file from the WordPress admin."

Explains why you cant find it !

Basically that suggest creating one yourself and uploading it to the server.

Create a note pad file, add your content, name file robot.txt (must change extension to .txt), and that should work

[Johnny4B]

Iain

Just to be sure, its actually just called a robot.txt (not .text)

I just checked, you do have one in your root file

http://www.iainmoran.com/robots.txt

Bizarrely, its blocking Google indexing your includes, which is a problem cause most of your links will be included in some sort of nav include, not to mention most of your website content.

Its defiantly there, maybe check your public folder and www folder

Let me know how you get on

John

[iainmoran]

Thanks for your quick response, Johnny,

I've just looked at my root directory via both my CPanel and FTP, but there is no robots.text file there.

I don't understand why this is, as in the Yoast section within each of my pages, there are some which I have set to "nofollow"

Also, within the Yoast CP there is a section titled: Robots.txt and says the following under it:
"If you had a robots.txt file and it was editable, you could edit it from here."

Iain.

[Johnny4B]

Iain

You robots.xt file can be found within your root directory

Where ever your website is hosted, log in there (if you have cpanel which is more than likely) and navigate to your public folder. the robots.text file should reside there, download it and edit in any software really, wordpad etc.

If you don't have cpanel, you can add your FTP credential using an FTP client and navigate the same way

When you edit it, upload it back to your server.

Regards

J