Moz Crawl Test: WordPress sites with and without /feed and /trackback entires?
-
I have multiple WP websites and on some of the websites, on my Moz Crawl test, I see an entry for every blog post but also entries for /feed and /trackback for that single blog post. For example,
www...com/someArticle
www....com/someArticle/feed
www...com/someArticle/trackback
1. Can anyone explain why the Crawl test is picking up the /feed and /trackback items? Is it simply because they are 301 redirects to the original post (www...com/someArticle)?
2. What setting(s) in WordPress are making this information appear? Or is it just that the site(s) that have the /feed and /trackback are displaying "normal" behavior for a WP site with a lot of trackbacks and feed entires?
3. Should /fee and /trackback, as well as /author be blocked in robots.txt?
Thanks in advance for your advice and input!
-
I have the same issue but instead of it redirecting to the parent post its just going to a 404 page.
-
So I solved the problem (or at least figured where it was coming from). On this particular site, under the comments area, there is a link for "trackback url" and a link for "comments rss feed". Naturally these are ../trackback and ../blog so that's why the crawl is picking them up. They are 301 redirected to the "parent" page so that's why they are not a duplicate content issue. Thank to everyone for their help!
-
1. If you check the source code of your blog posts, there must be some sort of link to the feeds - possibly even in the header. I'm not 100% on how the Moz crawler operates (if it only spiders <a>anchor links or if it spiders referenced links in the header - pretty sure the latter) - but either way that's how they're finding it, through some sort of link on the page.</a>
<a>You could try running a crawl with Screaming Frog SEO Spider and see if it also picks up the feed URLs and Screaming Frog will show you where it found the links as well.
2. Good question. Your theme may be displaying links to these things somewhere - the best way to find out is to crawl with Screaming Frog and it will show you which pages link to your feed and trackback URLs. Then if you don't need them, you can go into the editor and remove them from the code.
3. I agree with Thomas here, I would not block them with robots.txt - rather I would see if you can fix them at the source and remove the links if they are not needed.
-Dan</a>
-
Thanks, I'll check it out!
-
Hi, you should never block feeds they're really pretty beneficial to your site. Take a look at this from Joost it will explain it much better than I can
http://yoast.com/example-robots-txt-wordpress/
All the best sincerely, Thomas
-
Thank you.
When you say "TrackBacks are from people posting either identical or similar content to WordPress.com", what do you mean? I thought trackbacks were notifications of links back when someone links to your content?
And why does the codex recommend blocking feeds and trackbacks in robots.txt?
Thanks again!
-
the TrackBacks are from people posting either identical or similar content to WordPress.com I would follow up with that. unless that person is you.
No do not block a feed with robots.txt and do not block the TrackBacks use automatics Digital millennium act takedown if somebody is stealing your content.
Sincerely,
Thomas
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to do with a site of >50,000 pages vs. crawl limit?
What happens if you have a site in your Moz Pro campaign that has more than 50,000 pages? Would it be better to choose a sub-folder of the site to get a thorough look at that sub-folder? I have a few different large government websites that I'm tracking to see how they are fairing in rankings and SEO. They are not my own websites. I want to see how these agencies are doing compared to what the public searches for on technical topics and social issues that the agencies manage. I'm an academic looking at science communication. I am in the process of re-setting up my campaigns to get better data than I have been getting -- I am a newbie to SEO and the campaigns I slapped together a few months ago need to be set up better, such as all on the same day, making sure I've set it to include www or not for what ranks, refining my keywords, etc. I am stumped on what to do about the agency websites being really huge, and what all the options are to get good data in light of the 50,000 page crawl limit. Here is an example of what I mean: To see how EPA is doing in searches related to air quality, ideally I'd track all of EPA's web presence. www.epa.gov has 560,000 pages -- if I put in www.epa.gov for a campaign, what happens with the site having so many more pages than the 50,000 crawl limit? What do I miss out on? Can I "trust" what I get? www.epa.gov/air has only 1450 pages, so if I choose this for what I track in a campaign, the crawl will cover that subfolder completely, and I am getting a complete picture of this air-focused sub-folder ... but (1) I'll miss out on air-related pages in other sub-folders of www.epa.gov, and (2) it seems like I have so much of the 50,000-page crawl limit that I'm not using and could be using. (However, maybe that's not quite true - I'd also be tracking other sites as competitors - e.g. non-profits that advocate in air quality, industry air quality sites - and maybe those competitors count towards the 50,000-page crawl limit and would get me up to the limit? How do the competitors you choose figure into the crawl limit?) Any opinions on which I should do in general on this kind of situation? The small sub-folder vs. the full humongous site vs. is there some other way to go here that I'm not thinking of?
Moz Pro | | scienceisrad0 -
Help us with Moz Academy!
Hey Q&Aers! I just wanted to create a discussion thread here to talk solicit some feedback and help with Moz Academy! We want your help coming up with future lessons so we’re getting you the content you need. So, think about what you’d or your team would like to learn from Rand and others at Moz. Let us know! “So, what is Moz Academy?” you may ask. Well, Moz Academy gives subscribers the power to be better marketers, which will enable them to use Moz’s products with more depth and awesomeness. We want to provide a hub of marketing knowledge that will create a stronger community where people can teach each other while using Academy as a frame of reference. The key to this project is empowering you to kick even more ass than you already do! If you have any questions about Moz Academy, feedback about lessons, or ideas on a future curriculum please feel free to get involved in the discussion below. Looking forward to see what everyone comes up with.
Moz Pro | | Nick_Sayers1 -
Ignore Moz Notices?
Hi All, I read an old post (2011) about ignoring rel canonical notices from Moz. I wondered why they can be ignored? The thing is, it's showing a bunch of rel canonical errors for my pages but I know I have the tag right. At the risk of being offensive.. can the notices be ignored because the Moz data is erroneous? I'm not sure what to make of this. Thanks!
Moz Pro | | xvpn9020 -
Campaign Crawl
I have a site with 8036 pages in my sitemap index. But the MozBot only Crawled 2169 pages. It's been several months and each week it crawls roughly the same number of pages. Any idea why I'm not getting fully crawled?
Moz Pro | | JMFieldMarketing0 -
Why is my site not ranking?
I have done various link building strategies, and used all white hat seo methods. SEOMoz even shows that my site is better than all my competition, but yet it doesn't rank ANYWHERE on Google. Can someone point me in the right direction as to why this is happening? I have attached the opensitexplorer screenshot showing my competition. 2013-02-13_1456_zpsb86ab0e1.png
Moz Pro | | locallyrank0 -
Not all pages are being crawled
I am set up on the PRO plan, I was under the impression that it would crawl up to 10,000 pages. My site has just over 200 pages, but whenever I am crawled it only crawls 121 pages. Is this normal? It's hard to know how reliable my data is because a significant amount of pages are missing.
Moz Pro | | KristinHarding0 -
How to crawl the whole domain?
Hi, I have a website an e-commerce website with more than 4.600 products. I expect that Seomoz scan check all url's. I don't know why this doesn't happens. The Campaign name is Artigos para festa and should scan the whole domain festaexpress.com. But it crels only 100 pages I even tried to create a new campaign named Festa Express - Root Domain to check if it scans but had the same problem it crawled only 199 pages. Hope to have a solution. Thanks,
Moz Pro | | EduardoCoen
Eduardo0