Welcome to the Q&A Forum

Hey guys,

My site is Netspiren.dk and when I use a tool like Screaming Frog or Integrity, it only crawls my homepage and menu's - not product-pages.

Examples
A menu: http://www.netspiren.dk/pl/Helse-Kosttilskud-Blandingsolie_57699.aspx
A product: http://www.netspiren.dk/pi/All-Omega-3-6-9-180-kapsler_1412956_57699.aspx

Is it because the products are being loaded in Javascript? 
What's your recommendation?

All best,
Fred.

You are absolutely right.

The problem was that a plugin I installed messed with my robots.txt

I have tweak around with an XML SItemap-generater and I think it works. I'll give an update in a couple of hours!

Thansk!

Thanks for your comment Stubby and you are probably right.

But the problem is the Disallowing and not the sitemaps.. And based on my Robots.txt should everything be crawable.

What I'm worried about is that the virtuel Robots.txt that WP-generates is trouble.

I guys.. I have a very annoying issue..

My Wordpress-blog over at www.Trovatten.com has some indexation-problems..

Google Webmaster Tools data:
GWT says the following: "Sitemap contains urls which are blocked by robots.txt." and shows me my homepage and my blogposts..

This is my Robots.txt: http://www.trovatten.com/robots.txt
"User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Do you have any idea why it says that the URL's are being blocked by robots.txt when that looks how it should?
I've read a couple of places that it can be because of a Wordpress Plugin that is creating a virtuel robots.txt, but I can't validate it..

1. I have set WP-Privacy to crawl my site
2. I have deactivated all WP-plugins and I still get same GWT-Warnings.

Looking forward to hear if you have an idea that might work!

Hi Yannick,

Thanks for your response.

I agree that such links build Domain Authority. 
But great links from CNN etc. should also increase the moz-trust/rank... And Netspiren has a higher moz rank/trust. What I am wondering about, is the logic/correlation between: Mozrank/trust vs. Domain Authority. The screenshot shows that Netspiren.dk has more of any other metric, than the overall metric "domain Authority (And Total amount of links, which I don't take much into account).

Hi Moz'ers! I have a questions for you guys. Can you explain why the website "Netspiren" has a lower "domain Authority" than "Duft og Natur"? See screenshot. I know it's a tough question without a detailed analysis, but looking at OSE's data, does the mozrank,moztrust, C-block don't reflect the power of the Domain Authority. What's you take? BTW: "Netspiren" has always had more and better links, and also had a higher Domain Authority. Something about the Domain Authority changed since the past 1-3 OXE updates. Looking forward for your take on this! Have a good weekend!

mnRPI.png