Misbehaving Robots
-
I have a disallow entry in my robots file which is
/*src=
So I have urls that look like
www.website.co.uk/example?value=1&src=2
I would expect these URLs to be blocked.
However both Moz and AHREFS have these in the crawl logs
www.website.co.uk/example?value=1
It is as if the bots are reading the first parameter but being blocked when they reach the second and logging part of the URL.
Is this standard behavior for Roger (mozbot) or Ahrefs bot. Does google act in the same way?
-
Hi Paul
Thanks for your response. I'm always happy to hear your advise.
I've been through the page code line by line.
I've also fetched in google and the HTML does not contain a URL without the SRC value.
I'm really stuck
-
My suspicion would be that those URLs are accessible somewhere without the second variable. (I.e. That second variable isn't always present when the first variable is.)
-
Ok the URL looks like this
www.site.com/engine/referrer.asp?web=http%3a%2f%2fwww.example.co.uk&src=3078c98d2da385d5468f562
However google moz and ahrefs only get this far
www.site.com/engine/referrer.asp?web=http%3a%2f%2fwww.example.co.uk
Could it be because there is a domain name in the paramater
Could it be because they cannot pass the ampersand
Could it be because src is blocked in robots.
Any suggestions would be most welcome.
-
I have gone into search console > crawl > URL parameters.
I have found the SRC parameter and told google to crawl it. (as opposed to "let google decide")
Google has been indexing the pages without the parameter so I think it chose to ignore it. (because it doesn't alter page content)
Google now states. "This will be crawled unless overridden by other commands". So robots should kick in now and stop the URLs being crawled.
Why Moz and AHREFS are also ignoring this parameter is beyond me.
-
No, Google does not act in the same way.
-
Hey Andrew,
Would you be able to reach out to [email protected] so we could have a look at your specific Campaign/Website.
Thanks!
Eli
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Single Sign-On setup
Looking for steps to configure the single Sign-on for MoZ system for my organization.
Feature Requests | | Sandeep_amplity0 -
Unsolved Matomo Data Integration
As GA4 is not easily usable for all clients in the EU, it would be great if you could offer a Matomo integration. Matomo has recently been more and more popular and is used by a lot of companies, that are required to adhere to EU standards in data privacy.
Feature Requests | | datenwerk0 -
Unsolved Page claiming
Can we claim the Lawrence location with only having a Topeka location and also claiming the Topeka location
Feature Requests | | KleerVue0 -
Moz Link Explorer for Google Data Studio
I want to put Moz's link explorer into my data studio report. (Including Domain Authority, Linking Domains, Inbound Links, and Ranking Keywords). How can I do this?
Feature Requests | | Kaili0 -
Can the login page be updated (the tab order)?
When utilizing certain password manages, the tab order on the login page is set in such a way that the username fills in and then where the password should go is the Create an Account link. Thus, the password is not filled in and the user is brought to the Create an Account page.
Feature Requests | | mike.brodeur0 -
Are there any plans for a Moz connector to Google Data Studio
Hello all, As title reads, are there any plans for a Moz connector to Google Data Studio to help automate reporting? I'm surprised this doesn't exist already because Data Studio is so heavily used. Thanks in advance. Simon
Feature Requests | | parkersj9 -
Why my site's DA and Pa Not Show Accurate By MOZ Extension.
my site has some minor problem which about DA and PA that DA and PA do not show accurate some time my site is yts subtitle, please tell me something about it.
Feature Requests | | darkwebvampire0 -
Feature Request: connecting to Adobe Analytics
We're moving away from Google Analytics and setting up Adobe Analytics (previous Omniture).
Feature Requests | | KBC
It would be nice to incorporate this data, so I have a complete tool again...2