Welcome to the Q&A Forum

In terms of the on-site architecture I think that could be improved to rank better, e.g: convert:

https://www.nystatemls.com/profiles/NY/SAINT_ALBANS/NEW_YORK_ESTATES_GROUP_INC/Dale_Davids/68966/listings/

to:

https://www.nystatemls.com/new-york/saint-albans/new-york-estates-group-inc/dale-davids/

But keep in mind that if you did this, you'd have to do a massive internal 301 redirects project using rapid redirect technology like NginX instead of basic web.config / .htaccess mods (and NginX redirects aren't even available on all server types) to cover the change in architecture. I think it could be a good move but with that volume of URLs you'd need a forensic approach

Yes I do think basic SEO training for your members would be good, allowing them to access the H1, <title>, URL slug and Meta Description fields could be very useful. Also allowing them to set bespoke FB OG / Twitter Card share images (1200x630 for dimensions) as well as OG titles and descriptions, could significantly boost your traffic intake across SEO and social - but only if these fields were used correctly. Maybe you'd want to isolate some of the best profiles and take over the Meta updates yourself, to ensure they are 'proper'</p> <p>Incentivisation could be an issue. Yes these guys already promote their own listings, but getting them to do basic keyword research and Meta updates could be a big ask. A review system would be good. Say someone updates this data, then you review it and give it the 'good' stamp of approval - maybe they could get slightly enhanced listings for a duration in some way. Make it worth their while and you will really benefit!</p> <p> </p></title>

First of all those more detailed URLs should have been handled via canonical tags and not via robots.txt

You are probably safe to allow the detailed URLs to rank, try allowing a sample of them to rank whilst keeping others disallowed. First, fix the architecture. Stop using robots.txt and on the detailed URLs, make them canonical to their parents

Once that is done, select a volume of the detailed URLs as a test. Remove the canonical tags from those URLs, allowing them to index. Do they start ranking, performing? Do you get duplicate content warnings?

Depending on the outcome, you may want to lift the canonical tags from all detailed URLs, or even reverse the canonicals so that the detailed pages have ranking preference

Since you have such strong performance that must be a known site or brand. One thing you could do is, download all your historic backlinks and check for dead ones. Where the content has been taken down, or links have been removed from the content. You could contact editors and webmasters to try and get content and links restored, the WayBack machine may help to see what kind of content and links existed previously. If you can get an email address from your client based at their domain,  you usually get a slightly higher response rate

If you're an SEO / marketing person, the likelihood is that you'll need a developers help. They should know what you mean when you ask to exempt a certain user-agent (Googlebot) from the redirects

I actually wrote something relatively recently which might be of interest to you:

https://searchnewscentral.com/blog/2019/07/12/what-might-ai-machine-learning-image-experiments-teach-us-about-search/

The conclusion I basically came to was:

"If I were working on an eCommerce store selling rolls of fabric, I’d say that an image of a rolled up bit of fabric would be good for a mechanical mind to interpret. A zoomed in image of just the fabric’s texture, would also be pretty good! A lady standing by a fireplace with a wine-glass in one hand and a fabric-roll in the other? That would be very difficult for a mechanical mind to interpret."

Play with Google images. Type in your product (or competing products) and see which types of image gain the most prominent positions. That will give you an idea on, how advanced Google is in terms of interpreting certain objects. Do the images need to be super obvious with cut-outs against a blank background? Can you be more adventurous?

Also look at the image thumbnails for your products (or competing ones) on Google Shopping. See what's doing well there

IMO obvious is better for search algorithms, but then again may not have such good conversion rates as more adventurous creative

No impact if you exempt Google's user agent ('googlebot') from the redirects. That way Google can crawl properly without it turning into a fiasco (which it will if you don't exempt them, as they sometimes crawl from different data centres - but only one at once - which does have a fixed location)

If you fail to exempt googlebot then they will only see the page which is rendered to the location of the data centre they are crawling from, so they will think your site keeps going up and down, one page then many pages (which will cause a huge mess)

Remember: exempt Google from the redirects or things will get messy

IMO it's the only approach that works, but you are right risk is involved and it doesn't always work. That's marketing, though

The way to not sound annoying is to approach someone with an original, insightful and beneficial (for the editor and their audience) pitch. If you approach them with questions, you've already lost - as someone else IS in their inbox pitching much more complete ideas which they don't have to think about so much

You have to start building your network. Some will include your citiational links, others won't - or they won't do it all of the time (or they will nullify the links some clever way). In the end, if you represent a client AND a digital agency, editors and webmasters know what you are after. If they refuse to give it (or give a fair amount often enough) then don't work with them any more

The main thing to keep in mind is that, you want to make it clear you are interested in editorial opportunities and not opportunities which are advertorial in nature. You aren't trying to pay for an ad, you're trying to promote a story or a winning piece of content. You don't want to be in a sponsored slot on their site, you don't want to be in an ad-bar, you don't want no-followed links and you don't want to directly pay for the link (as that would make your content an ad)

There may be other way you can interact with the editor like helping them with a fee to review your content. In the main though, the content should sell itself without money needing to change hands. Google only want to count editorial links towards PageRank, not advertorial or sponsored links - so in the end if you have to win, the 10x content / editorial angle is the only one you got

The question is if you can handle it or not

This usually involves some significant input from your client and it really separates the serious digital PR enthusiasts from the 'link builders'. To all intents and purposes, digital PR as a discipline is much broader and has essentially replaced 'SEO link building' which these days, ironically has very little value for SEO (where as digital PR done right, has massive value)

It's about creating an amazing piece of content, or telling an incredible story that everyone would want to hear - then getting it out there. The links occur naturally as a byproduct. For example an SEO link builder would contact a web master and say something like: "I will write gr8 articles 4 u we can put them on your blog I only make good content". Snore, delete. "Good unique content" is seldom good enough to rank competitively any more:

https://mza.bundledseo.com/blog/why-good-unique-content-needs-to-die-whiteboard-friday

Webmasters and editors are now wise to this shady SEO tactic. Writing content that barely qualifies as 'good' or 'unique', stuffed with links to the author's client. Being able to type stuff, isn't a high  barrier for entry into 'proper' link building. Google only want to count the really great stuff when evaluating content and links online

So if you have managed to get it out of your head that you can offer people, free 500 word articles of tenuous quality for links (none of the best sites will rank to you that way) - what do you need to offer instead?

10x content:

https://mza.bundledseo.com/blog/how-to-create-10x-content-whiteboard-friday

My personal strategy is to create some kind of 10x content (or issue the creation of such content, if I can't create it myself - which is highly probable as most 10x content is built by teams and NOT individuals). I try to ensure that the 10x content will tell a story which is 'PR-able' in some way. I try to find homes online for the 10x content, with citational links back to the original piece on the client's site (which is usually more in-depth and granular)

The trick is to get it out of your head that 'just text' qualifies as good content, it usually doesn't (unless it's an exceptional or newsworthy story. Hint: your client opening a new office is NOT newsworthy)

Think about the unique data which is at your client's disposal. Maybe the head of the company is an inspirational entrepreneur with a large social following, maybe interviews with them could be 'digital PR-worthy'. Maybe your client has a load of aggregated, anonymised customer data (or can construct such a data set) which shows a unique trend within their industry - something useful and informative for business readers

You can produce interactive experiences, calculators, interactive (HTML5 / CSS3) infographics, answer questions in ways that no one else has. Every keyword is a query - every query, is a request for information or a question. Address those questions

So how does all this help you with link building? It means that you can go to editors with an actual, genuine pitch. Instead of saying: hey Mr Man from the BBC, want a 500 word post (snore)? You are saying  - "Hey. I realise you edit this specific section of this online publication. My client has conducted some unique research which we have visualised. It would be of genuine interest to your audience for X reason and you would stand to see Y benefit"

Unless you can pitch someone a genuine idea, something which they can see will easily benefit them - don't even bother to attempt link building in 2019. Don't curate email lists of editors and hammer them with flimsy automated email templates. That may have worked in 2003 but it doesn't work now

Not a problem. The UK is just in a weird situation with its listings where in the end, the input outweighs the outcome (IMO)

Basically I don't think theres's an automated, cost-effective, long-term and scaleable way to do them inside of the UK. The cost-effective options typically have poor UK coverage (Bright Edge) or they have no legal agreement with the data aggregators (like Central Index) which means they seem great to use at first, but then when you want listings updates they begin to show how poor quality they are (IMO Moz Local does fall into this bracket - we were totally unable to change address over a period of months it was really bad)

Note: I like and support Moz's tools, I think they're really good. But specifically in the context of the separate Moz Local tool, specifically across the UK (where older directory listing companies have a stranglehold and make it different for Moz to get in) - I can't recommend Moz Local

There are some great tools out there for the UK like YEXT but it's more expensive than it is worth it. There's also the Yell version which uses the same tech, but the support is so bad, inept and slow that again it's not worth it

IMO in the UK none of the automated tools have reached the standard to be considered long-term viable for single or multiple clients. In the USA it's a different story, but in the UK it's very tricky indeed

All this knowledge is about a year old though. You could look again but my expectation is, nothing has changed radically (or maybe even at all)

Note: you won't be doing it manually, sorry (well, maybe if you're in the USA! But in the UK you won't). Most of the directories that count like local newspaper business directories, are supplied (in terms of their listing data) by listing aggregators like Central Index. Those guys have special relationships with some of the tool providers, but unfortunately it's the really expensive ones like YEXT. They won't talk to you, or listen to you (period). Moz Local can get stuff into C.I. but doesn't seem to ever be able to do any updates to existing listings which is a nightmare

To be honest that's exactly right. In actuality you aren't ban evading, because you aren't that site and it no longer lives on that domain. But if Google's algorithms are unaware of that, more issues could arise

AFAIK there's nothing good for UK listings

Read my response to this question: https://mza.bundledseo.com/community/q/is-it-more-beneficial-to-use-yext-rather-than-doing-the-citations-manually

I went on this whole journey, and it turned out to be pretty pointless

Weird to see a "200" listed as a crawl issue! Usually status code 200 is returned when a page loads ok. In fact, it's the exact opposite of a crawl issue (under normal circumstances)

The 301s issue might mean you now have links to redirects. So like, you changed a page's URL and 301'd the old URL to the new one, but you failed / forgot to update the internal links on your site (nav, links from blog posts) to reflect the altered URL, and thus you now have links that point to the old URL which then 301s to the new URL (unnecessary redirects). If that is part of the problem, you simply update all the internal links to point to the new URL, thus evading the redirect (the redirect is really for off-site traffic and links, coming in externally)

You shouldn't have canonical tags on either pointing to the other IMO. A canonical tag, deployed on a web-page, says to Google "I am the non-canonical version of a page. Unless you have signals like links which contradict this tag strongly, don't index this non-canonical page at all. Only index the canonical URL which I am pointing you to"

So the page which you place the canonical tag on, becomes (itself) non canonical and therefore gives a medium-to-strong signal to Google that it should be de-indexed. As such, if you plaster your mobile site in canonical tags, you are essentially telling Google that the entire mobile site is non-canonical and thereby probably not a great candidate for indexation. Do you want your mobile site to rank? I assume you do

I don't know what guidance you have read. Google's guidance is often woefully out of date as their documentation update cycle for organic-search stuff is really poor. If it was something here on Moz, I personally disagree with it

I would just stick with the alternate tags. Anyway if you have canonicals going in two directions, you will create a soft redirect loop where both URLs specify themselves as non-canonical. That could make things way worse than they are now

There is a risk of the new domain being infected with the prior domain's negative equity. Basically if you put exactly the same site up as before with the same content and the domain-name is very similar, Google will assume it's the same site. For a lot of people that can be a good thing (preserving their SEO authority) but if you have negative equity it could be a very bad thing

If Google has mislabeled your site, that's what they believe your site is. Since they believe your site IS a porn site, their algorithms will probably see the new domain (with the same site on it) and interpret that as, hey this porn-site owner is trying to evade our listing bans. Ban evaders are bad, slap it back on!

Since you're already at a point where the algorithm(s) have failed you, I wouldn't anticipate them to randomly be more generous in the future. You might get lucky, but you might find the issue does creep across

The good news I guess is that usually, web-hosting and domain purchases aren't bank-breakingly expensive. As such it's worth a try I suppose. Just don't be shocked if it does well for a few days, and then suddenly declines again

I know that content similarity matters a lot for positive redirects (e.g: if you want to move SEO authority from one page to another, you'd better be sure that the content is very similar - otherwise it has little to no effect even with 301s). Does this also hold true for negative redirects? I'm not certain but I would suspect that the same logic would be at play. In this situation though, since it's exactly the same site and content it could work against you and let the negativity flow through

Maybe you could create 302 redirects instead of 301 redirects (keeps the authority, in this case negative, on the redirecting URL - not the redirect destination). Maybe you could also exempt the Googlebot user-agent from the redirects so that it doesn't see them at all

That would be some stuff I would try, but I wouldn't be amazed if Google got really cross with you for SERP ban-evasion behavior. If you could possibly re-brand slightly and re-write a load of the content that might help significantly. It's a lot of money though when in reality, it's a 50/50 gambit

Check using the organic search estimates in Ahrefs, those are usually pretty good and they keep quite a long history. You can also get similar estimates from SEMRush

I would adjust it personally. If for no other reason than, if someone analyses the site later on they should be able to get a good idea of your strategy. Even if you come back to the site later and re-crawl it. Leaving it as it is will inevitably cause strategic confusion down the line. Just set it to your original vision, keep all the signals under your control pointing in a single direction

If the errors are detected by Moz's crawler and Google Search Console (both at the same time) then I'd be much more concerned. It does also depend on the volume of them, if there are like three then it's probably not worth your time to sort it out. If there are hundreds or thousands, you might want to think about that

If you have hidden links in the coding which Moz is picking up on (that's how Moz's crawler works, by following links) then you can't really say: "We've checked each page and know that we are not linking to them anywhere on our site" - the fact that the crawler found the links means they exist and are there (even if you can't see them or find them). That is of course, unless your site is on one of the unusual architecture that Rogerbot (Moz's crawler) has difficulties with. That shouldn't be your first assumption, though - he usually knows where he's going

Where you say this:

"since we migrated our blog to Hubspot so we think it has something to do with the test pages their developers had set up" - pull them up on it! If their developers coded a load of errors into your site, that's their fault not yours and it should be their expense (not yours) to fix it

This is the page regarding their CMS:

https://www.hubspot.com/products/marketing/content-management-system

It does say "A Content Management System Built for Professional Marketers" - so migrating to it, shouldn't cause loads of SEO problems, as SEO is still the largest chunk of most site's online marketing and traffic. That should be nailed down, no problems, fewer problem than your prior system

In-fact, HubSpot know that SEO is important for a CMS: https://www.hubspot.com/cms-and-seo - "Every marketer has been told that they need to consider SEO when creating content. But what makes SEO a unique marketing strategy that marketers should prioritize? And why should your CMS have tools that help you execute your SEO strategy?" - I would argue that a load of 404 errors, could not be considered "tools that help you execute your SEO strategy"

Whether their developers messed up or their CMS is at fault is not really relevant. The main point is, the responsibility to sort it out should be on their side (not yours, IMO)

It can do under certain circumstances, particularly if vulnerabilities are detected. If your news section had malware or if links from your news section were detected to HTTP resources (internal / external pages, images embedded from HTTP) that could cause problems (Google want everything on HTTPS now)

It would be unusual but there are certain circumstances where you could see wider impact(s). Another thing is, if the news section garnered your site a manual penalty, then quite often that could spread to all pages / keywords / rankings. Simply deleting the news section would be unlikely to restore performance instantly

If it were something like poor content or duplicate content, that would be more likely to just affect your news section without impacting your other pages (e.g: products) unless of course they shared the same issues

If the news section is riddled with technical errors it could bring the site health metrics for your overall domain slightly but in general the impact to the rest of the site should be non-radical

Hope that helps

Yep this is widely known. In-fact, you may be surprised to learn that even those who both manufacture AND distribute their products are also under the kosh here. I work with a number of fashion brands who produce their clothing (which in many cases is really stylish, really high quality stuff) only to see their direct sales (from their own site) hindered by larger 'pure distribution' sites (sites which just distribute loads of items but don't manufacture any)

Here's the truth of the matter. It's not that distributors get some kind of unfair special treatment. Rather, it's that distributors aren't bound by the same limitations as manufacturers, or manufacturers with online distribution

I love all the brands that I work with, I think their goods are grade A (otherwise I wouldn't work with them). That being said, they can only list their own products on-site. Users commonly search for slightly broader terms which imply that they want to see a range of products to choose from (e.g: "men's shirts", "supportive bras" or "tailormade suits"). The truth is that for a user, the distribution sites ARE better. They can see a huge range of products on these sites and compare them all to find the best deal. This is part of the 'rise of the aggregators' phenomena. Nowadays, unless you're incredibly special - if you don't aggregate, you lose out (end of)

There's a reason that giant product aggregation sites (ASOS, eBay, Amazon, Bra-Stop) are killing it. They give people a one stop shop to solve their issue, then continue their lives happy that they have got a good deal. Even the services industry isn't immune to this, just look at sites like Money Supermarket and how dominant they are in Google's SERPs

Now you might say, ok - that makes sense for category level terms, but often these guys are beating my client even on product level terms. Why is that?

Well - by being such useful sites that make a big difference to user's lives, these sites organically spawn vast amounts of digital PR and UGC link coverage. They're frequently mentioned by the BBC, Forbes... you name it, they're in there (all the time and repeatedly). This builds up a colossal amount of SEO authority, which you basically can't compete with

Finally: often these distribution-only sites, because they are so dominant, are a client's main revenue stream (more than their own sites). That's because client's get paranoid that they will offend their distributors and they will then refuse to stock their products. That could be a disaster! But due to this paranoia, they often give distributors better prices for their products, than even their own native sites (if they are manufacturing and distributing directly as well - some do, some don't)

Due to this, they lose EVEN MORE product-level listings, as Google knows their listings are the worst deal for users. These patterns of thinking, keep the dominant distributors on top and leave crumbs for the rest of us

In the end, their success on Google is really a business  decision and NOT a pure-SEO decision. Do they want to bite the bullet and cut their distributors off from their products, keeping all the Google SERPs for themselves? Well certainly that would cut those guys out completely - but any distributor originating cross-traffic would be lost. What if a user visited the distributor site for another search query (not related to your client's product), then saw your client's product later in their journey and purchased it? All of THAT traffic (which is substantial) is going to be immediately lost. So although cutting out the middle-men might seem smart, it's an incredibly unwise knee-jerk reaction

Still, you can never grow above the distributors until you make this choice. It's a choice that WILL hurt you, but in the long run will allow you to reap revenue figures which you never previously dreamed of. So who makes the decision of when the time comes, to go through this necessarry pain? NOT you. Not an SEO person. That responsibility rests with the business owner alone and if I were you, I wouldn't seek to influence it (risky)

Hey Raoul

Just gonna do a quick Google and see what I can find from 2019, articles from sites which I have previously felt had a good say on one or two topics

https://www.bigcommerce.co.uk/blog/bounce-rates/#what-is-a-bounce-rate-on-an-ecommerce-website

The above was posted in 2019. Even though the post date isn't declared on the front-end, it is declared in the schema which you can review the results for here. Check the 'datePublished' entry under 'Article'. This is a more general post on bounce rates, but includes some sample stats from a few studies (how email segmentation affected bounce rates, how having a second review on a page instead of just one affected bounce rates). It might not have everything you need but if you give it a proper read, it might be a component that could help to build your story. I have read a few posts by Bigcommerce, I actually think they're usually pretty neat

https://conversionxl.com/guides/bounce-rate/benchmarks/

This one comes from a source I haven't read previously, so you'll have to evaluate how legitimate you think this content is. It's recent but not super recent, having been published in 2017. Again I had to detect this through schema (here). Look for 'datePublished' under 'WebPage'. It may be a bit more useful to you, as it's a series of bounce-rate benchmarks split by various dimensions (channel, site type etc)

http://www.fonemedia.co.uk/blog---mobile-bounce-rates.html

Posted on 18th Feb 2019. Short and sweet, but again I've never heard of these guys so I don't know how good / accurate the info is. Their main claim is that "The bounce rate on mobile is 28% higher than desktop". I would be wary, nothing is cited, there's no images of graphs, charts or data-sets. No citations are given

https://www.crazyegg.com/blog/why-users-leave-a-website/

Posted Feb 2019. These guys I do trust. They produce a pretty cool heat-mapping analytics solution for websites, so that you can see a heatmap of (aggregate) user mouse movements and clicks. Lots of people use CrazyEgg and they probably wouldn't go making outrageous claims without doing some ground-work. They give a break-down of different bounce-rates by site type, maybe you could aggregate their findings with the ones from Bigcommerce and ConversionXL - adding your own thoughts. What I like here is the focus on time rather than the actual percentage bounce-rate ("The 15 Second Rule")

https://www.gorocketfuel.com/the-rocket-blog/whats-the-average-bounce-rate-in-google-analytics/

I think you found this one too. It's neat, but it doesn't specify a publish date on-page or in the schema. Judging by the site's blog feed, the post was written 2014 or earlier. The content mentions a sample of data from 2013, so we can say that by now this is probably out of date (what a shame!)

https://golocalinteractive.com/blog/industry-news/marketing-in-the-age-of-short-attention-spans/

From June 2019. A really thorough, well presented study which is easy to read. This plays more to your short attention spans angle. I don't know these guys, but that's okay because they cite (most of) their claims including a very interesting research paper from Akamai who I am sure many are aware of (which itself was written in 2017, not too long ago)

... that's all I have time for right now! Hope that helps

Just so you know, EMA (exact-match anchor text), which is also referred to as 'over' link optimisation, is more a concern for your off-site links. In terms of your internal site structure, that's much more lenient. Obviously if it impacted UX (e.g: site nav buttons with ridiculous amounts of text that become over-chunky, annoying users) then that's bad. If you can satisfy UX and also do some light keyword optimisation of your internal site links, I honestly don't see that as a massive problem. If anything it just gives Google more context and direction

I don't think internal link over-optimisation is a myth, because there's always someone stupid enough to pick up a spoon and run with it (taking it to ridiculous extremes that would also impact UX and the readability of the site). But as long as you don't go completely mental and the links make sense for users (they end up where they would expect to end up, with concise link / button text that doesn't bloat the UI) then you're fine. Don't worry about this overly much, but don't take it to an unreasonable extreme

For people on Moz to help you, we need a more solid idea of  what the issue is which you are trying to get help on

For example:

1. I am trying to solve technical issues that would prevent funnel views being set up in Analytics, like changing paths or redirects which break the funnel (this would probably require sharing the site so people can get into the nitty-gritty of exactly what's happening, otherwise a resolution is less likely)
2. I am ok on the technical side, but the volume of potential paths through the funnel(s) are vast and labrynthian. The site seems to necessitate that I create new accounts over and over to explore these paths - is there a better way? (so correct me if I am wrong, but can't you just make one test account and then use that (logged in) to explore the different path variations?)
3. Actually - I think that the whole funnel has been set up incorrectly. I don't think it's a good idea to force users to sign up at the beginning of the funnel. Can you guys help me to find case studies, showing that this is true? (it is true, by the way - you tend to get better conversions doing this as the last step not the first)

Once you have decided upon a line of investigation, it will be easier for Moz users to assist you

I think what we really need to understand is, what is the purpose of hiding the link from Google? If it's to prevent the discovery of a URL or prevent the indexation of a certain page (or set of pages) - it's easier to achieve the same thing by using Meta no-index directives or wildcard-based robots.txt rules or by simply denying Gooblebot's user-agent, access to certain pages entirely

Is is that important to hide the link, or is it that you want to prevent access to certain URLs from within Google's SERPs? Another option is obviously to block users / sessions referred from Google (specifically) from accessing the pages. There's lots can be done, but a bit of context would be cool

By the way, no-follow does not prevent Google from following links. It actually just stops PageRank from passing across. I know, it was named wrong

Google Search Console actually has a URL removal tool built into it, unfortunately it's not really scaleable (mostly it's one at a time submissions) and in addition to that the effect of using the tool is only temporary (the URLs come back again)

In your case I reckon' that changing the status code of the 'gone' URLs from 404 ("temporarily not found, but will be returning soon") to 410 ("GONE!") might be a good idea. Google might digest that better as it's a harder indexation directive and a very strong crawl directive ("go away, don't come back!")

You could also serve the Meta no-index directive on those URLs. Obviously you're unlikely to have access to the HTML of non-existent pages, but did you know Meta no-index can also be fired through x-robots, through the HTTP header? So it's not impossible

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

(Ctrl+F for "X-Robots-Tag HTTP header")

Another option is this form to let Google know outdated content is gone, has been removed, and isn't coming back:

https://www.google.com/webmasters/tools/removals

... but again, URLs one at a time is going to be mega-slow. It does work pretty well though (at least in my experience)

In any eventuality I think you're looking at, a week or two for Google to start noticing in a way that you can see visually - and then maybe a month or two until it rights itself (caveat: it's different for all sites and URLs, it's variable)

This is also relevant to knowledge-graph boxes and the images which Google compiles into those. Not quite the same thing as the featured snippets but still pretty neat

Typically it's a good one for industrial materials or chemical compounds, e.g:

https://www.google.com/search?q=Poly%28methyl+2-methylpropenoate%29

https://d.pr/i/lQLl1v.png (screenshot)

or

https://www.google.com/search?q=polyethylene

https://d.pr/i/HwWXoc.png (screenshot)

... there are lots of 'material' based knowledge graph entries, which pull images from other sites in, in order to build up a good view of what the material is. Some people actually find the images from their sites which Google injects, and edit them to use a small and unobtrusive watermark (the trick is not to get too greedy, or Google notices and replaces the image from your site with an image from another site!)

Obviously where branded products and compounds / materials converge, it's easier to get some branding showing in the tiny little images:

https://www.google.com/search?q=cbd+oil

https://d.pr/i/jLn5uQ.png (screenshot)

A lot of these actually come through Google's image search results

You don't see so many successful injections in this particular area, these days though

This one is quite a neat example:

https://d.pr/i/582BzT.png (screenshot)

and this one also:

https://d.pr/i/kaibeC.png (screenshot)

... bit of free advertising for BirdsEye and Green Giant there

Good insights from Kate here. I quite like the proposed solution from Rajesh

Really interested to see that others have been receiving this too, we have been having this flagged on a couple of sites / accounts over the past month or two

Basically, Google Data Studio's schema error view is 'richer' than that of Google's schema tool (stand-alone) which has been left behind a bit in terms of changing standards. Quite often you can put the pages highlighted by GSC (Google Search Console) into Google's schema tool,  and they will show as having warnings only (no errors) yet GSC says there are errors (very confusing for a lot of people)

Let's look at an example:

• https://d.pr/i/xEqlJj.png (screenshot step 1)
• https://d.pr/i/tK9jVB.png (screenshot step 2)
• https://d.pr/i/dVriHh.png (screenshot step 3)
• https://d.pr/i/X60nRi.png (screenshot step 4)

... basically the schema tool separates issues into two categories, errors and warnings

But Google Search Console's view of schema errors, is now richer and more advanced than that (so adhere to GSC specs, not schema tool specs - if they ever contradict each other!)

What GSC is basically saying is this:

"Offers, review and aggregateRating are recommended only and usually cause a warning rather than an error if omitted.  However, now we are taking a more complex view. If any one of these fields / properties is omitted, that's okay but one of the three MUST now be present - or it will change from an warning to an error. SO to be clear, if one or two of these is missing, it's not a big deal - but if all three are missing, to us at Google - the product no longer constitutes as a valid product"

So what are the implications of having schema which generates erroneous, invalid products in Google's eyes?

This was the key statement I found from Google:

Google have this document on the Merchant Center (all about Google Shopping paid activity): https://support.google.com/merchants/answer/6069143?hl=en-GB

They say: "Valid structured markup allows us to read your product data and enable two features: (1) Automatic item updates: Automatic item updates reduce the risk of account suspension and temporary item disapproval due to price and availability mismatches. (2) Google Sheets Merchant Center add-on: The Merchant Center add-on in Google Sheets can crawl your website and uses structured data to populate and update many attributes in your feed. Learn more about using Google sheets to submit your product data. Prevent temporary disapprovals due to mismatched price and availability information with automatic item updates. This tool allows Merchant Center to update your items based on the structured data on your website instead of using feed-based product data that may be out of date."

So basically, without 'valid' schema mark-up, your Google Shopping (paid results) are much more likely to get rejected at a higher frequency, as Google's organic crawler passes data to Google Shopping through schema (and assumedly, they will only do this if the schema is marked as non-erroneous). Since you don't (well, you haven't said anything about this) use Google Shopping (PLA - Product Listing Ads), this 'primary risk' is mostly mitigated

It's likely that without valid product schema, your products will not appear as 'product' results within Google's normal, organic results. As you know, occasionally product results make it into Google's normal results. I'm not sure if this can be achieved without paying Google for a PLA (Product Listings Ad) for the hypothetical product in question. If webmasters can occasionally achieve proper product listings in Google's SERPs without PLA, e.g like this:

https://d.pr/i/XmXq6b.png (screenshot)

... then be assured that, if your products have schema errors - you're much less likely to get them listed in such a way for for free. In the screenshot I just gave, they are clearly labelled as sponsored (meaning that they were paid for). As such, not sure how much of an issue this would be

For product URLs which rank in Google's SERPs which do not render 'as' products:

https://d.pr/i/aW0sfD.png (screenshot)

... I don't think that such results would be impacted 'as' highly. You'll see that even with the plain-text / link results, sometimes you get schema embedded like those aggregate product review ratings. Obviously if the schema had errors, the richness of the SERP may be impacted (the little stars might disappear or something)

Personally I think that this is going to be a tough one that we're all going to have to come together and solve collectively. Google are basically saying, if a product has no individual review they can read, or no aggregate star rating from a collection of reviews, or it's not on offer (a product must have at least one of these three things) - then to Google it doesn't count as a product any more. That's how it is now, there's no arguing or getting away from it (though personally I think it's pretty steep, they may even back-track on this one at some point due to it being relatively infeasible for most companies to adopt for all their thousands of products)

You could take the line of re-assigning all your products as services, but IMO that's a very bad idea. I think Google will cotton on to such 'clever' tricks pretty quickly and undo them all. A product is a product, a service is a service (everyone knows that)

Plus, if your items are listed as services they're no longer products and may not be eligible for some types of SERP deployment as a result of that

The real question for me is, why is Google doing this?

I think it's because, marketers and SEOs have known for a long time that any type of SERP injection (universal search results, e.g: video results, news results, product results injected into Google's 'normal' results) are more attractive to users and because people 'just trust' Google they get a lot of clicks

As such, PLA (Google Shopping) has been relatively saturated for some time now and maybe Google feel that the quality of their product-based results, has dropped or lowered in some way. It would make sense to pick 2-3 things that really define the contents of a trustworthy site which is being more transparent with its user-base, and then to re-define 'what a product is' based around those things

In this way, Google will be able to reduce the amount of PLA results, reduce the amount of 'noise' they are generating and just keep the extrusions (the nice product boxes in Google's SERPs) for the sites that they feel really deserve them. You might say, well if this could result in their PLA revenue decreasing - why do it? Seems crazy

Not really though, as Google make all their revenue from the ads that they show. If it becomes widely known that Google's product-related search results suck, people will move away from Google (in-fact, they have often quoted Amazon as being their leading competitor, not another search engine directly)

People don't want to search for website links any more. They want to search for 'things'. Bits of info that pop out (like how you can use Google as a calculator or dictionary now, if you type your queries correctly). They want to search for products, items, things that are useful to them

IMO this is just another step towards that goal

Thank you for posting this question as it's helped me get some of my own thoughts down on this matter

Google can crawl JavaScript and its contents, but most of the time they are unlikely to do so. In order to do this, Google has to do more than just a basic source code scrape. Like everyone else seeking to scrape data from inside of generated elements, Google has to actually check the modified source-code, after all of the scripts have run (the render) rather than the base (non-modified) source code before any scripts fire

Google's mission is to index the web. There's no doubt that, non-rendered crawls (which do not contain the generated HTML output of scripts) can be done in a fraction of the time it takes to get a rendered snapshot of the page-code. On average I have found rendered crawling to take 7x to 10x longer than basic source scraping

What we have found is that Google are indeed, capable of crawling generated text and links and stuff... but they won't do this all the time, or for everyone. Those resources are more precious to Google and they crawl more sparingly in that manner

If you deployed the link in the manner which you have described, my anticipation is that Google would not notice or evaluate the link for a month or two (if you're not super popular). Eventually, they would determine the presence of the link - at which point it would be factored and / or evaluated

I suppose you could embed the script as a link to a '.js' module, and then use Robots.txt to ban Google from crawling that particular JavaScript file. If they chose to obey that directive, the link would pretty much remain hidden from them. But remember, it's only a directive!

If you wanted to be super harsh you could block Googlebot (user agent) from that JS file and do something like, 301 them to the homepage when they tried to access it (instead of allowing them to open and read the JS file). That would be pretty hardcore but would stand a higher chance of actually working

Think about this kind of stuff though. It would be pretty irregular to go to such extremes and I'm not certain what the consequences of such action(s) would be

Read what EGOL wrote. It depends upon the nature of your blog pagination

There are a few reasons you could have pagination within the blog area of your site

1. Your articles have next buttons and different parts of the article are split across multiple URLs. The content across the paginated elements is distinct

2. Your post feeds are paginated, purely so people can browse to pages of 'older posts' and see what your wrote way back into your archives

3. Your blog posts exist on a single URL, but when users comment on your posts, your individual posts gain paginated iterations so that users to browse multiple pages of UGC comments (as they apply to an individual post)

In the case of 2 or 3 it's not necessarry to have unique H1s or Page Titles on such paginated addresses, except under exceptional circumstances. In the case of #1 you should make the effort!

This is very true for multi-section articles (which span multiple addresses), and less true of articles which have only one address yet break down into multiple addresses in terms of UGC comment-based pagination

You can do that but you'd want to consider what you're trying to achieve first. Pages and sites can have positive or negative equity from an SEO POV. If you have pages which are low quality, I would assume that by removing them you'd be hoping to remove the negative 'stigma' from that content. If you 301 redirect those URLs which you remove, you may unwittingly end up re-capturing some of the negative equity appended to low-quality URLs

In actual fact that would probably be quite a niche situation, for that to happen the pages would have to have some seriously negative authority / trust signals. In reality it will probably be ok to do this, but always check your data and 'look before you leap'

The authority has probably decayed, I think it's more a case of starting over and rebuilding the authority - rather than waiting and hoping for the best. I know, it sucks when you have shelled out on a domain. But in my experience domain purchasing is really hit and miss. If you don't see an immediate difference, often you don't see one at all. Maybe others have different POVs though

Impressions don't really mean much at all, as Google often experiments in terms of ranking sites for new keywords, then decides they are not relevant and takes the rankings away again. What we really need to see is 1yr+ of traffic (Google Analytics) and Clicks (Search Console). Even that may not be enough to define exactly what's going on

The site is mainly tagged with the FA language, which is Persian. AFAIK (as far as I know) most Persian people live in Iran, which used to be called Persia (and which is still named Persia by some people, though on the international stage the nation is referred to as Iran)

Right now there's a lot going on in the news between the USA (where Google is based) and Iran. It does make me wonder, does make me consider - could that be part of the issue? Obviously it would be impossible to gain clarification but...

One thing I know, is that Google is foremost an American company, and that right now the USA has "engaged in a campaign of maximum financial pressure on the Iranian regime and intends to enforce aggressively these sanctions that have come back into effect" - source

Who knows what's going on behind the scenes. Right now, Google is really clamping down on 'soft' medical practices within their SERPs which we know from all the YMYL / Medic updates. I know that Google only has limited presence in Iran (as you can see there, they won't even give Google-Iran a TLD, they use parameters in the URL structure to sort of generate a relevant page). This could in part be due to internet censorship in Iran. We know that even the app market, Google Play is extremely locked down in Iran

Without taking sides or making any judgements on the international level (something we wouldn't do) - it does seem that Google have difficulties operating in Iran, in the same way that they operate in the West. The USA is clearly sending signals to Iran right now on the international stage (which are also being returned) - as such, it's not hard to see that an Iranian site (especially one with potential Medic / YMYL issues) might fail to rank on an American search engine

Your site seems to use the "Netmihan Communication Company Ltd" ISP, which would confirm that the site is based in Iran (rather than just being built for an Iranian audience by those who may be external to Iran). I have the city down as Rasht

Taking no sides here, it's possible that your site has become a casualty of international conflict (at least on the communication and economic level) and additionally of YMYL / Medic updates, which may have stung you regardless of your location

Hope this is helpful to you, hope you will have a great day

0. "We are a Grocery co-operative retailer and have chain of stores owned by different people. We are building a new website, where we would geo-locate the closest store to the customer and direct them to a particular store (selected based on cookie and geo location). All our stores have a consistent range of products + Variation in 25% range. I have few questions" - make sure you exempt Googlebot's user-agent from your geo-based redirects otherwise the crawling of your site will end up in a big horrible mess

1. "How to build a site-map. Since it will be mandatory for a store to be selected and same flow for the bot and user, should have all products across all stores in the sitemap? we are allowing users to find any products across all stores if they search by product identifier. But, they will be able to see products available in a particular store if go through the hierarchical journey of the website." - any pages you want Google to index should be in your XML sitemap. Any pages you don't want Google ti index should not be in there (period). If a URL uses a canonical tag to point somewhere else (and thus marks itself as NON-canonical) it shouldn't be in the XML sitemap. If a URL is blocked via robots.txt or Meta no-index directives, it shouldn't be in the XML sitemap.  If a URL results in an error or redirect, it shouldn't be in your XML sitemap.The main thing to concern yourself with, is creating a 'seamless' view  of indexation for Google. It seems like you'll have to have the same products available in multiple stores. You will want them all indexed, but will have to work hard to differentiate them (different images, different copy, different Meta data) otherwise Google will probably pick one product from one store as 'canonical' and not index the rest, leading to unfair product purchasing (users only purchasing X product from Y store, never the others). In reality, setting out to build a site which such highly divergent duplication is never going to yield great results, you'll just have to be aware of that from the outset

2. "Will the bot crawl all pages across all the stores or since it will be geolocated to only one store, the content belonging to only one store will be indexed?" - No it won't. Every time Google crawls from a different data centre, they will think all your other pages are being redirected now and that part of the site is now closed. Exempt Googlebot's user-agent from your redirects or face Google's fiery wrath when they fail to index anything properly

3. "We are also allowing customers to search for older products which they might have bought few years and that are not part of out catalogue any more. these products will not appear on the online hierarchical journey but, customers will be able to search and find the products . Will this affect our SEO ranking?" - If the pages are orphaned except in the XML sitemap, their rankings will go down over time. It won't necessarily hurt the rest of your site, though. Sometimes crappy results are better than no results at all!

Hope that helps

Firstly see what Google thinks of the page. On-page SEO checkers require wider experience to use accurately and efficiently. You can't just take insights from online tools and run with them

If the page has no associated keywords / search queries (in Ahrefs, GSC, Google Analytics, SEMRush etc) then that would show Google isn't really that interested. Because keyword data (even from Google) is always heavily sampled, you'd also want to check if, as a 'landing page' (in Google Analytics), the page had been receiving any traffic from Google (Organic Search segment)

If the page is doing well despite what online tools say, rules be damned. If the page isn't performing well (or at all) then just re-build it from scratch in line with best practices, and 301 redirect the old URL to the new one. If the URL stays the same then just rebuild on the active URL, that's fine too (but don't publish until the complete re-build is 100% finished and you are 110% happy with it)

I get you at position #6 on Google. You are there but admittedly it's not great:

• https://d.pr/i/XS14Cn.png (screenshot)

The site is ranking, just very badly which makes me think it's NOT technical SEO related

Let's look at that first though:

• https://d.pr/i/5a7y8D.png (screenshot)

By the way you can grab that Chrome extension here, it's quite handy.  DeepCrawl is actually a very powerful SEO crawling suite (they make their own crawler, and also this handy Chrome extension); many find it superior to Screaming Frog - the main reason it's not that talked about is the pricing which is pretty steep (IMO)

To confirm, there's nothing majorly wrong with your robots.txt file either:

• https://d.pr/i/7ZuIlf.png (screenshot)

On simple and relatively trustworthy one-page scanners, the site doesn't seem to have any massive technical SEO issues (at least at the homepage level): https://seositecheckup.com/seo-audit/blowfishseo.com

So let's look at links I guess?

https://ahrefs.com/site-explorer/overview/v2/subdomains/recent?target=blowfishseo.com

• https://d.pr/i/eH0GKc.png (screenshot)

That looks like it could be doing some damage. Massive spike in linking URLs May 2018, followed by a massive spike in referring domains July 2019. Doesn't look natural AT ALL

So what about anchor texts?

• https://d.pr/i/GagQUI.png (screenshot)
• https://d.pr/f/jPWNPr.csv (data)

In green you have partially branded or branded anchor texts, Google usually has no problem with these. In blue you have part keyword part geo terms, Google can have a problem with these if they begin to feel your backlink profile looks unnatural. In red you have exact-match anchor keywords, Google doesn't really like people link-building in this way any-more (most SEO firms would say EMA links are a no-no)

That spike in referring domains which we saw earlier really interests me. It seems to be concentrated... mostly between... 28th June 2019 and ... 6th August 2019

Let's take a look at new links specifically from within that time period (remember it's actually the time when Ahrefs discovered them, they may have gone live slightly earlier - but Ahrefs are pretty supreme for any kind of proper backlink research)

https://d.pr/i/Wi7p8S.png (screenshot)

https://d.pr/f/RhbuDv.csv (data)

... highlighted some stuff there that looks odd. Links from unrelated markets (languages) or locked regions (CN / Chinese firewall) and links from weird looking subdomain sites that look spam-made and mass produced

Yup, I'd say it's the backlink profile being too oldschool and full of crap, with recent spikes and anchor-text balancing problems. You unfortunately 'just' tipped the scales in a bad way and now you're suffering

Four possibilities:

1) You have been doing some bad backlink work, or had previously done so and never took the time to do a big disavow / link removals project when standards changed. Legacy is hitting you now, because of recent spiking link movements, which tipped you over the scales (has happened to many people, doesn't mean you're a terrible SEO or anything of the sort - was actually quite impressed with your on-site architecture and stuff)

2) One of your subordinates has gotten slack and done some bad backlink work (you'd need to find out who is playing you for a mug ASAP and sack them). If it's someone extremely junior show leniency I guess. But if you looked at the data I had supplied and found a bad network of shallow content links from no-name sites, and then found a paper-trail saying you had actively paid for those... I'd personally be pretty annoyed

3) Some scraper site has started scraping your posts onto their sh*tty network of sites, in the hopes of making their sites less terrible, but actually now it's hurting you as well

4) Deliberate negative-SEO intervention (we have seen a few of these lately on here, it seems to be a very 2019 thing IMO)

... hope that helps!

I would say that if the domain had been parked for an extensive duration it probably would count as fresh, especially if (once the domain were resuscitated) the content was very different from Google's last 'active' cache. They don't really want to give people free SEO authority just for buying old domains (that would make it way too easy to game Google's rankings)

They do a similar thing with 301 redirects now where, they check if the 301-receiving URL is 'similar' (probably in Boolean string similarity terms) to the last active cache of the old URL, so nowadays - even the mighty 301 often doesn't transfer much (or any) SEO authority. I guess it's because, the old URL (in this hypothetical redirect scenario) gained links from webmasters based upon the old content. If the new content is quite different, those webmasters may not have chosen to link to it, ergo the content is then expected to re-prove itself (sounds perfectly fair to me)

Another thing, Google don't use Moz's PA and DA metrics to rank pages. They're shadow metrics, metrics which our industry invented to mimic "PageRank" which Google don't show publicly, and never did (unless you count the old Toolbar PageRank, but that was grossly oversimplified and has been deprecated). As such, sometimes sites have moderate PA and DA without ranking well or at all on Google (Moz's link index is far superior to their keyword index)

Finally, Moz's PA and DA don't take into account hidden signals. The disavows on a domain, any penalties it might have. When a site gains a Google penalty (or algorithmic devaluation) Moz's tool does not get an update from Google on that

Buying old domains is a pretty hazy business, IMO there are too many variables to make most purchases (purely for SEO purposes) viable or worthwhile (or scaleable)

You could always force trailing slashes instead of removing all trailing slashes.

What you really want to establish, is which structure has been linked to more often (internally and externally). A 301 redirect, even a deeper more complex rule - is seldom the answer in isolation. What are you going to do (for example) when you implement this, then you realise most of the internal links use the opposite structure to the one which you picked, and then all your internal redirects get pushed through 301s and your page-speed scores go down?

What you have to do is crawl the site now, in advance - and work out the internal structure. Spend a lot of time on it, days if you have to, get to grips with the nuts and bolts of it. Figure out which structure most internal/external links utilise and then support it

Likely you will need a more complex rule than 'force all' or 'strip all' trailing slashes. It may be the case that most pages contain child URLs or sub-pages, so you decide to force the railing slash (as traditionally that denotes further layers underneath). But then you'll realise you have embedded images in some pages with URLs ending in ".jpg" or ".png". With those, they're files (hence the file extension at the end of the URL) so with those you'd usually want to strip the slash instead of forcing it

At that point you'd have to write something that said, force trailing slash unless the URL ends with a file extension, in which case always remove the slash (or similar)

Picking the right structural format for any site usually takes a while and involves quite a bit of research. It's a variable answer, depending upon the build of the site in question - and how it has been linked to externally, from across the web

I certainly think, that too many people use the canonical tag as a 'cop out' for not creating a unified, strong, powerful on-site architecture. I would say do stick with the 301s and consolidate your site architecture, but do some crawling and backlink audits - really do it properly, instead of just taking someone's 'one-liner' answer online. Here at Moz Q&A, there are a lot of people who really know their stuff! But there's no substitute for your own research and data

If you're aiming for a specific architecture and have been told it could break the site, ask why. Try and get exceptions worked into your recommendations which flip the opposite way -  i.e: "always strip the trailing slash, except in X situation where it would break the site. In X situation always force the trailing slash instead"

Your ultimate aim is to make each page accessible from just one URL (except where parameters come into play, that's another kettle of fish to be handled separately). You don't have to have EVERYTHING on the site one way or the other in 'absolute' terms. If some URLs have to force trailing slash whilst others remove it, fine. The point is to get them all locked down to one accessible format, but you can have varied controlled architectures inside of one website

I'd tread very carefully here as thing 1 and thing 2 seem to contradict each other at face value. You're right, Google can send traffic to a site in ways other than keywords, but it's not the norm. The next thing I'd look at is, hmm - how are we tracking keyword rankings? Is it an online, cloud based rank tracker that relies on you specifying all of (and all of the right) keywords to track? Most of those trackers track between 50 and 300 KWs (daily, weekly) but it's not uncommon for such sites to have 10,000+ keywords contributing. If they're not all in there, it's a bad sample you are looking at. Connect Google Search Console to Google Analytics. let it run for a few weeks, analyse the 'search query' data from within Google Analytics (which can be done once it's all hooked up). GSC only lets you export 1k keywords (usually, sometimes it can be more) but GA will take 5k and that's much better for your analysis. You might be surprised to find, those pages rank for more keywords than you thought. maybe hundreds of little ones, instead of a few big ones

It's horrible that you have been forced to adopt this as it's basically forced suicide of the website

You're likely up against large aggregator sites who have more commercial properties  (inside of and outside of New York) on  their books than you do. Places where users can go and just process one request, to see 20-30% of all available online results. These sites have a massive USP as for most of us who are lazy, they are one-stop shops we learn to trust

To assail and overcome these giants, even around the more-niche fringes, you need solid value-propositions and value-add for your site. Why go to you, instead of an easier, larger supplier?

Maybe you have some reasons:

• Better prices
• Better customer service
• Easier to use website once you register

BUT - being forced to register is a massive value-subtraction measure. It subtracts HUGE amounts of bonus points from your value-proposition, which is ultimately what you ride on to rank well

Not only are you having bounce rate issues, your search traffic may soon tumble into the dirt. Right now, your organic search (SEO) performance seems pretty solid, looking at the charts on Ahrefs (which I still think, are superior to the SEMRush ones by a long shot). This single change, could jeopardize all of that

By the way, even if I change my use agent to Googlebot, the form still comes up - so they will definitely be aware very soon (if not already) of this user obstruction. It's such a shame because it looks as if you guys started lifting in May 2018, and have been on the up-and-up ever since. To have such a change forced is literal insanity

Maybe there are some small CRO things you can do to get 3-15% more conversions or leads. But if your SEO traffic plummets 60% you will just be left thinking "wow, that was a terrible, terrible deal"

You've already given people the easy options by allowing them to log-in with Google or FaceBook. But people only really like to do that for big brands they can trust, otherwise they feel they are somehow connecting their personal accounts with a site they are not 'quite sure' that they trust yet

Sorry to say but I think this was a bit of a Doomsday maneuver

Your only real option is to incentivise account creation with deals and discounts or something like that. People are pretty astute these days, they will want something back for their trouble

In reality it would be better to wait until users were about to convert (going through whatever conversion funnel you have set up) and then at the very end, when you already have all of their info (which they need to give you to properly contact you) - set up an auto-filled form (based on the data they have already input) and ask if they want to create an account in a simple, easy 1-click style

That would have (probably) increased your uptake in account creations. But whomever was making the decisions, got too greedy and got forceful. It won't work and it may have severe repercussions

Be VERY careful

Thing #1) Just because you stop Google indexing and crawling some pages, that doesn't mean they will give that same traffic (keywords linking to those pages) to other URLs on your site. They may decide that your other URLs, do not satisfy the specific keywords connecting with the fundraiser URLs

Thing#2) CHECK. Go onto Google Analytics and actually check what percentage of your Google traffic (and overall traffic, I guess) comes specifically through these URLs. If it's like 2-3%, no big deal. If most of your traffic comes to and lands on these pages, no-indexing them all could be the single largest mistake you'll ever make

Blog posts and articles are fun but no substitute for checking your own, real, actual, factual data. Always always do that

It's all in this help document:

https://mza.bundledseo.com/help/moz-procedures/crawlers/rogerbot

"Crawl Delay To Slow Down Rogerbot

We want to crawl your site as fast as we can, so we can complete a crawl in good time, without causing issues for your human visitors.

If you want to slow rogerbot down, you can use the Crawl Delay directive. The following directive would only allow rogerbot to access your site once every 10 seconds:

User-agent: rogerbot

Crawl-delay: 10"

So you'd put the specified rule in your robots.txt file

I'm with Joe on the over 80% rule myself

Just so you know Meta no-index can be applied through the HTML but also through the HTTP header which might make it easier to implement on such a highly generated website

The pages will compete against each other under normal circumstances, but that's not necessarily an awful thing. For example, maybe your older page only achieved positions 16-30 to the keyword, but the new page might achieve a higher ranking. Unless you pit them against each other, how will you know what's best?

Stopping newer pages competing for old rankings, doesn't give a magical bonus to the old page and make it rank higher. Unless you're absolutely certain that the old page should be the 'definite' landing page for the keyword, a bit of friendly competition doesn't usually hurt much

The pages which really contend for your rankings, are those from other websites. Good luck emailing all the webmasters and complaining at them, that they are using your keywords

Sometimes, under very specific circumstances, keyword cannibalisation can come into play and cause problems. But 90% of the time it's just not really that big of a deal

The big deal is that if you write loads of pages with the same focus keyword, you're NOT writing about new keywords. And if you're not doing that, how will you increase your footprint? Often it's more lucrative to cover other, newer material rather than re-hashing old stuff

The worst you tend to get are rankings that stay largely in the same place, but their ranking URL jumps around as Google tries to decide which page to rank (and then eventually settles on one)

IMO, the worst part about keyword cannibalisation is not the fall-out from it (which is usually minimal) - it's the WASTED time, in terms of getting onto new topics to attract new visitors. Always be expanding

Darn. If you'd be willing to share the domain I'd be willing to do a full crawl of it and return the data to you. If there are no glaring technical errors then at least we'd know it was probably a combo of content issues and off-site popularity metrics which could be corrected over time

Up to you though. Seems complex though, like we should begin ruling stuff out

Robots.txt controls crawling, not indexation. Google will still sometimes index pages they cannot crawl. Canonical tags are for duplicate content consolidation, but are not a hard signal and Google frequently ignores them. Meta no-index tags (or X-robots no-index through the HTTP header, if you cannot apply Meta no-index in the HTML) is a harder signal and is meant to help you control indexation

To be honest if the pages are identical why not just 301 redirect the relevant searches (the top-line ones, which result in pages exactly the same as your regional ones) to the regional URLs? If the pages really are the same, it won't be any different for users except for a small delay during the redirect (which won't really be felt, especially if you are using Nginx redirects)

If you can't do that, you're really left with the Meta no-index tag and the canonical tag. Canonical tags avoid content duplication penalties but are a softer signal and they don't consolidate link equity like 301 redirects do (so in many way, there's not actually that much different between Meta no-index and canonicals, except canonical tags are more complex to set up in the first place as they require a destination field)

I'd probably just Meta no-index all the search URLs. Once Google had swallowed that, I would then (after 2-3 weeks) apply the relevant robots.txt rules

If you apply them both at the same time, Google won't be able to crawl the search URLs (since your robots.txt rule will block them) and therefore they will be blind to your canonical / Meta no index directive(s). So you have to handle de-indexation first, and THEN after that block the crawling to save your crawl allowance a bit

But don't do it all at once or you'll get in an unholy mess!