Disallow: /jobs/? is this stopping the SERPs from indexing job posts
-
Hi,
I was wondering what this would be used for as it's in the Robots.exe of a recruitment agency website that posts jobs. Should it be removed?Disallow: /jobs/?
Disallow: /jobs/page/*/Thanks in advance.
James -
Hi James,
So far as I can see you have the following architecture:
- job posting: https://www.pkeducation.co.uk/job/post-name/
- jobs listing page: https://www.pkeducation.co.uk/jobs/
Since from the robots.txt the listing page pagination is blocked, the crawler can access only the first 15 job postings are available to crawl via a normal crawl.
I would say, you should remove the blocking from the robots.txt and focus on implementing a correct pagination. *which method you choose is your decision, but allow the crawler to access all of your job posts. Check https://yoast.com/pagination-seo-best-practices/
Another thing I would change is to make the job post title an anchor text for the job posting. (every single job is linked with "Find out more").
Also if possible, create a separate sitemap.xml for your job posts and submit it in Search Console, this way you can keep track of any anomaly with indexation.
Last, and not least, focus on the quality of your content (just as Matt proposed in the first answer).
Good luck!
-
Hi Istvan,
Sorry I've been away for a while. Thanks for all of your advice guys.
Here is the url if that helps?
https://www.pkeducation.co.uk/jobs/
Cheers,
James
-
The idea is (which we both highlighted), that blocking your listing page from robots.txt is wrong, for pagination you have several methods to deal with (how you deal with it, it really depends on the technical possibilities that you have on the project).
Regarding James' original question, my feeling is, that he is somehow blocking their posting pages. Cutting the access to these pages makes it really hard for Google, or any other search engine to index it. But without a URL in front of us, we cannot really answer his question, we can only create theories that he can test
-
Ah yes when it's pointed out like that, it's a conflicting signal isn't It. Makes sense in theory, but if you're setting it to noindex and then passing that on via a canonical it's probably not the best is it.
They're was link out in that thread to a discussion of people who still do that with success, but after reading that I would just use noindex only as you said. (Still prefer the no index on the robots block though)
-
Sorry Richard, but using noindex with canonical link is not quite a good practice.
It's an old entry, but still true: https://www.seroundtable.com/noindex-canonical-google-18274.html
-
I don't think it should be blocked by robots.txt at all. It's stopping Google from crawling the site fully. And they may even treat it negatively as they've been really clamping down on blocking folders with robots.txt lately. I've seen sites with warning in search console for: Disallow: /wp-admin
You may want to consider just using a noindex tag on those pages instead. And then also use a canonical tag that points back to the main job category page. That way Google can crawl the pages and perhaps pass all the juice back to the main job category page via the canonical. Then just make sure those junk job pages aren't in the sitemap either.
-
Hi James,
Regarding the robots.txt syntax:
Disallow: /jobs/? which basically blocks every single URL that contains /jobs/**? **
For example: domain.com**/jobs/?**sort-by=... will be blocked
If you want to disallow query parameters from URL, the correct implementation would be Disallow: /jobs/*? or even specify which query parameter you want to block. For example Disallow: /jobs/*?page=
My question to you, if these jobs are linked from any other page and/or sitemap? Or only from the listing page, which has it's pagination, sorting, etc. is blocked by robots.txt? If they are not linked, it could be a simple case of orphan pages, where basically the crawler cannot access the job posting pages, because there is no actual link to it. I know it is an old rule, but it is still true: Crawl > Index > Rank.
BTW. I don't know why you would block your pagination. There are other optimal implementations.
And there is always the scenario, that was already described by Matt. But I believe in that case you would have at least some of the pages indexed even if they are not going to get ranked well.
Also, make sure other technical implementations are not stopping your job posting pages from being indexed.
-
I'd guess that the jobs get pulled from a job board. If this is the case, then the content ( job description, title etc.) will just be a duplication of the content that can be found in many other locations. If a plugin is used, they sometimes automatically add a disallow into the robots.txt file as to not hurt the parent version of the job page by creating thousands of duplicate content issues.
I'd recommend creating some really high-quality hub pages based on job type, or location and pulling the relevant jobs into that page, instead of trying to index and rank the actual job pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I no-index categories of my blog?
I have blog with lots of articles & it also has lots of categories. These categories are currently indexed in the google and moz showing missing title and description for these categories. Should I place no-index tag in all the categories or leave it as it is?
Intermediate & Advanced SEO | | jhakasseo0 -
Redirect wordpress from /%post_id%/%postname%/ to /blog/%postname%/
Hi what is the code to redirect wordpress blog from site.com/%post_id%/%postname%/ to site.com/blog/%postname%/ We are moving the site to a new server and new url structure. Thanks in advance
Intermediate & Advanced SEO | | Taiger0 -
The Value of Backlinks - Blog posts
Alright, So I'm trying to understand the value of a backlink from a blogger/site owner. Now, as I know, (let's use fashion industry), there are thousands of fashion bloggers with good metrics... Domain Authority 40+ Page Authority 40+ etc. etc. for their home page. And there are THOUSANDS of fashion bloggers that link back to ecommerce sites (backlinks). My question is, is there really much of a value with these backlinks? Sure, the domain authority and page authority is high ( for the given example), BUT wouldn't the page authority automatically be a 1 for a "post" ? Le'ts say one blogger writes a new blog for July 14th and dedicates that blog post about the awesome shoes they have and it links back to an ecommerce site; well, what impact does that make in regards to SEO? And if it does, how long would that take? The biggest issue I see is the ROI. You link build, you get links - and then ....you wait. Or you hope that you'll move up in rankings, but yet they can take months for it to even move the needle. Especially the fact that if it's not a HOME PAGE link...there isn't that "much" of a boost compared to an internal page? So then the NEXT question I have too, wouldn't influencer outreach be the same as far as getting a PBN link, creating web 2.0's, etc.? Let's think about it. You're outreaching to someone so you can benefit something back. It's all unnatural at the end of the day. Would love to discuss. So in summary 1 - Value of influencer outreach links - especially if the post is a brand new content piece.. (page authority being automatically 1) 2- - What exactly IS white hat when really any type of link building is mimicking?
Intermediate & Advanced SEO | | ggpaul5620 -
Page Title shown in SERPS not the same as
Hi all, I'm trying to get a homepage to rank for a certain term, but the page keeps showing up in the SERPS with the "Brand Name: Keyword" when I have written it as "Keyword - Brand Name" in the <title>tag. I can't even see "Brand Name" Keyword" in the code of the page so I don't know where Google is pulling this from? </p> <p>I have <meta name="robots" content="noodp,noydir"/> on the page.</p> <p>I'm running Yoast and have removed the Brand from the Site Name and the Page Title for the homepage is "Keyword - Brand Name" in WordPress. I've changed the meta description so I can see the page has been crawled and re-indexed as the new meta description is showing in the SERPs</p> <p>Any idea, where Google is pulling this Page Title from and how I can get it changed to read the actual <title> tag? Or is there something I need to change in WordPress?</p> <p>Thank you!</p></title>
Intermediate & Advanced SEO | | Marketing_Today0 -
Keyword Targeting / Cannibalisation
Hi Guys We're about to launch a very large website for a flooring company and would like to find out more about _key word _cannibalisation - to put my mind at rest. I know Rand posted a Whiteboard Friday early last year about this topic and mentioned using part of the same keyword was ok to use. All our keywords are specifically geared for "user intent" meaning each keyword has relevance and the content to back up the keyword. We've ensured the keywords are located within each url, placed at the start of the page title, h1 etc.
Intermediate & Advanced SEO | | GaryVictory1 -
Back links Building and article/blog posting
Hi all, I have been researching the best way for back links building, and I would like to ask few questions before I start. Which one of these tools would you recommend for back link building diagnostics. www.linkrisk.com - www.linkdetox.com What would be the best procedure to begin creating healthy back links? Would looking at my competitors back links help me? What would be the recommended amount of back links created per week? Also how many blogs entries should we aim to create per week? The website i'm working on is manvanlondon.co.uk If you guys have any further suggestions please let me know. Many thanks for your time.
Intermediate & Advanced SEO | | monicapopa0 -
Sitelinks (breadcrumbs) in SERPs
Hi there, I have a .co.uk & .ie website both have the exact same content, only differences is the UK website is selling the product in pounds and the Irish website is selling in Euros plus both websites have different contact numbers. I decided to use rel canonical on the .ie pointing to the .co.uk website as I think it was having an issue in my SERPs for the .co.uk website in Google.co.uk, anyway since doing this, I am seeing strange things happening in SERPs for my keywords, for example if you click the link below, my website is number 2 for 'hot flushes' if you hover over or click on 'health or 'menopause' in the breadcrumbs in SERPs it takes you to the .co.uk website, is this normal? Click here
Intermediate & Advanced SEO | | Paul780 -
Is there a way to stop my product pages with the "show all" catagory/attribute from duplicating content?
If there were less pages with the "show all" attribute it would be a simple fix by adding the canonical URL tag. But seeing that there are about 1,000 of them I was wondering if their was a broader fix that I could apply.
Intermediate & Advanced SEO | | cscoville0