Home Google Index Pages Should I block duplicate pages using robots.txt?

Should I block duplicate pages using robots.txt?

Halfdeck from Davis, CA asks: “If Google crawls 1000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using META ROBOTS NOINDEX,NOFOLLOW a better alternative?” Short answer: No, don’t block them using robots.txt. Learn more about duplicate content here:

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn
21 Comments  comments 

21 Responses

  1. kristinedwards58

    great clip keep it up? =)

  2. agapitoflores001

    I think this has already been answered on previous videos. But anyway, like other? videos, it helps a lot.

  3. John Britsios

    Using 304 If-Modified-Since in combination of a meta robots directives “noindex,nosnippet,noarchive,follow” would be the best way? to go. Everything else is simply BS.

  4. krishthewiz

    It sounded like the answer at the end was that we should not block. I also think that depending on the case you have to do a combination of these techniques, meta-robots, robots.txt and canonical, especially if re-architect-ing the site is not an easy option (in some CMSes’ its never an option). Tried the parameter filter that Google provided, and it doesn’t work as fast as I wanted it to. leaving Google to identify the dupe increase the dupes cache count 10x and resulted in a? rank fall.

  5. ericofranco

    agreed. He should mention canonical as the best choice here.?

  6. palbertus

    “We can figure out the dups on our own”.
    Looks like Google would prefer to? crawl all your site and take the filtering job on their own !

  7. dyoungprod

    Surprised canonical isn’t mentioned as a solution? here

  8. iceveiled

    Actually web users have made google popular and the most used search engine, so if you want to point fingers, blame the collective world using the internet. I don’t know about you but I don’t want to go back to 1998 when search query results were filled with pages with ridiculous keyword spamming, hidden text, and the like. I’m not saying that people? aren’t gaming the google algorhytms as they are and forever will. But search has improved, thanks to google.

  9. SEOMofo


    At the beginning of the video, it sounds like your answer is we SHOULD NOT block the URLs, because Google needs to crawl everything and figure out the duplicates for itself.? But then at about 0:57 you seem to reverse your stance by saying we SHOULD block them.

    Can you please clarify?



    You only have to jump through the hoops if you want Google to index your site and if? you want to rank highly. If you aren’t concerned about search engines or ranking of your site, then you can completely ignore the “hoops”.

  11. infiltrator7777

    That still doesn’t answer the question.

    There are no rules? here. He’s only making assumptions.

    Do you think you know?

  12. bigal21110

    so does it mean that if we have? dups on our page, that doesnt hurt our ranking?

  13. wicked4u2c

    They are not rules by “Google”. They are standards for their Search Engine, just like Bing and others have theirs.? At least Google has a great channel to help developers make sure their sites are optimized well for Google Search Engine. After all, Google is the most used Search Engine out there.

  14. PropellerBusted

    Buddha? knows many things about the Internet.

  15. MsIleaneSpeaks

    I’m not a webmaster and after hearing this answer, I don’t think? most people that “think” they are webmaster really are either. LOL

    Thanks for answering a question that I’m sure many people have.

  16. jschroeffel

    No mention of canonical? with this question? odd

  17. infiltrator7777

    Isn’t it SAD? how we have to jump through hoops now because of Google rules??

    Who says Google makes the internet rules?

  18. damianemanuel

    thx Matt, i? love your videos.

  19. carterblizz

    what about? canonical link tag?

  20. jazz0900

    doesn’t creating duplicate content in? the first place cause you to get ranked down?

  21. fferrero2008

    ..and leaves the? train 4 hours… LOL

    Nice video, nice answer!