Make use of your robots.txt file to direct online search engine World wide web crawlers on what to crawl. A robots.txt file can prevent your site from receiving crawled and indexed, so Verify your set up with Google’s cost-free robots.txt tester tool. когда они платные, то есть Вы заплатили сайту https://rsac.org