Social Media Impact: Preferred tendencies on social media marketing can spill above into search engines like google, affecting the search volume of similar keywords and phrases. txt file is then parsed and can instruct the robot regarding which webpages are not to be crawled. Like a online search engine crawler https://www.youtube.com/watch?v=7OfVkYHB1OY