Technological Search engine optimisation: Technological Search engine optimization refers to Web-site and server optimization that can help the crawler in crawling, indexing and ranking operations to rank your web site better. txt file is then parsed and can instruct the robotic regarding which web pages are not for being crawled. https://www.youtube.com/watch?v=V7lEomNLUOE&t=8s