What is Robot.txt File /simple guideline for Robots.txt in SEO.

 

WHAT IS ROBOTS.TXT ?

Robots.txt is very is important in SEO. Robots.txt file tells a spider or crawler where they can go and where they not go.

Robots.txt file is actually a file where we want to explain a crawler a complete pathway.

The Sample of the Robots.txt File:

User-agent: *
Disallow: /search
Allow: /

                    
Sitemap: https://YourDomain.com/atom.xml?redirect=false&start-index=1&max-results=500

Why Robots.txt is important?

There are many example where a robots.txt file is important

1:For example we have a several pages of a website and we want only few pages to crawl. For this purpose we use  robots.txt file. Google crawler has a time budget they have a specific time to crawls our website so in this time we have to crawl a important pages and for others pages we use a robots.txt file.

2: for many reasons we don’t want to crawl a few website pages for this we use a robots.txt file

Some important things of robots.txt file .

·         Robots.txt file is always in footer .

·         Robots.txt file is always saved in robots.txt file .

·         https://www.your website.com /robots.txt.

·         search engine is not bound to follow.

·         Robot.txt have only one file in a website

·         Robots.txt file has one or more groups.

·         Then test the rebots.txt file

For testing rebort.txt file we use a rebort.txt Tester

How can we see Robots.txt file in Google ?

If we want to see a robots.txt file of facebook we have to just write a

Facebook/robots.txt

Syntax of Robots.txt file 

user -agent *

Disallow:

above statement means user agent have allowed to crawl a website.

* means all robots have allowed to crawl a website. 

user -agent *

Disallow:/ 

if we use a / in a user agent then it means it is not allowed to crawl.

We can use a robots name instead of *

for example 

user-agent :Googlebot

Disallow : it means we allowed Google bot to crawl  a website.

 Example 2:

user-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

user agent name googlebot is not allowed to crawl any URL in a website

all other bots are allowed to crawl .
if we have a / then boots have a option to crawl and if they have / in opening and /in end then they have not allowed to crawl  website