welcome to XRM blog

Keep in touch with latest CRM/ERP articles

To remain competitive your organisation must be efficient across the business process spectrum. To do so you need to take sound decisions based on a balance between the cost and risk. To do so you will be heavily dependent on your content management in itself needs...

image
Blog

Benefits of using robots.txt file in your website.

By Abhay Choudhary on 11/4/2016

The robots.txt file is not a HTML file it’s a simple text file that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters uses the robots.txt file to help the search engines index the content of their websites. All major search engines support the basic functionality it offers. There are some extra rules that are used by a few search engines which can be useful too.

If webmasters can tell the search engines spiders to skip pages that they do not consider important enough to be crawled, then they have a better opportunity to have their most valuable pages featured in the search results pages. The robots.txt file is a simple method of essentially easing the process for the spider to return the most relevant search results.

Making an empty robots.txt file is simple, just open up a text editor such as notepad, save the blank file as robots.txt or you can add this bit of code before you save it:

User – agent *

Disallow:

That bit of code tell all search engines that are welcome to crawl all areas of the website, “user-agent” means what search engine bits the instructions are for, the star (*) that comes after it means the instructions are for all bots. “Disallow:” means what parts of the website are not to be crawled, because there is nothing after it, bots know they can crawl everything. Be sure to keep the code on separate lines.

There are a couple of things that always need to be remembered when managing robots.txt records and these include:

·         The filename is case sensitive; please remember to use robots.txt and not Robots.txt.

 

·         The record is publically accessible and anyone has access to the file. It’s important to apply better security to sections of your website you are trying to keep hidden for security reason.

 

·         A single passage of prohibit is permitted per URL.

 

·         Along with the document the “no index, take after” tags should also be used on every related page.

 

This also increase spider ability for the search engines. Obviously, despite the fact that this a small aspects of the search engine optimization process, if used correctly, a robots.txt can be a significant benefit.

 

DotNetNuke
Webpage
Blog Calendar
Blog Calendar List
2024 Aug  4  1
2024 Apr  42  4
2024 Mar  77  4
2024 Feb  135  3
2024 Jan  24  7
2023 Dec  22  6
2023 Nov  219  5
2023 Oct  342  12
2023 Sep  878  9
2023 Aug  202  6
2023 Jul  44  6
2023 Jun  25  4
2023 May  44  5
2023 Apr  58  5
2023 Mar  155  6
2023 Feb  149  5
2023 Jan  59  4
2022 Dec  95  7
2022 Nov  275  2
2022 Sep  13  1
2022 Aug  31  2
2022 Jun  11  2
2022 May  6  2
2022 Apr  12  2
2022 Mar  2  1
2022 Feb  2  1
2022 Jan  1  1
2021 Dec  4  1
2021 Nov  2  1
2021 Oct  2  1
2021 Sep  14  1
2021 Aug  49  5
2021 Jul  50  4
2021 Jun  1549  5
2021 May  39  3
2021 Apr  2142  3
2021 Mar  207  5
2021 Feb  2420  7
2021 Jan  3564  9
2020 Dec  501  7
2020 Sep  80  3
2020 Aug  750  3
2020 Jul  134  1
2020 Jun  93  3
2020 Apr  85  3
2020 Mar  19  2
2020 Feb  34  5
2020 Jan  46  7
2019 Dec  17  4
2019 Nov  35  1
2019 Jan  23  2
2018 Dec  102  4
2018 Nov  68  3
2018 Oct  18  3
2018 Sep  1196  11
2018 Aug  7  2
2018 Jun  17  1
2018 Jan  70  2
2017 Sep  588  5
2017 Aug  17  1
2017 Jul  17  2
2017 Jun  64  2
2017 May  21  1
2017 Apr  38  2
2017 Mar  138  4
2017 Feb  817  4
2016 Dec  207  3
2016 Nov  881  8
2016 Oct  314  10
2016 Sep  749  6
2016 Aug  39  1
2016 Jun  1881  6
2016 May  111  3
2016 Jan  71  2
2015 Dec  590  6
2015 Nov  4  1
2015 Oct  13  1
2015 Sep  1470  6
2015 Aug  14  1
2015 Jul  128  2
2015 Jun  11  1
2015 May  20  1
2015 Apr  30  3
2015 Mar  80  3
2015 Jan  5340  4
2014 Dec  17  1
2014 Nov  2260  4
2014 Oct  69  1
2014 Sep  107  2
2014 Aug  5302  1
2014 Jul  49  2
2014 Apr  2588  12
2014 Mar  303  17
2014 Feb  220  6
2014 Jan  1510  16
2013 Dec  21  2
2013 Nov  693  2
2013 Oct  256  3
2013 Sep  11  1
2013 Aug  40  3
2013 Jul  214  1
2013 Apr  61  6
2013 Mar  2326  10
2013 Feb  131  3
2013 Jan  349  2
2012 Nov  61  2
2012 Oct  518  10
Tag Cloud
Interested in our services? Still not sure about project details? get a quote