Sometimes a website does not all want to be recorded, such as a page that is being tested, a dead link, a copy page, etc. These page grabs consume the total time allocated to the site by the search engine, making it impossible to capture and record really meaningful pages。
If it is found through the log file that these meaningless pages have been seized repeatedly and in large numbers, and if they have not been captured at all, then it should be directly prohibited to capture meaningless pages. The capture and recording are different processes. Use robots files to ensure that pages are not captured。
When the search engine spider visits the website, you will first see whether there is a pure text file named robots. Txt under the root directory of the site, which is used to direct the search engine to prohibit or specify permission to capture certain elements of the site。
I. Uncle now teaches you to use a few commonly used robots:
A: allow all search engines to access all pages and content
United:*
Allow: /
B: prohibit all search engines from accessing all pages and content
United:*
Disallow: /
C: prohibits getting caught on a particular page of the website (e. G. The page is “xinwen”)
United:*
Allow: /
Disallow/xinwen/
I'm sorry.
Of course, we have a lot of orders from uncles, and we want to know more about uncles。
The robots files have to be set according to the capture of their website. They need to look at the website logs to see which pages spiders have captured, and do they have any impact on the overall traffic ranking of the website? Practical analysis of the situation。
Ii. Map of the website:
Uncle here says the site map is not a 100-degree map showing the company's address. Correct as follows:
Definitions: site maps, also known as site maps, are a page with links (note: not necessarily all pages) to all pages that need to be captured by the search engine on the site, a container for all links to the site, and spiders very much like site maps。
Format: 1. HTML version is a page available to users on the website listing links to all major pages of the website. Depending on your website, if you have too much navigation and too much content, you can list a site map on a major page that is user-friendly。
2. Xml version, sitemap. Xml. It's a map captured by the spiders of the search engine, and the site map creates an easy and fast route for spiders to climb。
Impact: easy to climb by spiders; easy to view the website by visitors; it can increase the weight of the website while increasing its intake rate。
