To improve the natural ranking of search engines, it is necessary to have a basic understanding of how the engines work. Clarifying the rationale allows the root causes to be drugged, thereby effectively increasing the flow。
I. Spider-crawl and capture page contents
Spider. Search engine crawling and access to page programs. Spiders visit any website first, robots. Txt of the root directory, climb the site according to agreement。
2. Tracking links. In big pages, spiders need some creeping strategies to track as many pages as possible. Depth and breadth priorities are common and the two strategies are usually used in combination。
3. Attract spiders. Because of time constraints, spiders are not able or will not be able to capture all the pages, and will, as far as possible, be able to capture the pages they consider important. So it has to make the page attractive enough for the spider to climb down。
Address library. In order to avoid double crawling and scratching, the backstage needs to group the pages that have been discovered: uncaptured, captured. When a spider discovers a link, it does not immediately grab it, but first puts it in the address library and then arranges for it to be retrieved。
Document storage. The captured data will be placed in the original page database, where the file contents stored are exactly the same as those shown during user browsing. Each url has a unique file number。
Reproduction of content testing. When climbing, spiders perform a degree of replicating, which is likely to stop and not be recorded for low weight copying。
Pre-treatment
Pretreatment, also referred to as “index”. The original pages taken by spiders are not directly for ranking purposes. Because the number of pages in the database is as high as hundreds of billions, it is not possible to rank them in real time by analysing relevance, so pre-processing is necessary to prepare for subsequent rankings。
1. Extracting text. The web page contains a wide range of content not related to the HTML label code or actual content and needs to be processed to extract visible and special text associated with the content (m)(e) tea tag, anchor text, etc.)。
2. In chinese. This is a step unique to the chinese search engine and unlike english, which has a natural separator, chinese needs a first word. Storage, processing and searching are done on a word basis. There are two dictionaries: based on dictionaries matching, and statistically based, both methods are usually used in combination. Seos can do less on the syllables, but can in some form indicate search engines, for example, by marking the word "sugar" in bold。
To stop the words. In practice, there are often a number of very frequent words or adjectives that do not affect content, such as “the” and “the” adjectives, “the” and “the” adjectives or adjectives, which need to be removed。
4. Elimination of noise. There are also many items that do not contribute to the subject of the page, such as copyright statements, navigational regulations, advertising, etc. The basic method of noise abatement is to divide pages according to HTML labels, where a large number of repeat blocks tend to be noise。
Weight. It is now common to copy an article, which may soon appear on a number of different sites, and the search results generally only return to one of the same articles, which needs to be weighed earlier。
6. Positive index. Abbreviated as an index refers to the conversion of the original page content to a collection of keywords, where information such as the weight of keywords is also recorded and a vocabulary structure is stored in the index library. The positive index is a mapping of files to keywords, and such information is still not directly available for use in ranking, as it requires scanning all documents in the index library to identify keywords, which are too inefficient to meet real-time needs。
7. Inverted index. Reconstruct the database as a map of keywords to the file, and the keywords become the primary key, corresponding to a series of documents. At this point, when the user searches for a keyword, all documents containing the keyword are quickly available without having to go through all the files。
8. Linking calculation. Google pr is the main reference, and other search engines have similar reference values, except not pr。
9. Processing of special documents. Processing of PDF, word, wps, xls, ppt, txt, etc。
10. Quality judgement. During the pre-processing phase, the search engine makes a qualitative judgement of the page, based on many factors, not just keyword extraction, links, etc。
With this series of preparatory work, the search engine is ready to process the user search。
