Back to search

South Dakota State University

An Enhanced Malicious Web Crawler Detection and Classification System

Abstract

dc:description.abstract

In recent years, the impact of Web crawler is becoming more and more significant. Web crawler is widely used on both commercial and research institutional Web sites which brought convenience to many people. The sword has two blades. With the positive function, there are many intentions behind Web crawler and they are often unscrupulous when it comes to Web site integrity, and traditional network security technology has limitation. Therefore, it is very important to find an effective approach to analysis and identify visits and distinguish Web crawler from other accesses. However, common methods used to detect Web crawlers can detect Web crawler, but still cannot distinguish undesirable crawlers from welcome crawlers. This thesis proposed a trap-based approach to detect Web crawler and determining rules for classifying Web crawlers accurately. Three methods are used in the detection system - hidden link, robots.txt and submission button. Compared with the system just use hidden link or behavior analysis, the detection system can distinguishes undesirable crawlers from human users and welcome crawlers. Finally, the test results are evaluated and analyzed to show the improvements.

Degree

thesis:*
Name thesis:degree_name
Master of Science (MS)
Level thesis:degree_level
Thesis - University Access Only
Discipline thesis:degree_discipline
Electrical Engineering and Computer Science

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zhong, Tianying

Identifiers

dc:identifier.*
Repository record dc:identifier
https://openprairie.sdstate.edu/etd2/1697

Chain of custody

source
Harvested from
South Dakota State University
Base URL
openprairie.sdstate.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Zhong, Tianying. An Enhanced Malicious Web Crawler Detection and Classification System. Thesis - University Access Only thesis, https://openprairie.sdstate.edu/etd2/1697