Web Spider

🕷️ Module: Spider & Crawler Analytics

📍 Access:

After logging into UStat and selecting a website, navigate via left sidebar → [Spiders & Crawlers].

🧩 Functionality:

Critical for:

• SEO indexation health checks
• Scraping/security threat detection
• Sitemap optimization

📄 Page Structure:

1. Time Range Selector (Top)

• Presets: Today/Yesterday, This Week/Month, Last 7/30 Days, Custom Range

2. Crawl Activity Trends (Bar Chart)

• Tracks major search engine bots: (Googlebot, Baiduspider, Bingbot, YandexBot)
• Visualizes crawl volume fluctuations

📊 Measures indexation activity & site structure health

3. Traffic Composition (Pie Chart)

• Human Visitors vs. Bots ratio

      4. Crawl Detail Tables

• Top Crawled Pages: Highest-frequency URLs
  Bot Source Domains: Originating IPs/networks

📘 Key Terminology:

• Spider/Crawler: Automated programs that scan/webpages for search engine indexing

⚙️ Interactive Controls:

• Toggle between Page Views (PV) and IP-based metrics; Export crawl frequency reports

✅ Best Practices:

• Crawl Spike?
  Check: New content launches / Backlink surges / Sitemap updates
• Suspicious Bots?
  Block IPs or implement robots.txt rules for:
  Unknown crawlers
  Excessive duplicate scans
• Key Pages Ignored?
  Strengthen internal linking
  Resubmit sitemaps
  
• SEO Validation:
  Cross-check with Search Engine Analytics for organic traffic impact