Screaming Frog Log File Analyser

The tool lets users upload server log files and parses them to verify the activity of search‑engine and AI bots. It extracts crawled URLs, response codes, redirect types and timing information, storing the data in a database that can handle millions of log events. By analysing this information, users can see which URLs are accessed, how often bots visit, and which pages generate errors or large payloads.

Typical users are SEO professionals and web analysts who need to understand bot behaviour, audit redirects, detect broken links, and identify orphan or uncrawled pages. The software also supports importing external URL lists for cross‑referencing, enabling deeper investigations such as comparing crawl data with internal metrics or link reports.

What distinguishes the analyser is its ability to process very large log files efficiently on macOS, Windows and Linux, and to provide both a free tier (limited to 1 000 log events) and a paid licence that removes that limit. The interface offers built‑in reports on crawl frequency, crawl budget usage, and performance‑related metrics, helping users make data‑driven decisions about site optimisation.