# Web Grater

A very simple scrapy app for finding broken hrefs/images on your website. I've
used this on a number of website to help find broken content and hopefully it
can help you.

## How do I use this?

The project is dockerized but if don't want to use docker you can make a
virtualenv. First you need to update the site you want to crawl in
`web_grater/spiders/grater_spider.py`. After that you can just run:

```
make up && make crawl
```

This will print off all the urls the scraper can find and the broken ones in
red.

## How do I find which page contains the broken link?

In `grater_spider.py` there is a `CustomLinkExtractor` with a method called
`extract_links`. There is some commented out code that shows how you can search
for the broken link during link extraction and print the parent page in yellow.