Website-Crawler-Python
This is a simple website crawler which asks for a website link from the user to crawl and find specific data from the given website address. After getting the website address, it asks for how much crawling depth the user wants in between the number of links has been found after providing the website address.
Website Crawler takes 3 inputs:
- A website address
- Integer value for the crawling depth
- A user specified regular expression to find user specific data
General tasks:
- Find all the Nowgegian mobile numbers and saves into a text file.
- Find all the sub-links inside the given website and saves into a text file.
- Saves the website’s raw HTML code into a text file.
- Find all email addresses and save into a text file.
- Find all the comments used in the website and saves it into a text file.
- Find five most used words and print it into the terminal.
This is a Python based project and used some dependent libraries to execute the functionalities.
- RegEx
- Urllib3
- BeautifulSoup 4
- Counter in Collections
