Scraping_Nairaland
This project scraped comments from the political section of popular Nigerian blog www.nairaland.com using the Python BeautifulSoup library, and saved the extracted comments and links in a CSV file.
Create a new environment and install requirements
Create a new Anaconda environment with Python 3.8( Other Python versions could also be used), and pip install the required librairies using:
‘pip install -r Requirements.txt’.
Run the Nairaland_pol_section.py file
This could be done using ‘python Nairaland_pol_section.py’ on Mac’s terminal.
Saved CSV File
The generated CSV file would be saved as ‘Nairaland_df.csv’
Cleaning
To read the generated ‘Nairaland_df.csv’ file in Jupyter Notebook, Google Colab, etc; you may want to replace ‘Nairaland_df.csv’ with own path in, for example, ‘full_data= pd.read_csv(‘Nairaland_df.csv’,lineterminator=’\n’)’. Codes for cleaning are available in the ‘Cleaning_Nairaland_Scraping.py’ file.
Number of Comments
The script only extracts all comments in the first 10 pages as shown in “for i in range(0,10):” from the ‘Nairaland_pol_section.py’ file.
Possible Use
A sentiment analysis could be performed on the extracted comments.
