Web Scraper in Python — With CSV Export

Web scraping is one of the most valuable tools in a programmer’s toolbox. Whether you're gathering data for research, monitoring prices, or automating content aggregation, scraping allows you to extract useful information directly from websites.

In this article, we’ll build a simple web scraper in Python to:

• Scrape article titles and links from Hacker News

• Save the scraped data to a CSV file for further use

Let’s walk through the entire process step-by-step, with clear explanations of what’s happening at each stage.

What You’ll Need

Before we begin, install the following Python libraries if you haven’t already:

pip install requests beautifulsoup4

We’ll be using:

• requests – to download the webpage

• BeautifulSoup (from bs4) – to parse and search HTML

• Python’s built-in csv module – to write the scraped data to a file

Step-by-Step Guide

Let’s build the scraper together from the ground up.

Step 1: Import the Required Modules

import requests                    # For sending HTTP requests
from bs4 import BeautifulSoup     # For parsing HTML
import csv                        # For writing to a CSV file

We import the necessary libraries. These will help us:

• Fetch a web page (requests)

• Parse and extract specific data from it (BeautifulSoup)

• Save that data in structured format (csv)

Step 2: Define the Target URL and Fetch the Page

url = 'https://news.ycombinator.com/'  # The website we want to scrape
response = requests.get(url)           # Send HTTP GET request

We’re targeting Hacker News, a tech-focused site where article links are clearly marked.
requests.get() sends a GET request and stores the response in response.

Step 3: Check for Request Success

if response.status_code != 200:
    print("Failed to load page. Status code:", response.status_code)
    exit()

We check if the request was successful (HTTP status code 200). If not, we exit the script to avoid processing invalid content.

Step 4: Parse the Page Content with BeautifulSoup

soup = BeautifulSoup(response.text, 'html.parser')

We turn the raw HTML text into a BeautifulSoup object. This allows us to search for specific tags and attributes easily.

Step 5: Find All Article Titles

titles = soup.find_all('a', class_='titlelink')

On Hacker News, every article link is inside an <a> tag with class titlelink. This line finds all such tags and stores them in a list called titles.

Step 6: Display and Collect the Data

scraped_data = []

for i, title in enumerate(titles, 1):
    article_title = title.text
    article_link = title['href']

    print(f"{i}. {article_title}")
    print(f"   Link: {article_link}\n")

    scraped_data.append([article_title, article_link])

• We loop through each title tag.

• Extract and print the article's text and URL.

• Append each [title, link] pair to a list called scraped_data for later saving.

Step 7: Save Results to a CSV File

with open('headlines.csv', mode='w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(['Title', 'Link'])      # Write header row
    writer.writerows(scraped_data)          # Write all rows of scraped data

print("✅ Data saved to 'headlines.csv'")

We open a new file called headlines.csv and:

• Write a header row: "Title", "Link"

• Write each title/link pair from scraped_data

• Confirm the process is complete with a print message

The file is saved in the same directory as your script. You can open it with Excel, Google Sheets, or a text editor.

Summary of What You’ve Built

• Scraped data from a live website

• Parsed specific content from HTML

• Saved the data in a structured format (CSV)

• Printed the data for visual verification

This project is small but powerful. It’s a solid template for scraping many other types of pages.

What You Can Try Next

• Scrape additional details like author names or points

• Add pagination support to scrape multiple pages

• Schedule the scraper to run daily or weekly

• Add error handling for missing data or invalid links

Sample Output (CSV)

Final Tip: Always Scrape Responsibly

• Respect website robots.txt rules

• Avoid sending too many requests in a short time

• Never use scraped data for spamming or violating terms of service