# Web Scraper in Python — With CSV Export

Web scraping is one of the most valuable tools in a programmer’s toolbox. Whether you're gathering data for research, monitoring prices, or automating content aggregation, scraping allows you to extract useful information directly from websites.

In this article, we’ll build a **simple web scraper** in Python to:

* Scrape **article titles and links** from [Hacker News](https://news.ycombinator.com/)
    
* Save the scraped data to a **CSV file** for further use
    

Let’s walk through the entire process step-by-step, with clear explanations of what’s happening at each stage.

## What You’ll Need

Before we begin, install the following Python libraries if you haven’t already:

```bash
pip install requests beautifulsoup4
```

We’ll be using:

* `requests` – to download the webpage
    
* `BeautifulSoup` (from `bs4`) – to parse and search HTML
    
* Python’s built-in `csv` module – to write the scraped data to a file
    

## Step-by-Step Guide

Let’s build the scraper together from the ground up.

### Step 1: Import the Required Modules

```python
import requests                    # For sending HTTP requests
from bs4 import BeautifulSoup     # For parsing HTML
import csv                        # For writing to a CSV file
```

We import the necessary libraries. These will help us:

* Fetch a web page (`requests`)
    
* Parse and extract specific data from it (`BeautifulSoup`)
    
* Save that data in structured format (`csv`)
    

### Step 2: Define the Target URL and Fetch the Page

```python
url = 'https://news.ycombinator.com/'  # The website we want to scrape
response = requests.get(url)           # Send HTTP GET request
```

We’re targeting Hacker News, a tech-focused site where article links are clearly marked.  
`requests.get()` sends a GET request and stores the response in `response`.

### Step 3: Check for Request Success

```python
if response.status_code != 200:
    print("Failed to load page. Status code:", response.status_code)
    exit()
```

We check if the request was successful (HTTP status code 200). If not, we exit the script to avoid processing invalid content.

### Step 4: Parse the Page Content with BeautifulSoup

```python
soup = BeautifulSoup(response.text, 'html.parser')
```

We turn the raw HTML text into a BeautifulSoup object. This allows us to search for specific tags and attributes easily.

### Step 5: Find All Article Titles

```python
titles = soup.find_all('a', class_='titlelink')
```

On Hacker News, every article link is inside an `<a>` tag with class `titlelink`. This line finds all such tags and stores them in a list called `titles`.

### Step 6: Display and Collect the Data

```python
scraped_data = []

for i, title in enumerate(titles, 1):
    article_title = title.text
    article_link = title['href']

    print(f"{i}. {article_title}")
    print(f"   Link: {article_link}\n")

    scraped_data.append([article_title, article_link])
```

* We loop through each title tag.
    
* Extract and print the article's text and URL.
    
* Append each `[title, link]` pair to a list called `scraped_data` for later saving.
    

### Step 7: Save Results to a CSV File

```python
with open('headlines.csv', mode='w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(['Title', 'Link'])      # Write header row
    writer.writerows(scraped_data)          # Write all rows of scraped data

print("✅ Data saved to 'headlines.csv'")
```

We open a new file called `headlines.csv` and:

* Write a header row: `"Title", "Link"`
    
* Write each title/link pair from `scraped_data`
    
* Confirm the process is complete with a print message
    

The file is saved in the same directory as your script. You can open it with Excel, Google Sheets, or a text editor.

## Summary of What You’ve Built

* Scraped data from a live website
    
* Parsed specific content from HTML
    
* Saved the data in a structured format (CSV)
    
* Printed the data for visual verification
    

This project is small but powerful. It’s a solid template for scraping many other types of pages.

## What You Can Try Next

* Scrape additional details like author names or points
    
* Add pagination support to scrape multiple pages
    
* Schedule the scraper to run daily or weekly
    
* Add error handling for missing data or invalid links
    

## Sample Output (CSV)

| Title | Link |
| --- | --- |
| GPT-4 Release Details | [https://openai.com/blog/gpt-4](https://openai.com/blog/gpt-4) |
| Firefox’s New Privacy Features | [https://mozilla.org/blog/new-features](https://mozilla.org/blog/new-features) |
| How SQLite Works Internally | [https://sqlite.org/internals](https://sqlite.org/internals) |

## Final Tip: Always Scrape Responsibly

* Respect website `robots.txt` rules
    
* Avoid sending too many requests in a short time
    
* Never use scraped data for spamming or violating terms of service