How to Get Data from Websites Without Copying and Pasting Forever

This is the fun part—you don’t write any code. AI tools (ChatGPT, Claude) write it for you.

What to Tell the AI

It’s like ordering food at a restaurant: be clear about what you want. Use your list from Step 2.

Copy this and send to ChatGPT or Claude:

I need help with web scraping. Here's what I need:

PROJECT: Get dentist information from a website

MY URL LIST: I have 300 page addresses saved in a file called "dentist_urls.txt"

DATA I WANT:
- Business name: //h1[@class='business-title']
- Phone: //a[@class='phone-link']/text()
- Email: //a[contains(@href, 'mailto:')]/text()  
- Address: //div[@class='address']/text()

SAVE TO: Excel file called "dentist_contacts.csv"

OTHER STUFF:
- Wait 3 seconds between each page (be nice to the website)
- Show me progress every 25 pages
- If there's an error, skip that page and keep going
- Add comments to explain what the code does

Make it simple for someone who doesn't know programming.

What the AI Gives You

The AI will give you complete code. Don’t worry about understanding every line—the AI explains it with comments.

Here’s what it looks like (comments explain what each part does):

# This gets dentist information from websites

import requests  # Visit websites
from lxml import html  # Read the pages
import csv  # Make Excel files
import time  # Add waiting time

# This visits one page and gets the data
def get_one_page(url):
    try:
        # Go to the page
        response = requests.get(url, timeout=10)

        # Read it
        tree = html.fromstring(response.content)

        # Get the data using your directions
        business_name = tree.xpath('//h1[@class="business-title"]/text()')
        phone = tree.xpath('//a[@class="phone-link"]/text()')
        email = tree.xpath('//a[contains(@href, "mailto:")]/text()')
        address = tree.xpath('//div[@class="address"]/text()')

        # Clean it up
        data = {
            'Business Name': business_name[0] if business_name else '',
            'Phone': phone[0] if phone else '',
            'Email': email[0] if email else '',
            'Address': address[0] if address else ''
        }

        return data

    except:
        print(f"Problem with {url}")
        return None

# Main program
print("Starting...")

# Read your page list
with open('dentist_urls.txt', 'r') as file:
    urls = [line.strip() for line in file]

print(f"Found {len(urls)} pages to scrape")

# Store results
all_data = []

# Visit each page
for number, url in enumerate(urls):
    print(f"Page {number + 1} of {len(urls)}...")

    data = get_one_page(url)
    if data:
        all_data.append(data)

    # Wait 3 seconds
    time.sleep(3)

    # Show progress
    if (number + 1) % 25 == 0:
        print(f"Done: {number + 1} pages")

# Save to Excel
with open('dentist_contacts.csv', 'w', newline='') as file:
    if all_data:
        writer = csv.DictWriter(file, fieldnames=all_data[0].keys())
        writer.writeheader()
        writer.writerows(all_data)

print(f"Finished! Got {len(all_data)} results in dentist_contacts.csv")

The lines starting with # explain what happens. You don’t need to remember this—AI writes it!

How to Run It (Even If You Never Coded)

Way 1: Google Colab (Easiest – Free)

1. Go to colab.research.google.com
2. Click “New Notebook”
3. Paste the code
4. Upload your page list file
5. Click the “Play” button ▶️
6. Download your results

Way 2: Put Python on Your Computer

1. Go to python.org and download (free)
2. Install it (just keep clicking “Next”)
3. Save the code as scraper.py
4. Open Command Prompt (Windows) or Terminal (Mac)
5. Type: python scraper.py
6. Watch it work!

Way 3: Pay Someone ($10-50)

• Go to Fiverr.com or Upwork.com
• Find someone to run it for you
• Send them: your page list + your data list + the code
• They send back your data

Why This is Powerful

Before: 10 hours of copy-paste work for 300 contacts

After: 15 minutes to set up, 15 minutes to run = 300 contacts done

Time saved: 9+ hours you can use for real marketing work

This is where things get real. Websites don’t always like being scraped, so they try to stop you. Here’s how to fix the most common problems.

Problem #1: Website Blocks You

What happens:

• Works fine for 50 pages, then suddenly stops
• You get errors like “403 Forbidden” or “Access Denied”
• You see a “I’m not a robot” checkbox

Why: Websites can tell when someone visits too many pages too fast. They think you’re a robot (you are!) and block you.

How to fix:

Fix 1: Go Slower (Free)

• Wait longer between pages – change from 3 seconds to 5-10 seconds
• Tell the AI: “Make it wait random time between 5-10 seconds”
• Yes, this takes longer, but it works

Fix 2: Look Like a Real Browser (Free)

• Your scraper says “I’m a bot”—make it look like Chrome or Firefox
• Tell the AI: “Make it look like a real browser”

Fix 3: Use Proxies (Costs Money, But Works Best)

• A proxy is like wearing a disguise—makes you look like different people
• See the proxy part below
• Good for big jobs (1,000+ pages) or when you keep getting blocked

Problem #2: Empty or Missing Data

What happens:

• Your Excel has lots of blank spots
• Some data is there, some isn’t

Why:

• Not every page has all the information (some businesses don’t list email)
• Your XPath might be a little bit wrong
• The website might load data with JavaScript (tricky to scrape)

How to fix:

1. Check if the data really exists – Look at 5-10 pages yourself. Is the email really there?
2. Double-check your XPath – Go back to Step 2, make sure you copied it right
3. JavaScript problem? – If you see the data in your browser but scraper can’t find it, tell the AI: “The data loads with JavaScript – use Selenium instead”

Problem #3: Messy Data

What happens:

Business Name: "\n\n    John's Plumbing    \n\n"
Phone: "Call us: (555) 123-4567"

Why: Web pages have extra spaces, line breaks, and random text that gets scraped too.

How to fix: Tell the AI: “Clean the data. Remove extra spaces and text like ‘Call us:’ from phone numbers”

Problem #4: Scraper Stops With Error

What happens:

• Scraper stops at page #137 with an error
• You lose everything

Why:

• One page might be broken or offline
• Internet connection timed out
• Page looked different than others

How to fix: Tell the AI: “If one page has an error, skip it and keep going. Also, save progress every 50 pages so I don’t lose everything if it crashes.”

This is like auto-save in Word.

Problem #5: Takes Too Long

What happens:

• You have 2,000 pages
• Each takes 5 seconds
• Total time: 2.7 hours 😱

Why: You’re waiting between pages (which is good!), but it adds up.

How to fix:

Way 1: Run at night

• Start it before bed, check it in the morning
• Free and easy

Way 2: Split it up

• Break 2,000 pages into 4 groups of 500
• Run them one by one (or ask friends to help!)

Way 3: Use proxies + go faster

• With proxies, you can scrape faster without getting blocked
• Changes a 3-hour job to 30 minutes
• Costs money but worth it if you do this often

When Do You Need Proxies?

You DON’T need them if:

• Less than 500 pages
• One-time thing
• Website doesn’t block you
• You don’t mind waiting a few hours

You DO need them if:

• 1,000+ pages often
• Watching competitors daily (prices, products, content)
• Keep getting blocked even when going slow
• Time is important—you need data fast
• Scraping same site over and over

Money example:

• Doing it by hand: 20 hours × $50/hour = $1,000
• Using proxies: $50/month + 2 hours = $150
• You save: $850 (and get it done 10x faster)