How to Crawl the Web with Scrapy (Python)
added on 2021/09/14 @ 12:42:44 | 1847 views| category: programming

Web scraping is the process of downloading data from a public website. For example, you could scrape ESPN for stats of baseball players and build a model to predict a team’s odds of winning based on their players stats and win rates.

One use-case I will demonstrate is scraping the website for job postings. Let’s say you are looking for a job but you are overwhelmed with the number of listings. You could set up a process to scrape indeed every day. Then you can write a script to automatically apply to the postings that meet certain criteria.

tags: #programming #web #scraping #python