Python & Web Scraping Projects for €250 - €500. Goal objective: Create a crawler that checks a set of urls (100-30.000 per client) on a daily basis and Extracts relevant pre-defined data (H1, Title, Meta-Data, status codes, canonical, as well as o. Apr 20, 2021 2) Octoparse Octoparse is a web scraping tool easy to use for both coders and non-coders and popular for eCommerce data scraping. It can scrape web data at a large scale (up to millions) and store it in structured files like Excel, CSV, JSON for download.

2014-04-21T07:00:03Z

A little over a year ago I wrote an article on web scraping using Node.js. Today I'm revisiting the topic, but this time I'm going to use Python, so that the techniques offered by these two languages can be compared and contrasted.

The Problem

As I'm sure you know, I attended PyCon in Montréal earlier this month. The video recordings of all the talks and tutorials have already been released on YouTube, with an index available at pyvideo.org.

I thought it would be useful to know what are the most watched videos of the conference, so we are going to write a scraping script that will obtain the list of available videos from pyvideo.org and then get viewer statistics from each of the videos directly from their YouTube page. Sounds interesting? Let's get started!

The Tools

There are two basic tasks that are used to scrape web sites:

1. Load a web page to a string.
2. Parse HTML from a web page to locate the interesting bits.

Python offers two excellent tools for the above tasks. I will use the awesome requests to load web pages, and BeautifulSoup to do the parsing.

The best project management software for mac. We can put these two packages in a virtual environment:

If you are using Microsoft Windows, note that the virtual environment activation command above is different, you should use venvScriptsactivate.

Basic Scraping Technique

The first thing to do when writing a scraping script is to manually inspect the page(s) to scrape to determine how the data can be located.

Web Scraping Flask Download

To begin with, we are going to look at the list of PyCon videos at http://pyvideo.org/category/50/pycon-us-2014. Inspecting the HTML source of this page we find that the structure of the video list is more or less as follows:

So the first task is to load this page, and extract the links to the individual pages, since the links to the YouTube videos are in these pages.

Loading a web page using requests is extremely simple:

That's it! After this function returns the HTML of the page is available in response.text.

The next task is to extract the links to the individual video pages. With BeautifulSoup this can be done using CSS selector syntax, which you may be familiar if you work on the client-side.

To obtain the links we will use a selector that captures the <a> elements inside each <div> with class video-summary-data. Since there are several <a> elements for each video we will filter them to include only those that point to a URL that begins with /video, which is unique to the individual video pages. The CSS selector that implements the above criteria is div.video-summary-data a[href^=/video]. The following snippet of code uses this selector with BeautifulSoup to obtain the <a> elements that point to video pages:

Since we are really interested in the link itself and not in the <a> element that contains it, we can improve the above with a list comprehension:

And now we have a list of all the links to the individual pages for each session!

The following script shows a cleaned up version of all the techniques we have learned so far:

If you run the above script you will get a long list of URLs as a result. Now we need to parse each of these to get more information about each PyCon session.

Scraping Linked Pages

The next step is to load each of the pages in our URL list. If you want to see how these pages look, here is an example: http://pyvideo.org/video/2668/writing-restful-web-services-with-flask. Yes, that's me, that is one of my sessions!

From these pages we can scrape the session title, which appears at the top. We can also obtain the names of the speakers and the YouTube link from the sidebar that appears on the right side below the embedded video. The code that gets these elements is shown below:

A few things to note about this function:

• The URLs returned from the scraping of the index page are relative, so the root_url needs to be prepended.
• The session title is obtained from the <h3> element inside the <div> with id videobox. Note that [0] is needed because the select() call returns a list, even if there is only one match.
• The speaker names and YouTube links are obtained in a similar way to the links in the index page.

Flask Web Scraping

Now all that remains is to scrape the views count from the YouTube page for each video. This is actually very simple to write as a continuation of the above function. In fact, it is so simple that while we are at it, we can also scrape the likes and dislikes counts:

The soup.select() calls above capture the stats for the video using selectors for the specific id names used in the YouTube page. But the text of the elements need to be processed a bit before it can be converted to a number. Consider an example views count, which YouTube would show as '1,344 views'. To remove the text after the number the contents are split at whitespace and only the first part is used. This first part is then filtered with a regular expression that removes any characters that are not digits, since the numbers can have commas in them. The resulting string is finally converted to an integer and stored.

To complete the scraping the following function invokes all the previously shown code: Hp universal print driver for mac os.

Parallel Processing

The script up to this point works great, but with over a hundred videos it can take a while to run. In reality we aren't doing so much work, what takes most of the time is to download all those pages, and during that time the script is blocked. It would be much more efficient if the script could run several of these download operations simultaneously, right?

Back when I wrote the scraping article using Node.js the parallelism came for free with the asynchronous nature of JavaScript. With Python this can be done as well, but it needs to be specified explicitly. For this example I'm going to start a pool of eight worker processes that can work concurrently. This is surprisingly simple:

The multiprocessing.Pool class starts eight worker processes that wait to be given jobs to run. Why eight? It's twice the number of cores I have on my computer. While experimenting with different sizes for the pool I've found this to be the sweet spot. Less than eight make the script run slower, more than eight do not make it go faster.

The pool.map() call is similar to the regular map() call in that it invokes the function given as the first argument once for each of the elements in the iterable given as the second argument. The big difference is that it sends all these to run by the processes owned by the pool, so in this example eight tasks will run concurrently.

Web Scraping Flasks

The time savings are considerable. On my computer the first version of the script completes in 75 seconds, while the pool version does the same work in 16 seconds!

The Complete Scraping Script

The final version of my scraping script does a few more things after the data has been obtained.

I've added a --sort command line option to specify a sorting criteria, which can be by views, likes or dislikes. The script will sort the list of results in descending order by the specified field. Another option, --max takes a number of results to show, in case you just want to see a few entries from the top. Finally, I have added a --csv option which prints the data in CSV format instead of table aligned, to make it easy to export the data to a spreadsheet.

The complete script is available for download at this location: https://gist.github.com/miguelgrinberg/5f52ceb565264b1e969a.

Web Scraping Flask And Holder

Below is an example output with the 25 most viewed sessions at the time I'm writing this:

Conclusion

I hope you have found this article useful as an introduction to web scraping with Python. I have been pleasantly surprised with the use of Python, the tools are robust and powerful, and the fact that the asynchronous optimizations can be left for the end is great compared to JavaScript, where there is no way to avoid working asynchronously from the start.

Miguel

Hello, and thank you for visiting my blog! If you enjoyed this article, please consider supporting my work on this blog on Patreon!

Web Scraping Facebook Comments

60 comments

Web Scraping Flask Download

• #51Rob Lineberger said 2015-10-08T20:12:25Z

• #52Tushar said 2015-11-30T12:42:51Z

• #53Mani Teja Varma said 2016-01-12T07:25:11Z

• Epson drivers for mac. #54Miguel Grinberg said 2016-01-14T15:37:26Z

• #55Julio Guzman said 2016-04-12T05:19:59Z

• #56Raj said 2016-05-02T10:14:21Z

• #57Miguel Grinberg said 2016-05-04T19:57:20Z

• #58pvlbzn said 2016-06-19T13:06:15Z

• #59Claire said 2017-01-30T03:53:23Z

• #60Miguel Grinberg said 2017-01-30T17:58:20Z

Web Scraping Tools

Leave a Comment