Best Practices for Advanced Python Web Scraping

 

Scraping is an easy concept in its crux, however, it's also a tricky one! It's like the cat-and-mouse game between a website owner as well as a developer working in the legal area. This blog throws light on a few obstructions that a programmer might face while doing web scraping, as well as different ways of getting around.

What is Web Scraping?

Web scraping services are the work of extracting data from different websites and other online sources. This could either be a manual procedure or an automated process. Although manually scraping data from web pages could be a redundant and tedious procedure that justifies the whole ecosystem of different libraries and tools built to automate data scraping procedure. In auto web scraping services, rather than letting a browser reduce pages, we utilize self-written scripts for parsing raw responses from a server. In this blog post, we will utilize "Web Scraper" for implying "Automated Web Scraping."

How to Do Web Scraping?

Before moving to things, which can make web scraping complicated, let's break the procedure of data scraping into comprehensive steps:

• Visual inspection: Finding what to scrape
• Making an HTTP request for a webpage
• Parsing HTTP responses
• Use relevant data

The initial step involves the use of in-built browser tools (including Chrome DevTools as well as Firefox Developer Tools) for finding the information we want on a webpage as well as identifying structures or patterns to scrape it programmatically.

Following are the steps that involve systematically making requests for a webpage as well as implementing the logic of scraping data, using patterns that we have identified. Finally, we utilize the data for whatever objective we planned to.

For instance, let's say that we wish to scrape the total subscribers of PewDiePie as well as compare that with T-Series. An easy Google search results in a YouTube Subscriber Count Page of Socialblade.

Difficulties of Web Scraping

• Analyzing the Request Rate
• Asynchronous Loading as well as Client-Side Rendering
• Authentication
• Captchas and Redirects
• Choosing the Right Libraries, Frameworks, and Tools
• Header Inspection
• Honeypots
• Pattern Detection
• Resolving Complexities of Python and Web Scraping

For data scraping in Python, many tools are available. We'll use some popular options as well as when to utilize which. For extracting easy websites rapidly, we've found a grouping of Python Requests (for handling sessions as well as making HTTP requests) as well as BeautifulSoup (to parse the response as well as navigate through that to scrape data) to make a perfect pair.

For big-size web scraping projects (where we need to collect as well as process ample data as well as cope with non-JS-related difficulties), Scrapy has been extremely useful.

Scrapy is the framework that extracts many intricacies to scrape efficiently (memory utilization, concurrent requests, etc.), and permits to plug in a group of middleware (for redirects, sessions, cookies, caching, etc.) to cope with various complexities. Scrapy gives a shell also, which can assist in rapid prototyping and authenticating your data extraction approach (responses, selectors, etc.). The framework is quite extensible, mature, and has a very good support community too.

For heavy JavaScript websites (or websites that look too complex), Selenium is generally a way to go. Though data scraping with Selenium isn't as effective as compared to Beautiful Soup or Scrapy, it always offers you the required data (that is a thing, which matters the most).

Authentication Handling

For authentication, we'll need to preserve cookies as well as persist login, the better option is to make a session that deals with all. For unseen fields, we could manually try and log in as well as inspect the payloads getting sent to a server with network tools given by a browser to recognize the hidden data getting sent.

We could also review what headers are getting sent to the server with browser tools so that we could replicate the behavior in code also like if authentication relies on headers like Authentication and Authorization). If a website uses an easy cookie-based authentication (that is highly doubtful these days), we could also copy cookie contents as well as add that to a scraper's code (we could use in-built browser tools to do that).

Handling of Asynchronous Loading

Detect Asynchronous Loading

We can find asynchronous loading during a visual inspection by viewing a page source (a "View Source" alternative in a browser by doing a right-click) as well as searching for the content we're searching for. In case, you don't get text in a source, and yet you can still see that in a browser, then probably it's getting rendered using JavaScript. An additional inspection could be done using the browser's network tools to review if all XHR requests are getting made by a website.

Bypassing Asynchronous Loading

Use a Web Driver

Using a web driver is like a simulation of the web browser with interfaces to get controlled using scripts. This is capable to do browser stuff including rendering JavaScript, organizing sessions and cookies, and more. Selenium Web Driver is the web automation framework specially designed for testing UI/UX of websites, however, it has become a well-known option for scraping dynamically rendered websites over time.

As web drivers are an imitation of browsers, they're source intensive as well as moderately slower compared to libraries including Scrapy and BeautifulSoup.

Inspect AJAX Calls

This technique works on the idea of "In case, it's getting displayed on a browser, this needs to come from someplace." We can utilize browser developer tools for inspecting AJAX calls as well as try and find requests that are accountable to fetch the data we're searching for. We could require to set the X-Requested-With header for mimicking AJAX requests in the script.

Tackle Infinite Scrolling

We can deal with infinite scrolling by injecting a few JavaScript logics in selenium. Also, generally, the infinite scrolling comprises more AJAX calls to a server that we could inspect using different browser tools as well as replicate it in the scraping program.

Getting the Right Selectors

When we locate an element, which we wish to scrape visually, the following step is to discover a selector’s pattern for such elements, which we can utilize to scrape them from HTML. We could filter different elements depending on their attributes and CSS classes with CSS selectors.

CSS selectors are the first choice to do the scraping. Although another method to select elements named XPath (the query language to select nodes in the XML documents) could be helpful in some scenarios. This gives more flexible capabilities.

Handling Captchas and Redirects

Contemporary libraries like requests take care of the HTTP redirects through following them (maintaining the history) as well as returning a final page. Scrapy comes with the redirect middleware for handling redirects. These redirects aren't given much trouble if we redirect to the pages we seek. However, if we get redirected to any captcha, then problems may come.

Easy text-based captchas could be solved with OCR. Text-based captchas are smooth slopes for implementing these days using the arrival of advanced OCR methods (that depends on Deep Learning), therefore it's getting tougher to make images, which could beat machines as well as humans.

Handling Unstructured Responses and iframe Tags

For different iframe tags, you can request the right URLs to get data back. We need to request an outer page, and then get the iframe, as well as then make one more HTTP request to an iframe’s SRC attribute. Moreover, there's nothing much we could do about formless HTML or URL patterns moreover having come up with the hacks (coming with multipart XPath queries with regexes, etc.).

Conclusion

Web Scraping is like a cat-&-mouse game working in the legal gray-shade area, as well as may cause problems to both sides if haven’t been done carefully. Information abuse and copyright violations may result in legal consequences. Some examples, which have sparked controversies include OK Cupid data released by researchers as well as HIQ labs utilizing LinkedIn information for HR products.

If you want to know more about the best practices of Advanced Python Web Scraping then contact 3i Data Scraping or ask for a free quote!