Innaz Review, Samsung Galaxy, price and specifications, Flash Android, Games For Your Apple, Jailbreak Tool For iOS, IMEI Unlock Method

Kamis, 31 Desember 2015

HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python)

HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python) - In the past, when I started to start blogging, many thoughts disturbed me. I want to have a blog with a nice and interesting look. I am constantly looking for basic tutorials from some web and blogs on the internet. And thankfully, one by one I started to do it, and of course have to go through some confusion process first, but the most important of a blog that is content, yes on the blog Innaz Review we will discuss a lot of information about gadgets that are very in need by you, now we will discuss first about HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python) please refer to the information we will convey until completion:

Articles : HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python)
full Link : HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python)

You can also see our article on:


HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python)

Websites are made of web pages. Every webpage is made of html tags. And every webpage has a unique address called URL which stands for Uniform Resource Locator. Using which we can request the webpage html code from the server. Once we send the request, the server will give us the html file for the webpage and our browser translates the code into readable English. This is the process that takes place every time we browse the internet.

In today’s post, we will see how to do that without the browser and rather through python. And also we will see how to grab all the links present on a website. A link is another URL. There can be many uses of doing this. For example, a web crawler does this all the time. It goes through the html code of a website and copies the links. And it visits those individual links and looks for other links in that page too. This is how the Google indexes pages. It crawls the whole of the internet every day.
Python is very easy, as is this process. So let’s get started with the program. For this program we will need two modules:
  • Urllib2
  • Beautifulsoup4

Urllib2 is used for sending GET requests to the server where the website resides.  This will allow us to locally save the HTML code of the webpage on our own machine.

BeautifulSoup4 is used for finding specific elements present in the HTML code. It allows us to isolate any specific tag like 'a' or 'div' or any other tag. It also lets us play around with its contents.

Steps:

  1. First of all we are going to import both libraries into our program.
  2. Then we will get the HTML code of the desired webpage using urllib2.urlopen() function.
  3. Then we are going to parse the HTML code using BeautifulSoup() function.
  4. Once we have the beautifulsoup object ready, it is very easy to fetch all the links. All the links are written inside the anchor tag .
  5. To get all the anchor tags, we use the function find_all(‘a’) which comes with beautifulsoup. This will give us a list of all the anchor tags.
  6. But we don’t want the whole anchor tag. We only need the URLs. For  that we can use the function get(‘href’) which also is a function of beautifulsoup. It will fetch all the URLs from their respective tags.
  7. We make a list of all such URLs on the page and we are done.

This will only crawl the page specified. If you want to crawl the whole website, the process should be repeated for each link in the list you have made. That will take a lot of time.

Also not all the links in the anchor tags will be complete links. Some of them will not have the domain name mentioned in them. For example, instead of   http://innaz2.blogspot.com /2015/12/checking-for-email-address-validity.html">http://innaz2.blogspot.com/2015/12/checking-for-email-address-validity.html, only http://innaz2.blogspot.com /2015/12/checking-for-email-address-validity.html">/2015/12/checking-for-email-address-validity.html will be given. So to access them we will have to prefix them with the domain name of the website starting with http://.

All theses links can be copied to a file for future use of can be used directly used in this program itself.

This is the code for the program:


This is what the output would look like:


Pretty easy isn’t it?

That’s all for today. If you have any questions, please ask in the comments below or make use of the ‘contacts us’ button.

Don’t forget to subscribe to the newsletter to receive future such posts right to your inbox.

Thank you.



so much information HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python)

hopefully HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python) information can provide benefits for you in determining the gadgets that fit your needs in daily life.

you just read HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python) if you feel this information is useful and want to bookmark or share it please use link https://innaz2.blogspot.com/2015/12/how-to-extract-all-links-of-given.html if you want more information please search on other pages this blog.

Tag :
Share on Facebook
Share on Twitter
Share on Google+
Tags :

Related : HOW TO EXTRACT ALL THE LINKS of A GIVEN WEBPAGE (with python)

0 komentar:

Posting Komentar