BeautifulSoup4

Like lxml, BeautifulSoup4 is an HTML/XML parser primarily used for parsing and extracting data from HTML/XML documents.

Installation: pip install bs4

Since BS4 needs a document parser to work with pages, you also need to install lxml as the parsing library.

Parsing Principle

- Data parsing principle:
    1. Locate the tag
    2. Extract the tag and the data stored in tag attributes

- bs4 data parsing principle:
    1. Instantiate a BeautifulSoup object and load the page source code into it
    2. Use methods or properties of the BeautifulSoup object to locate tags and extract data

Usage Workflow

1. Import module:
    from bs4 import BeautifulSoup

2. Instantiate object:
    - Load a local HTML file into the object:
        fp = open('sample.html','r',encoding='utf-8')
        soup = BeautifulSoup(fp,'lxml')
        fp.close()

    - Load page source obtained from the internet into the object:
        content = response.text
        soup = BeautifulSoup(content,'lxml')

3. Write bs4 expressions to extract data

Parsing Methods and Properteis

3. Methods and properties for data parsing:
    - soup.tagname : returns the first occurrence of the given tag in the document, returns a `bs4.element.Tag` object
    - soup.find('') :
        (1) soup.find('tagname') = soup.tagname
        (2) Attribute location: soup.find('tagname', class_/id='  ')
                            soup.find('tagname', attrs={'class':''})
           Returns all content under that attribute
        (3) soup.find_all(''): returns all tags that meet the criteria (returns a list)
    - soup.select():
        (1) soup.select('some selector (id, class, tag).book-mulu')   returns a list
            Example: soup.select('.book-mulu')  -- find by class name
                     soup.select('a')          -- find by tag name
                     soup.select("#link1")     -- find by id
                     soup.select('[href]')     -- find by attribute
                     soup.select('a[href="https://www.baidu.com/"]')  -- find by attribute
                     soup.select("p #link1")  -- combined find
                     soup.select("head > title")  -- combined find
        (2) Hierarchical selector: > means one level, space means multiple levels
            Example: soup.select('.book-mulu > ul > li > a')
                     soup.select('.book-mulu > ul  a')[0]

- Getting text data between tags:
    - soup.a.text/get_text()  gets all text content within the tag
    - soup.string  gets a single text content from the immediate child, returns a string
    - soup.strings: gets all non-tag descendant strings of a tag, returns a generator
    - soup.stripped_strings: gets all non-tag descendant strings, removes whitespace characters, returns a generator
- Getting attribute values inside a tag:
    - soup.a['']   Example: soup.a['href']