Server-side DOM and Parsing in Web2py

Introduction

Web documents have a programming interface called the Document Object Model (DOM). So that software can alter the document's structure, appearance, and content, it serves as a representation of the page. Programming languages can communicate with a page by interacting with the nodes and objects that the DOM uses to represent the document.

A web page is a document that can be used as the HTML source or displayed in the browser window. Although it is the same document in both instances, it can be altered by using the Document Object Model (DOM) representation. It is a representation of the web page that is object-oriented and can be changed using a scripting language like Python.

Server-side DOM

Elements

The search methods element and elements are provided by the DIV helper, and all helpers descend from it.

element returns the initial child element that meets the given criteria (or None if no match).

elements provide a list of all children that match.

The matching condition for element and elements is specified using the same syntax, allowing for three possible outcomes that can be combined and matched: Expressions that resemble jQuery, match by precise attribute value, and match using regular expressions.

Below is a simple example of it:
>>> a = DIV(DIV(DIV('a', _id='target', _class='abc')))
>>> x = a.elements('div#target')
>>> x[0][0] = 'changed'
>>> print a
<div><div><div id="target" class="abc">changed</div></div></div>

You can also try this code with Online Python Compiler

Run Code

A string that can contain the name of a tag, the class preceded by a dot, the id of a tag preceded by a pound sign, or the explicit value of an attribute in square brackets is the un-named argument of elements.

The previous tag can be searched in 4 equivalent ways using the id:

x = a.elements('#target')
x = a.elements('div#target')
x = a.elements('div[id=target]')
x = a.elements('div', _id='target')

You can also try this code with Online Python Compiler

Run Code

The previous tag can be searched by class in 4 equivalent methods, as follows:

x = a.elements('.abc')
x = a.elements('div.abc')
x = a.elements('div[class=abc]')
x = a.elements('div', _class='abc')

You can also try this code with Online Python Compiler

Run Code

The function element can take multiple named parameters, so any attribute can be used to find an element (other than id and class), but only the first matched element will be returned.

Multiple search criteria can be specified using the jQuery syntax "div#target," separated by commas:

a = DIV(SPAN('a', _id='t1'), DIV('b', _class='c2'))
x = a.elements('span#t1, div.c2')
We can also write it as:
a = DIV(SPAN('a', _id='t1'), DIV('b', _class='c2'))
x = a.elements('span#t1', 'div.c2')

You can also try this code with Online Python Compiler

Run Code

When a name argument is used to specify an attribute's value, either a regular expression or a string may be used:

a = DIV(SPAN('a', _id='test123'), DIV('b', _class='c2'))
x = a.elements('span', _id=re.compile('test\d{3}')

You can also try this code with Online Python Compiler

Run Code

The DIV (and derived) helpers have a unique named argument that is find. In the tag's text content, it can be used to define a search value or a search regular expression. For instance:

>>> a = DIV(SPAN('abcde'), DIV('fghij'))
>>> x = a.elements(find='bcd')
>>> print x[0]
<span>abcde</span>

You can also try this code with Online Python Compiler

Run Code

or

>>> a = DIV(SPAN('abcde'), DIV('fghij'))
>>> x = a.elements(find=re.compile('fg\w{3}'))
>>> print x[0]
<div>fghij</div>

You can also try this code with Online Python Compiler

Run Code

Components

Here is an illustration of how to list every element in an HTML string:

>>> html = TAG('<a>xxx</a><b>yyy</b>')
>>> for element in html.components:
...    print element
... 
<a>xxx</a>
<b>yyy</b>

You can also try this code with Online Python Compiler

Run Code

Parent and Siblings

The parent shows the current element's parent.

>>> a = DIV(SPAN('a'), DIV('b'))
>>> s = a.element('span')
>>> x = s.parent
>>> x['_class']='abc'
>>> print a
<div class="abc"><span>a</span><div>b</div></div>
>>> for element in s.siblings(): print element
<div>b</div>

You can also try this code with Online Python Compiler

Run Code

Replacing Elements

By using the replace option, you can additionally change or remove elements that match. Keep in mind that the standard return value still contains a list of the first matched components.

>>> a = DIV(SPAN('x'), DIV(SPAN('y'))
>>> b = a.elements('span', replace=P('z')
>>> print a
<div><p>z</p><div><p>z</p></div>

You can also try this code with Online Python Compiler

Run Code

The callable replace may exist. In this scenario, the original element will be supplied to it, and it is anticipated to return the substitute element:

>>> a = DIV(SPAN('x'), DIV(SPAN('y'))
>>> b = a.elements('span', replace=lambda t: P(t[0])
>>> print a
<div><p>x</p><div><p>y</p></div>

You can also try this code with Online Python Compiler

Run Code

If replace attribute is set to None (replace=None), matching elements will be removed completely.

>>> a = DIV(SPAN('x'), DIV(SPAN('y'))
>>> b = a.elements('span', replace=None)
>>> print a
<div></div>

You can also try this code with Online Python Compiler

Run Code

Flatten

Recursively serializing the content of an element's children into plain text (without tags) is what the flatten method does:

>>> a = DIV(SPAN('this', DIV('is', B('a'))), SPAN('test'))
>>> print a.flatten()
thisisatest

You can also try this code with Online Python Compiler

Run Code

An optional argument, render, which is a function that flattens the content using a different protocol, can be supplied to flatten. As an illustration, consider the following Markmin wiki syntax:

>>> a = DIV(H1('title'), P('example of a ', A('link', _href='#test')))
>>> from gluon.html import markmin_serializer
>>> print a.flatten(render=markmin_serializer)
# titles

example of [[a link #test]]

You can also try this code with Online Python Compiler

Run Code

web2py currently offers a markdown serializer and a markmin serializer.