python使用beautifulsoup包编写爬虫时的一些关键点

1.善于利用soup节点的parent属性

比如对于已经得到了如下html代码:

november

2012

的soup变量eachmonthheader了。

想要提取其中的

month的label的值:november

和year的label的值:2012

最简单,也是最省事的办法是,直接搜两个label,然后肯定会找到这两个label,然后分别对应着month和year的label,然后获得对应的string即可:

foundtwolabel = eachmonthheader.findall(“label”);
print “foundtwolabel=”,foundtwolabel;
monthlabel = foundtwolabel[0];
yearlabel = foundtwolabel[1];
monthstr = monthlabel.string;
yearstr = yearlabel.string;
print “monthstr=”,monthstr; # monthstr= november
print “yearstr=”,yearstr; # yearstr= 2012

但是很明显,这样的逻辑性很不好,而且万一处理多个这样的soup变量,而且两者的顺便颠倒了,那么结果也就错误了。

此时,可以考虑利用soup变量的parent属性,从一个soup变量本身,获得其上一级的soup变量。
示例代码如下:

# november
#
#
# 2012
#
#
foundcbomonth = eachmonthheader.find(“input”, {“id”:re.compile(“cbomonth\d+”)});
#print “foundcbomonth=”,foundcbomonth;
tdmonth = foundcbomonth.parent;
#print “tdmonth=”,tdmonth;
tdmonthlabel = tdmonth.label;
#print “tdmonthlabel=”,tdmonthlabel;
monthstr = tdmonthlabel.string;
print “monthstr=”,monthstr;
foundcboyear = eachmonthheader.find(“input”, {“id”:re.compile(“cboyear\d+”)});
#print “foundcboyear=”,foundcboyear;
tdyear = foundcboyear.parent;
#print “tdyear=”,tdyear;
tdyearlabel = tdyear.label;
#print “tdyearlabel=”,tdyearlabel;
yearstr = tdyearlabel.string;
print “yearstr=”,yearstr;

我们再来看一个例子:

from beautifulsoup import beautifulsoup
doc = [‘page title’,

this is paragraph one.’,

this is paragraph two.’,
”]
soup = beautifulsoup(”.join(doc))
print soup.prettify()
#
#
#
# page title
#
#
#
#

# this is paragraph
#
# one
#
# .
#

#

# this is paragraph
#
# two
#
# .
#

#
#

这个例子中, tag的parent是 tag. tag 的parent是beautifulsoup 剖析对象自己。 剖析对象的parent是none. 利用parent,你可以向前遍历剖析树。

soup.head.parent.name
# u’html’
soup.head.parent.parent.__class__.__name__
# ‘beautifulsoup’
soup.parent == none
# true

2.当解析非utf-8或ascii编码类型的html时,需要指定对应的字符编码

当html为ascii或utf-8编码时,可以不指定html字符编码,便可正确解析html为对应的soup:

#这里resphtml是ascii或utf-8编码,此时可以不指定编码类型,即可正确解析出对应的soup
soup = beautifulsoup(resphtml);

当html为其他类型编码,比如gb2312的话,则需要指定相应的字符编码,beautifulsoup才能正确解析出对应的soup:

比如:

#此处resphtml是gb2312编码的,所以要指定该编码类型,beautifulsoup才能解析出对应的soup
htmlcharset = “gb2312”;
soup = beautifulsoup(resphtml, fromencoding=htmlcharset);

Posted in 未分类