550 字
3 分钟
python 标准库应用
轮询
demo: 对UA池无限轮询
import timefrom itertools import cycle
ua_pool = ["ua1", "ua2", "ua3"]
polling = cycle(ua_pool)while True: # 轮询UA ua = next(polling) # 使用UA print(ua) time.sleep(1)输出:
ua1ua2ua3ua1ua2多层嵌套循环
99乘法表
from itertools import product
nums = range(1, 10)
for i, j in product(nums, repeat=2): print(f"{i} * {j} = {i * j}")但是只能是确定的多个循环, 不能是动态获取的数组;
可用于确定的多层级目录, 比如 学科.年级
数组切片(分组)
如果有一个超大页的数据(10000页), 就可以使用batched函数切片为多个数组, 多线程同时处理
from itertools import batched
nums = range(1, 10)batched_nums = batched(nums, 3)print(list(nums))for batch in batched_nums: print(list(batch))[1, 2, 3, 4, 5, 6, 7, 8, 9][1, 2, 3][4, 5, 6][7, 8, 9]
进程已结束,退出代码为 0数组筛选
从数组中筛选一些合法的数据
from itertools import dropwhile, filterfalse
arr = [1, 7, 6, 2, 3, 1, 9, 9, 5, 2, 8]arr = dropwhile(lambda x: x < 5, arr)print(list(arr))
arr = [1, 7, 6, 2, 3, 1, 9, 9, 5, 2, 8]arr = filterfalse(lambda x: x < 5, arr)print(list(arr))[7, 6, 2, 3, 1, 9, 9, 5, 2, 8][7, 6, 9, 9, 5, 8]
进程已结束,退出代码为 0更多
文档地址: itertools --- 为高效循环创建迭代器的函数 — Python 3.13.0 文档
更多可迭代对象工具: More Itertools — more-itertools 10.5.0 documentation
python -m pip install more-itertools统计计数
有经常需要统计爬虫程序的失败次数和成功次数, 或者分类统计各种类型的数据
from collections import Counter
# 不初始化默认值时, 默认值为1;# 未初始化的键值, 也可直接计数counter = Counter({ "success": 0, "fail": 0})
counter["success"] += 10counter["fail"] += 1
print(counter.most_common())[('success', 10), ('fail', 1)]
进程已结束,退出代码为 0命名元组
通常一些目录使用难以理解的id值作为请求, 而为了理解需要对应一个名称
import timefrom collections import namedtuplefrom itertools import product
Catalog = namedtuple("Catalog", ["id", "name"])
course_catalog = [ Catalog(1, "Python"), Catalog(2, "Java"), Catalog(3, "C++"),]
grade_catalog = [ Catalog(1, "初级"), Catalog(2, "中级"), Catalog(3, "高级"),]
for course, grade in product(course_catalog, grade_catalog): print(f"爬取课程:{course.name},难度:{grade.name}") print(f"调用课程id: {course.id}, 难度id: {grade.id} 爬取中...") time.sleep(0.5) print("爬取完成")