550 字
3 分钟
python 标准库应用

轮询#

demo: 对UA池无限轮询

import time
from itertools import cycle
ua_pool = ["ua1", "ua2", "ua3"]
polling = cycle(ua_pool)
while True:
# 轮询UA
ua = next(polling)
# 使用UA
print(ua)
time.sleep(1)

输出:

ua1
ua2
ua3
ua1
ua2

多层嵌套循环#

99乘法表

from itertools import product
nums = range(1, 10)
for i, j in product(nums, repeat=2):
print(f"{i} * {j} = {i * j}")

但是只能是确定的多个循环, 不能是动态获取的数组;

可用于确定的多层级目录, 比如 学科.年级

数组切片(分组)#

如果有一个超大页的数据(10000页), 就可以使用batched函数切片为多个数组, 多线程同时处理

from itertools import batched
nums = range(1, 10)
batched_nums = batched(nums, 3)
print(list(nums))
for batch in batched_nums:
print(list(batch))
[1, 2, 3, 4, 5, 6, 7, 8, 9]
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
进程已结束,退出代码为 0

数组筛选#

从数组中筛选一些合法的数据

from itertools import dropwhile, filterfalse
arr = [1, 7, 6, 2, 3, 1, 9, 9, 5, 2, 8]
arr = dropwhile(lambda x: x < 5, arr)
print(list(arr))
arr = [1, 7, 6, 2, 3, 1, 9, 9, 5, 2, 8]
arr = filterfalse(lambda x: x < 5, arr)
print(list(arr))
[7, 6, 2, 3, 1, 9, 9, 5, 2, 8]
[7, 6, 9, 9, 5, 8]
进程已结束,退出代码为 0

更多

文档地址: itertools --- 为高效循环创建迭代器的函数 — Python 3.13.0 文档

更多可迭代对象工具: More Itertools — more-itertools 10.5.0 documentation

Terminal window
python -m pip install more-itertools

统计计数#

有经常需要统计爬虫程序的失败次数和成功次数, 或者分类统计各种类型的数据

from collections import Counter
# 不初始化默认值时, 默认值为1;
# 未初始化的键值, 也可直接计数
counter = Counter({
"success": 0,
"fail": 0
})
counter["success"] += 10
counter["fail"] += 1
print(counter.most_common())
[('success', 10), ('fail', 1)]
进程已结束,退出代码为 0

命名元组#

通常一些目录使用难以理解的id值作为请求, 而为了理解需要对应一个名称

import time
from collections import namedtuple
from itertools import product
Catalog = namedtuple("Catalog", ["id", "name"])
course_catalog = [
Catalog(1, "Python"),
Catalog(2, "Java"),
Catalog(3, "C++"),
]
grade_catalog = [
Catalog(1, "初级"),
Catalog(2, "中级"),
Catalog(3, "高级"),
]
for course, grade in product(course_catalog, grade_catalog):
print(f"爬取课程:{course.name},难度:{grade.name}")
print(f"调用课程id: {course.id}, 难度id: {grade.id} 爬取中...")
time.sleep(0.5)
print("爬取完成")
python 标准库应用
https://blog.tiamix.cn/posts/python-标准库应用/
作者
Tiam
发布于
2024-11-20
许可协议
CC BY-NC-SA 4.0