按以下正常情况下启动的playwright,仅把无头模式设置为False,便于观察结果。
antcpt.com 竟然都没有判断出来,apivoid.com 评分为55,高概率为机器人。其余均认定为机器人。
所以antcpt直接排除测试集。
import asynciofrom playwright.async_api import async_playwright
URLS = [ "https://deviceandbrowserinfo.com/are_you_a_bot", "https://www.browserscan.net/bot-detection", "https://pixelscan.net/bot-check", "https://antcpt.com/eng/information/demo-form/recaptcha-3-test-score.html", "https://www.apivoid.com/tools/bot-detection-test/", ]async def main(): async with async_playwright() as p: browser = await p.chromium.launch( headless=False ) context = await browser.new_context() pages = await asyncio.gather( *[context.new_page() for _ in URLS] ) # 同时导航到4个url await asyncio.gather( *[ page.goto(url, wait_until="domcontentloaded", timeout=30000) for page, url in zip(pages, URLS) ] ) await pages[0].wait_for_timeout(60000) await pages[0].pause() await browser.close()
if __name__ == "__main__": asyncio.run(main())接下来添加一些启动参数,browserscan判定为正常了,apivoid评分只有5了,也正常了。
deviceandbrowserinfo和pixelscan依然认定为机器人。
其实最关键的只有 —disable-blink-features=AutomationControlled
—disable-blink-features=AutomationControlled 是 Chrome 123 之后替代旧 excludeSwitches 方案的入门级反检测参数。它能阻止 navigator.webdriver = true
browser = await p.chromium.launch( headless=False, args=[ "--disable-blink-features=AutomationControlled", # 禁用自动化特征 "--no-sandbox", # 提升兼容性 "--disable-dev-shm-usage", # 避免共享内存问题 ], ignore_default_args=[ "--enable-automation", # 移除默认自动化参数 '--disable-extensions', '--disable-default-apps', '--disable-component-extensions-with-background-pages' ], )playwright + https://pypi.org/project/playwright-stealth/
均被检测为机器人。apivoid为30分。看起来不如添加一点启动参数好使。
# https://pypi.org/project/playwright-stealth/from playwright_stealth import Stealth
async def main(): # async with async_playwright() as p: async with Stealth().use_async(async_playwright()) as p: browser = await p.chromium.launch( headless=False, )使用stealth.js: https://github.com/berstend/puppeteer-extra/tree/master/packages/extract-stealth-evasions
browserscan通过,apivoid评分为30,被检测到了**Browser tamper detected。**
deviceandbrowserinfo和pixelscan未通过。
async def main(): stealthjs = "stealth.min.js" async with async_playwright() as p: browser = await p.chromium.launch( headless=False, ) context = await browser.new_context() await context.add_init_script(path=stealthjs)playwright + https://pypi.org/project/undetected-playwright/
apivoid.com 评分为55,navigator.webdriver is true、Browser tamper detected
其余也都被认定为机器人。
# uv add undetected-playwright# https://pypi.org/project/undetected-playwright/from undetected_playwright import Malenia
async def main(): async with async_playwright() as p: browser = await p.chromium.launch( headless=False, ) context = await browser.new_context() await Malenia.apply_stealth(context)使用cloakbrowser,结果通过全部检测。
还有就是我正常使用了vpn,但是cloakbrowser也把这个屏蔽了,没有检测到。
import asynciofrom cloakbrowser import launch_async,launch_context_async
URLS = [ # "https://abrahamjuliot.github.io/creepjs/", 必须 wait_until="commit" "https://deviceandbrowserinfo.com/are_you_a_bot", "https://www.browserscan.net/bot-detection", "https://www.apivoid.com/tools/bot-detection-test/", "https://pixelscan.net/bot-check", "https://bot.sannysoft.com/" ]async def main(): browser = await launch_async( # proxy="http://user:pass@residential-proxy:port", # residential IP, not datacenter # geoip=True, # match timezone + locale to proxy IP headless=False, # some sites detect headless even with C++ patches humanize=True, # human-like mouse, keyboard, scroll ) context = await launch_context_async( # user_agent="Custom UA", # viewport={"width": 1920, "height": 1080}, # locale="en-US", # timezone="America/New_York", ) context = await browser.new_context() pages = await asyncio.gather( *[context.new_page() for _ in URLS] ) # 同时导航到4个url await asyncio.gather( *[ page.goto(url, wait_until="domcontentloaded", timeout=60000) for page, url in zip(pages, URLS) ] ) await pages[0].wait_for_timeout(60000) await pages[0].pause() await browser.close()
if __name__ == "__main__": asyncio.run(main())使用drissionpage: https://drissionpage.cn/versions/statement
也能通过全部的检测。
对了,值得一提的是,DrissionPage 有一个副作用。它竟然能够通过 cloudflare、Google 等人机检测工具,这个是作者都没有想到的。也许是自己写的小众玩意,这些大厂还不认识吧。
看起来作者也不知道。
根据文档描述的特性:可处理非open状态的 shadow-root
应该也可以处理cloudflarede 的#shadow-root (closed) 情况,那确实很适合爬虫了。
from DrissionPage import Chromium, ChromiumOptions
URLS = [ "https://deviceandbrowserinfo.com/are_you_a_bot", "https://www.browserscan.net/bot-detection", "https://pixelscan.net/bot-check", "https://antcpt.com/eng/information/demo-form/recaptcha-3-test-score.html", "https://www.apivoid.com/tools/bot-detection-test/",]
def main(): # 启动浏览器(headless=False 表示有界面模式) # DrissionPage 默认就是有界面模式 browser = Chromium() # 获取标签页管理器 tab = browser.latest_tab # 创建多个标签页 tabs = [] for i, url in enumerate(URLS): if i == 0: # 第一个URL使用现有标签页 tabs.append(tab) else: # 创建新标签页 new_tab = browser.new_tab() tabs.append(new_tab)
# DrissionPage 主要设计为同步 for tab_obj, url in zip(tabs, URLS): tab_obj.get(url)
# 暂停(保持浏览器打开,等待用户操作) print("浏览器保持打开状态,按回车键关闭...") input() # 关闭浏览器 browser.quit()
if __name__ == "__main__": main()camoufox运行异常。可能是不兼容的问题。
依赖于scrapling和playwright,若已安装这两个,camoufox可能不会安装最新版。
import asynciofrom camoufox.async_api import AsyncCamoufox
URLS = [ "https://deviceandbrowserinfo.com/are_you_a_bot", "https://www.browserscan.net/bot-detection", "https://www.apivoid.com/tools/bot-detection-test/", "https://pixelscan.net/bot-check", "https://bot.sannysoft.com/" ]# https://v4.gh-proxy.org/https://github.com/daijro/camoufox/releases/download/v152.0.4-beta.28/camoufox-152.0.4-beta.28-mac.arm64.zip# bash ./scripts/install-local-build.sh /path/to/camoufox-152.0.4-beta.28-mac.arm64.zip# bash ./CloudFlare5s/install-local-build.sh /Users/tiam/Downloads/camoufox-152.0.4-beta.28-mac.arm64.zip# xattr -cr ~/Library/Caches/camoufox/browsers/local/152.0.4-beta.28/Camoufox.app## 运行异常:# playwright._impl._errors.Error: Page.goto: NS_ERROR_FAILURE# 这是最常见的原因。在 Ubuntu/Debian 等 Linux 环境下,Firefox headless 模式可能因为缺少 X Server、GPU 驱动或某些系统库而出现 NS_ERROR_FAILURE2。日志中常见的关键行:async def main(): async with AsyncCamoufox(headless=False) as browser: pages = await asyncio.gather( *[browser.new_page() for _ in URLS] ) # 同时导航到4个url await asyncio.gather( *[ page.goto(url) for page, url in zip(pages, URLS) ] ) await pages[0].wait_for_timeout(60000) await pages[0].pause() await browser.close()
if __name__ == "__main__": asyncio.run(main())其他
https://github.com/botswin/BotBrowser 应该类似比特指纹浏览器的产品
https://github.com/CloakHQ/CloakBrowser 隐形 Chromium,通过所有机器人检测测试。可直接替换 Playwright,并带有源代码级指纹补丁。
https://github.com/daijro/camoufox 修补Firefox的反检测浏览器
主要是通过自然语言控制浏览器,SDK形式,主要是写进程序中。
https://github.com/browserbase/stagehand 增强版 Playwright
https://github.com/browser-use/browser-use
https://github.com/lightpanda-io/browser zig写的,主打性能,全新的浏览器引擎,代替chrome, 有Agent 模式
https://github.com/h4ckf0r0day/obscura rust写的,也主打性能,全新的浏览器引擎,代替chrome,反检测/隐身更强
https://github.com/vercel-labs/agent-browser 也是rust实现的,主要是cli命令行控制浏览器,是浏览器之上的自动化层,它驱动 Chrome(也支持 Lightpanda 作为引擎)