1385 字
7 分钟
作业帮视频抓取踩坑

需求: 采集作业帮搜题后的视频

先抓包, 找到视频的包

搜索相关关键字, 找到这个接口 https://apivip.zuoyebang.com/vipvideoui/innerv2/hls_898e6b180e94f8ac356c381fc7d60aa5.m3u8

后续发现每次进入时, 都会调用这个接口, 然后响应类型为 application/x-mpegURL or vnd.apple.mpegURL

网上搜索相关信息, 找到: Deploying HTTP Live Streaming (apple.com)

发现是一个 apple 提供的一个叫HSL的视频流方案, 不过响应体却是加密的, 不能直接看到响应内容

image-20241114211139110

继续播放视频会有很多 ts 的包, 响应类型为 application/octet-stream

参考: 常见 MIME 类型列表 - HTTP | MDN (mozilla.org)

多半确定这个就是视频地址相关信息, 准备逆向解密响应内容

打开算法助手 hook 常用算法,

image-20241114212840921

搜索响应没找到, 但是搜接口相关信息找到了

image-20241114213150541

查看堆栈信息, 找到下面两个堆栈信息, 就是那种一看类名就是跟视频相关的; LSPHooker栈是 hook框架的, 不用管

(reqable的堆栈跟踪一直用不了, 不知道为什么, )

com.zybang.sdk.player.ui.download.M3U8Download.downloadM3uU8File
com.zybang.sdk.player.ui.download.MPlayInfoRetriever.downAndSave

image-20241114213418644

下面就可以尝试hook这两个堆栈, 看看能不能找到什么信息

长按该行, 可快速添加到hook, 但是有时候没有效果, 不知道为什么

JADX反编译作业帮的apk文件, 搜索刚刚堆栈里的方法, 看方法

image-20241114214908288

接下来就是读代码, 需要懂java, 有java基础

image-20241114215258270

右键方法名, 可以复制为frida片段

image-20241114222115737

延迟5秒, 避免找不到类

function hook() {
Java.perform(function () {
let M3U8Download = Java.use("com.zybang.sdk.player.ui.download.M3U8Download");
M3U8Download["downloadM3uU8File"].implementation = function (z, str, str2, str3, z2, m3U8DownloadCallBack) {
console.log(`M3U8Download.downloadM3uU8File is called: z=${z}, str=${str}, str2=${str2}, str3=${str3}, z2=${z2}, m3U8DownloadCallBack=${m3U8DownloadCallBack}`);
this["downloadM3uU8File"](z, str, str2, str3, z2, m3U8DownloadCallBack);
};
});
}
setTimeout(hook, 5000);

但是但是, frida被检测了, 会自动退出

使用算法助手导入frida脚本也不行, 会一直重进, 只能使用自定义hook中去导入

hook到这两个方法如下

image-20241114225859798

好了, 现在知道了一些没用的信息

videoPlayKey: hls_898e6b180e94f8ac356c381fc7d60aa5
videoUrl: https://apivip.zuoyebang.com/vipvideoui/innerv2/hls_898e6b180e94f8ac356c381fc7d60aa5.m3u8?p=ymNgbuzMAd7IyzQB3IOCcRE52XtfhpN3XfwbStYkzsDd-YtUN-7ORVi3KJIraUj2JYtqKsuJUoi2o-uzQ0PpkkQ1bXjAIfZ5NaTU6HHGqlHF9wvhvUnjNPs8ps8PjCQcb4MeAgtIpFSHCmCyx_-UbtgiWwi3C8n6JHO1QNuhjkarI49oEqdRe9Ltr9xKzaU2YcsX9uEsKUlwnCXwWbWOtMTS0NarDTRQr_jnfvw7UNbbF70xzroh9baq3pjFKWVkSNgq9Z5P8OnEoJylAm3bZ4jo5HLkdXhhzWDpzI_HTu2X2tGrJuZsmhb36yij4mP2CJiJ2w7YMyh5ecwaYInj1gnEV4rHg-FsRYUbO3Tj0E2Jp3ez9X6FtdRKs8Bv2NCd1HLVC7fFKlVED-hh6OEZ-Y8BQKa7N_oCREMgeepUOe6-np2mlCP-JEgytl-Xd-0-viyqtQ%3D%3D&t=1731596061042
文件名: md5(videoUrl)+"temp.m3u8"

这里应该是卡住了

但是但是, 在读 com.zybang.sdk.player.ui.download.M3U8Download类时, 发现了方法中的一些其他可疑的方法, 都可以hook试试

甚至可以hook类中的所有方法, 然后搜索寻找蛛丝马迹

image-20241114231703474

再次搜索响应值, 可以找到这几个

image-20241114232150758

且有如下返回值, 虽然不知道是什么, 但是感觉是有用的, 丢给AI问问相关信息

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:15
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-KEY:METHOD=AES-128,URI="https://apivip.zuoyebang.com/vipsk/sk/get?token=1731597191-2319653858-5f4706dc59d94adbae3a512e89d430581731597191-U2FsdF9aWUJvpy7sR4phOJbRmaewrjRTBgGvI1xvXCPb8OI2KvvAt7WKv24lbBAa-U2FsdF9aWUJU~nvHv~Zob40bVte2A65ghfeZgcIXZoU-b4f30f94e1db42788784680f97a00197"
#EXTINF:14.705556,
https://vipvideo-o.cdnjtzy.com/hls_898e6b180e94f8ac356c381fc7d60aa5%2Fts%2F0a49e7e9f6f21d8ca9d0834962e73424.ts?Expires=1734189191&OSSAccessKeyId=LTAI5tARozQj899V1ddNxHL3&Signature=zFXuS4kfCXqeAn4R%2FZ5Ko44F%2Fbs%3D
#EXTINF:14.705556,
https://vipvideo-o.cdnjtzy.com/hls_898e6b180e94f8ac356c381fc7d60aa5%2Fts%2F201b638fe6b70284e0b56395b18f8492.ts?Expires=1734189191&OSSAccessKeyId=LTAI5tARozQj899V1ddNxHL3&Signature=2KwUAbv6Hwl0SwbNbY%2BidM5qPV4%3D
#EXTINF:14.705556,
https://vipvideo-o.cdnjtzy.com/hls_898e6b180e94f8ac356c381fc7d60aa5%2Fts%2Ff916b507f232322040b322640e64aa7d.ts?Expires=1734189191&OSSAccessKeyId=LTAI5tARozQj899V1ddNxHL3&Signature=W3XMcNPZHr4ldM2OrC6HoaOpbKc%3D
#EXTINF:14.705556,
https://vipvideo-o.cdnjtzy.com/hls_898e6b180e94f8ac356c381fc7d60aa5%2Fts%2Fa0492ad801b2ea8d4f8cd8557c510d9e.ts?Expires=1734189191&OSSAccessKeyId=LTAI5tARozQj899V1ddNxHL3&Signature=xDFEXuvSFQdVk7rjgahIiunwLWA%3D
#EXTINF:14.705556,
https://vipvideo-o.cdnjtzy.com/hls_898e6b180e94f8ac356c381fc7d60aa5%2Fts%2F8ff00708955ecf16caeb1084cb496ed2.ts?Expires=1734189191&OSSAccessKeyId=LTAI5tARozQj899V1ddNxHL3&Signature=PU6qNJdicgJ9m7e4YTokJbRDc3o%3D
#EXTINF:14.705556,
https://vipvideo-o.cdnjtzy.com/hls_898e6b180e94f8ac356c381fc7d60aa5%2Fts%2Febc78d65c618a7a5933673ffe2226271.ts?Expires=1734189191&OSSAccessKeyId=LTAI5tARozQj899V1ddNxHL3&Signature=4Vey2UM%2FuVMjxfGexVdcxFhZg%2Bw%3D
#EXTINF:5.999867,
https://vipvideo-o.cdnjtzy.com/hls_898e6b180e94f8ac356c381fc7d60aa5%2Fts%2F7f6a97412922b398f20a7a868b353e95.ts?Expires=1734189191&OSSAccessKeyId=LTAI5tARozQj899V1ddNxHL3&Signature=CFKuTST3J%2B%2F0jogFul%2F3xeWXh8Q%3D
#EXT-X-ENDLIST

image-20241114232442541

这个应该怎么播放呢? 参考: https://juejin.cn/post/7108257429355233293

写了个web网页

我本地伪造了一个接口, 返回了以上文本

但是网页代码中还是无法播放, 分析网络请求, 发现获取key的接口产生了跨域问题

然后又伪造了一个获取key的接口, 因为他返回的是一个固定的key, 不然就要做代理了

终终于在网页可以播放了, 但是这也太麻烦了

询问AI获得流媒体播放工具: 官方下载:VLC media player,最棒的开源播放器 - VideoLAN

最后可得知, 可直接将文本保存为.m3u8文件, 然后使用FFmpeg转为mp4文件

但是这个.m3u8文件并不能使用 VLC 直接播放, 可能也是由于跨域问题

命令:

Terminal window
ffmpeg -protocol_whitelist file,http,https,crypto,tls,tcp -i "test.m3u8" -c copy output.mp4

python代码如下

# pip install ffmpeg-python
import ffmpeg
def convert_hls_to_mp4(input_file: str, output_file: str):
"""
使用 ffmpeg 将 HLS (.m3u8) 转换为 mp4 格式,允许 'crypto' 协议。
ffmpeg -protocol_whitelist file,http,https,crypto,tls,tcp -i "test.m3u8" -c copy output2.mp4
:param input_file: 输入的 .m3u8 文件路径
:param output_file: 输出的 .mp4 文件路径
"""
try:
ffmpeg.input(input_file, protocol_whitelist='file,http,https,crypto,tls,tcp').output(output_file, c='copy').run()
print(f"Conversion successful: {input_file} -> {output_file}")
except ffmpeg.Error as e:
print("Error occurred while converting:", e.stderr.decode())

image-20241114234254580

确定了这是一段可播放的视频, 下面就开始逆向还原加密逻辑, 如何解密响应值得到这段文本

python 还原逻辑如下

filename = hls + ".m3u8"
reverse = filename[::-1]
substring = md5_string(filename) + md5_string(reverse)
substring = substring[16:48]

得到substring后调用了 decryptString 方法, 分析整体算法如下标注

image-20241115000543921

完整还原如下

import hashlib
# pip install pycryptodome
from Cryptodome.Cipher import ARC4
import base64
def md5_string(string):
md5_hash = hashlib.md5()
md5_hash.update(string.encode('utf-8'))
return md5_hash.hexdigest()
def base64_decrypt(base64_str):
"""
com.zybang.sdk.player.ui.download.M3U8Download.base64decrypt 函数还原
:param base64_str:
:return:
"""
# android.util.Base64.decode(str, 2) # 2 表示 Base64.NO_WRAP:输出不会用换行符分隔,即使输入非常长
base64_str = base64_str.replace('-', '+').replace('_', '/')
# 填充'='以确保长度为4的倍数
padding = len(base64_str) % 4
if padding != 0:
base64_str += '=' * (4 - padding)
# 解码Base64字符串
try:
return base64.b64decode(base64_str)
except Exception:
return base64_str.encode('utf-8')
def decrypt_by_key(hls, encrypted):
"""
com.zybang.sdk.player.ui.download.M3U8Download.decryptByKey 函数还原
:param hls: 以 hls_ 开头的字符串
:param encrypted: 接口返回值
:return:
"""
filename = hls + ".m3u8"
reverse = filename[::-1]
substring = md5_string(filename) + md5_string(reverse)
substring = substring[16:48]
# rc4 秘钥
key = substring.encode('utf-8')
cipher = ARC4.new(key)
encrypted_bytes = base64_decrypt(encrypted)
decrypted = cipher.decrypt(encrypted_bytes)
decrypted_str = decrypted.decode('utf-8')
return decrypted_str

调用测试, 正常使用

def main():
str2 = "tf2ksQBQdzS0O4pozJZG55N+h+NaUCxoVjA/le6WgVraBSKTbmTMk5Bs/afw/3ZHSAmWr3q29XCMStxZdvhXNFCMyAJEmzZHMg47xbL1C+JHi6aPP7bte+nc5sl0nssaXNOb4uLBZrxcRTXf5eUzY9sGacePXL9lBWBppsEmt6rko4x4BIgFSeL61mBhx0ugsrIiK9vwp1B0Gnm95glDa2TQYfNZPxP0/tC/4wuHbiWzZHtJr9JAjc30FNBUrWbD6fJRNdpdpBTsrVVidLavscc8lkYZbKIUyro/PCbJvLnCWwuNUUXH1Fi+Qr9zmlchbnm+6JZNkq2awgTuz0OidYa+sUW/oGYYFw98PKkCAwV/XhedYGku/VB9Yu06CzsrHFtBiEzDhtF2VJrE/IN/vUgbO0MHtlaJdmJsJQT2/7krJayaV/19OpL3ap+xmJjC2m2I4FRijiam02P9EJNL2okjOGF6aiSBzh4Zr6ds1XdODwHawPvBWKUhRF3454oml73heHfCay7x/sbBPxgrbl7GthVzLMqXrSROtgEhE04lhJ+249NM1E8xhSa5GsZCKbcxCKFfizofLjztSJocDNY+CCFzaW6RzOpapanFzuB2as4n7CK53lpFHin059mV79MLB+vWDRRUYPjnsdFMj6R1X/GuQnWWD9giHRogF33T5fzkinE5gEtgBq1TkQyHHu18QdeOVD7IdkGSx1fBvtxF8rCAvcBQoXhCMMMP2zBmScEnJ4Fm86P688HpM7y0k0fyJV7H2WzABXV1I4W83YkUBxV2Oh//MDMxIVhfId8ZMfXGGPtdaQkh2ddtP3v9H0h7tkJ2sunDz7D+sGSGPLZu8SBUwh9yh92zgLleoFbl93BQCmmiuqW9WKpBJzyak23UEqZNdDUOlR0CsGQWl0HpDql1aiyM/hrVhgMxf4YdaipOzAdVIK0ykSS5jkzKHbQVIiP/eTz7qg5vctHumJBeF9bHH7yFAR/+RtjxCdODVNCkIAiCe1uuw4i94m8P9MVoA8N92jZrPEKh7gXdu6RztOA+4DQTgGiydRxHie11FYsl4UoMH/Qy3ICkkA6t7H/up9TF37Pv/aDwJzl+TywOboa2jH9jY05XYjr9mVzGlDZ1Zgoz+eOzEGxi0A0OYQTeMw4vO3bJDrfXkFAbvQWWfwkJxViA5vVWgRg2DPdDtcAHPVB6wGuLko+8PXQ/wtU+yUIPtYrudYLC7zJxVdDuRWOeLfoW8xPip9HvyTtA9QjHUqc+Vz9riowWiexAUSeCsSgIS5jdCLnGctBLevqTM3K0ypECzEmskbOFjkBJmCqYU7AW38nJUD0Kwj4wAKU0X5EVM3GX6QByn+LAxSME6iNyMgrqmrkkvF0GMcScutthmWcrH/5rqUiHTpVEdTn0G/doGeJAj8wsvAEj7ZuVAQ0nC26bhrWyvdGUabp5CtPcUmgvlmR36nr8LmRHVUcmIaoKiriHOKQUYaxH0A0GHAv+Z4THE7/WxnPrNfZUSTgnKhc/AK6HrClZAjcqjXpqyf251qsO/l9HwIpgoPpjCdxgOfRqY4HGLfJt1qcDjJs/Ue+h+t3HE2PJVVEE2y2qpoAUlNKmtFcBJ0qe0/ASIT5yUIrFwydM1/Y+eBgbr06me0VQEOhOVgRrgiihxG9BPugpRdwJ2I5FLZTAMkXJEMxG4GotqQYxu08z/uCODDmZJB0PTR+Vq7MEb2u7HDts8gsrnah3jszQ73lBG2JL7LZyi2Xb7mXH60ydVUcnDJD5qwNYe6/T7BdHNXpnsw8ERSx8nsASkVwQI3kOSwQIeNx70j30I8xDyDDoCJt/6Ck5LVK//vFWbXTKMt9ytpXFzQ++tVg15K31oWQrDXLqx1OtQ3i22172IJ2kLTxyBJ9CIaKMbD18Fpku6PZUVyqITqpmV4eLBgq4+38cBCUqStStgwAl7I3mzKKmiMY0ckq6Tggt2iUwprnATR2DssuGhWP7IaMsAzj+W4sqarW2oTd5B+LGOg3HjgN5W3WPVW6smpQNXuwmMU+tCBPyLbGl5xJb8puWCHEfL4JpGGZ6LWhoLHoxIPNrknhz1EI3vh+2hnT/xyYrppVc3OTrTtl8JvAgqVkjwgVwmMEErj8mBkP/oE4eDZhpmVrv06Q2fZWDYgPnf+EfErXSE6UWYFTzoc7T6/GSEXlarLgxWk9JrOEA70r00jQL+3gdFC5RVAi0J51qDr2fApuYp8420nvu5u58ZZRYypLuX1+Vm2msZhEuwNX1/CezTD2aJOUSa0DJcKNKlMKECsuzXck/oUdV0OeyHS3gGeMjggHhs2S3AxkcaPSQDw0nkfhc0WVY+H0HZVjW0WtnlJHLnivRVTKfxjtQEdh03LSGxcblKns8Vm54Nj18FbI+RovK+Iunn9y4S+yfpsnX2Yari7VPiJgyR0UxS6xdNAbNKB5U7T+4dqOSR4Fv3hfDNH0Me0fxc77oYnWc+VjPeoR7UUaY+2udKk0/20sOZUbTE2cqy/epLdPg0y1A3BxGUXcoGCqQrj2AYIq7saXpTqN4pN3VNoE0SHhwjxKAbCIgGr5b3En4WbxIekWGPw5MH8mKBtiyRZYfu3cTdz3cFtnKYbHEQa3r27AKEsOcUqk1QXBokcbX1rquP/E="
str3 = "hls_898e6b180e94f8ac356c381fc7d60aa5"
test_arr = [(str3, str2)]
for hls, encrypted in test_arr:
decrypted_str = decrypt_by_key(hls, encrypted)
print("HLS m3u8 解密文本为: ")
print(decrypted_str)
if __name__ == '__main__':
main()

不过加密的请求参数p还没有找到, 但是这是使用 mitmproxy 拦截的响应, 无需关心请求参数

踩坑一天, 总结一下

主要还是对这种视频文件格式什么的都不太了解, 导致播放都费了许多时间

看反编译的代码并不要去纠结那些细节, 很多代码并不关解密逻辑, 能看懂个大概就好了, 还可以大胆猜测然后去验证

很多加密算法可以直接使用原生算法库去尝试还原, 魔改算法的相对较少

【2024-11-15 0:11:32 星期五】

作业帮视频抓取踩坑
https://blog.tiamix.cn/posts/作业帮视频抓取踩坑/
作者
Tiam
发布于
2024-11-15
许可协议
CC BY-NC-SA 4.0