基于 selenium 批量下载抖音小姐姐Bulk Download Douyin Videos with Selenium

注意：这篇文章上次更新于1128天前，文章内容可能已经过时。

This article was last updated1128 days ago, the content may be outdated.

前言

考虑到硬盘空间不使用是一种极大的浪费现象，为了提高磁盘空间的利用率，想要存储更多学习资料在里面，抖音上有铺天盖地的小姐姐教我们跳舞，是极好的学习资源库。因此，出于学习目的，就想要把她们的学习资料保存到我的本地空间，免得哪一天她们被封了我想看也看不到学也学不到了。

Preface

Considering that leaving disk space unused is a huge waste, and in order to improve the utilization of disk space and store more study materials in it — Douyin is full of young ladies teaching us how to dance, making it an excellent learning resource library. Therefore, for study purposes, I wanted to save their study materials to my local space, lest they get banned one day and I can no longer watch learn from them.

工欲善其事 必先利其器

考虑到 GayHub 上应该有很多和我一样的好学人士，已经上传了他们下载学习资料的工具。经过一番检索和尝试，都以失败告终，原因应该是多样的，推测是他们的项目没跟上抖音网站的更新。

迫于无奈，只能尝试自己写一个。奈何自己前端知识极其匮乏，无法写出爬虫程序。

想起两年前用 RPA-Python 写过一个自动登陆校园网 的程序，这个库简单实用，直接操作浏览器，不需要懂什么逆向等乱七八糟的，会点击下载就行，于是想继续使用这个库。但这个库在 Linux 环境下需要特殊安装，所以还是算了吧。

因为，实际上在控制浏览器这个需求里，更著名的库是 selenium 。

Ubuntu22 环境下安装 selenium

安装 selenium 在不同的操作系统下应该都是差不多的。但安装过程中我也才过一些坑，这里记录一下成功安装的过程。

1. 安装 selenium

   pip install selenium

2. 安装 chrome 浏览器

   我这里是图形界面环境，直接去官网下载 deb 包安装即可。

3. 下载 chromedriver

   下载 chromedriver 之前首先检查 chrome 版本。可以在浏览器关于界面中进行查看，也可以在终端执行 google-chrome --version 来查看。

   输出如下：

   wgx@wgx-Lenovo:~$ google-chrome --version
   Google Chrome 114.0.5735.198

   接下来打开下载 chromedriver 的网址：https://chromedriver.chromium.org/downloads

   选择与浏览器对应的版本进行下载。这里我的浏览器版本太新了，没有能够完全对应的 chromedriver ，于是我下载了 114.0.5735.90 版本，只有最后一位小版本号对不上，应该不会有问题，事实证明确实如此。

   

   下载完成后，将 chromedriver 复制到 /usr/local/bin/ 目录下。

   执行下面代码进行测试，如果能够成功启动浏览器表示安装成功。

   from selenium import webdriver
   from selenium.webdriver.chrome.options import Options
   
   options = Options()
   options.add_argument("--disable-dev-shm-usage")
   options.add_argument("--no-sandbox")
   
   driver = webdriver.Chrome(options=options)
   input()
   driver.quit()

在浏览器上下载抖音视频

抖音网站上没有提供下载地址，需要 F12 在 video 标签中找到视频的地址。

以周姐的视频为例：

至于这 3 个视频地址的区别，我还没发现。任意一个都可以下载视频。

我以为是存在清晰度的区别，但我下载了这三个地址的视频，大小都是一样的。

学会了手动下载，接下来就可以使用 selenium 控制浏览器自动下载了。

selenium 的基本操作

启动浏览器并打开一个网页

from selenium.webdriver.chrome.options import Options
from selenium import webdriver
options = Options()
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")

driver = webdriver.Chrome(options=options)
driver.get("https://www.douyin.com")

获取页面中的元素

获取页面中的元素主要通过 find_element 或者 find_elements 方法，前者返回元素对象，后者返回元素对象的列表。

查找元素需要使用 By 类中提供的定位策略，以下是一些常用的 By 类方法及其解释：

1. By.ID：通过元素的唯一标识符 id 来定位元素。

2. By.NAME：通过元素的 name 属性来定位元素。

3. By.XPATH：通过元素的 XPath 表达式来定位元素。XPath 是一种用于在 XML 或 HTML 中导航和定位元素的语言。

4. By.CSS_SELECTOR：通过元素的 CSS 选择器来定位元素。CSS 选择器是一种通过样式规则匹配元素的方法。

5. By.CLASS_NAME：通过元素的 class 属性来定位元素。

6. By.TAG_NAME：通过元素的标签名称来定位元素。

7. By.LINK_TEXT：通过元素的完整文本内容来定位超链接元素 <a>。

8. By.PARTIAL_LINK_TEXT：通过元素的部分文本内容来定位超链接元素 <a>。

   下面是基本的示例代码：

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 使用 By.ID 定位元素
element_by_id = driver.find_element(By.ID, "element_id")

# 使用 By.XPATH 定位元素
element_by_xpath = driver.find_element(By.XPATH, "//div[@class='example']")

# 使用 By.NAME 定位元素
element_by_name = driver.find_element(By.NAME, "element_name")

# 使用 By.CSS_SELECTOR 定位元素
element_by_css = driver.find_element(By.CSS_SELECTOR, ".example-class")

# 使用 By.CLASS_NAME 定位元素
element_by_class = driver.find_element(By.CLASS_NAME, "example-class")

# 使用 By.TAG_NAME 定位元素
element_by_tag = driver.find_element(By.TAG_NAME, "div")

# 使用 By.LINK_TEXT 定位超链接元素
element_by_link_text = driver.find_element(By.LINK_TEXT, "Click here")

# 使用 By.PARTIAL_LINK_TEXT 定位超链接元素
element_by_partial_link_text = driver.find_element(By.PARTIAL_LINK_TEXT, "here")

在网页中按键盘

在网页中按键盘依赖于 Keys 类，常见的按键如下：

1. Keys.ENTER：模拟 Enter 键。
2. Keys.RETURN：与 Keys.ENTER 相同，模拟 Enter 键。
3. Keys.TAB：模拟 Tab 键。
4. Keys.ESCAPE：模拟 Escape 键。
5. Keys.SPACE：模拟空格键。
6. Keys.BACK_SPACE：模拟退格键。
7. Keys.DELETE：模拟删除键。
8. Keys.UP：模拟方向键上。
9. Keys.DOWN：模拟方向键下。
10. Keys.LEFT：模拟方向键左。
11. Keys.RIGHT：模拟方向键右。
12. Keys.HOME：模拟 Home 键。
13. Keys.END：模拟 End 键。
14. Keys.PAGE_UP：模拟 Page Up 键。
15. Keys.PAGE_DOWN：模拟 Page Down 键。
16. Keys.CONTROL：模拟 Control 键。
17. Keys.SHIFT：模拟 Shift 键。
18. Keys.ALT：模拟 Alt 键。

Keys类的使用方法依赖于 send_keys() 方法，该方法用于将指定按键输入到特定的网页元素上。例如：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 使用 By.ID 定位元素
element_by_id = driver.find_element(By.ID, "element_id")

# 按下 Enter 键
element_by_id.send_keys(Keys.ENTER)

鼠标操作

在 Selenium 中，要实现鼠标点击一个元素，你需要使用 ActionChains 类。ActionChains 类提供了一组用于模拟鼠标操作的方法，包括点击、双击、拖放等。

以下是一个示例，展示如何使用 ActionChains 类来实现鼠标点击一个元素：

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 找到需要点击的元素
element_to_click = driver.find_element(By.ID,"element_id")

# 创建 ActionChains 对象
actions = ActionChains(driver)

# 执行鼠标点击操作
actions.click(element_to_click).perform()

获取元素的属性

以上这些内容可以完成基本的浏览器操作了，目前距离实现我的学习目标还差最后一步，就是如何获得标签的属性，也即获取 source 标签下的 src 属性。

实际上只需简单一个方法 get_attribute()

例如：

source.get_attribute("src")

完整代码

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import requests
import os
import time 

options = Options()
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")

driver = webdriver.Chrome(options=options)


def download(config):
    user_count = 1
    login = False

    for user in config['user_list']:
        # 打开主页
        driver.get("https://www.douyin.com/user/"+user)
        driver.implicitly_wait(10)
        time.sleep(5)
        if not login:
            input("登陆后请关闭提示框，然后按回车键：")
            login = True

        ## 创建存储目录
        dir_path = f"{config['save_to']}/{user}"
        if not os.path.exists(dir_path):
            os.mkdir(dir_path)

        # 点击第一个视频
        first = driver.find_element(By.XPATH,'/html/body/div[2]/div[1]/div[2]/div[3]/div/div/div[3]/div[2]/div/div[2]/ul/li[1]/div/a/div')
        first.click()
        driver.implicitly_wait(5)
        time.sleep(2)

        # 跳过向导
        if user_count == 1:
            next_btn = driver.find_element(By.TAG_NAME,"body")
            next_btn.send_keys(Keys.DOWN)

        # 打开列表
        xpath = '//*[@id="sliderVideo"]/div[1]/div[2]/button'
        open_list = driver.find_element(By.XPATH,xpath)
        open_list.click()

        if user_count == 1:
            input("即将开始正式下载，手动选择清晰度后，按回车键：")

        video_name = 0 
        last_src = ""
        while True:
            driver.implicitly_wait(2)
            ul = driver.find_element(By.XPATH,'//*[@id="relatedVideoCard"]/div/div[3]/div/div/div[2]/div/div[1]/div/ul')
            lis = ul.find_elements(By.TAG_NAME,'li')
            li_size = len(lis)
            if(video_name == li_size):
                ul = driver.find_element(By.XPATH,'//*[@id="relatedVideoCard"]/div/div[3]/div/div/div[2]/div/div[1]/div/ul')
                lis = ul.find_elements(By.TAG_NAME,'li')
                li_size = len(lis)
                if(video_name == li_size):
                    print(f"第{user_count}个用户下载完毕，共下载{li_size}个视频")
                    break
            lis[video_name].click()
            video = driver.find_element(By.XPATH,'//*[@id="slideMode"]/div[1]/div[1]/div/xg-video-container/video')
            try:
                source = video.find_element(By.TAG_NAME,"source")
                if source == last_src:
                    driver.refresh()
                    driver.implicitly_wait(5)
                    xpath = '//*[@id="sliderVideo"]/div[1]/div[2]/button'
                    open_list = driver.find_element(By.XPATH,xpath)
                    open_list.click()
                    continue
                else:
                    last_src = source
            except Exception as e:
                print(e)
                driver.refresh()
                driver.implicitly_wait(5)
                xpath = '//*[@id="sliderVideo"]/div[1]/div[2]/button'
                open_list = driver.find_element(By.XPATH,xpath)
                open_list.click()
                continue
            response = requests.get(source.get_attribute("src"))
            if response.status_code == 200:
                content = response.content
                with open(f"{dir_path}/{video_name+1}.mp4", "wb") as f:
                    f.write(content)
                    video_name += 1
                    print(f"第{video_name}个下载完成！")
            else:
                print(f"第{video_name}个下载失败！")
        user_count += 1

if __name__ == '__main__':

    config = {
        'save_to': ".",
        'user_list': [
            'MS4wLjABAAAAYIN1c9oIG_oCZuDe2rYbRdfXs86P9hMhoFW4pMq3FZU',
            'MS4wLjABAAAAbgCnupO_NGaTAmzWnXSivCeHWrOe0wC2ZcpNvVoQfEk'
        ]
    }

    download(config)

    # 关闭浏览器
    driver.quit()

使用方法

文章开头已经用视频演示了使用方法，这里补充图文说明。

首先，代码中的 config 字典两个字段分别表示保存路径和用户列表，用户列表中填写的内容是下载用户抖音主页的链接。

修改后，运行代码。

启动浏览器后，首先遇到验证码中间页，这里手动验证。

然后进入登录页，这里还是手动扫码登录。

登录后，有时会遇到下面的提示框，这个框需要手动关闭。

这里其实可以做成自动的，但是我调试的时候不是每次都出，所以就没有编写在代码里。这里的另一种解决方案是干脆睡眠5秒钟。

接下来回到终端，输入回车键，让程序继续执行。

接下来手动选择清晰度后，继续在终端回车。

OK！学习资料已经源源不断地流入本地磁盘了😋。

Sharpen Your Tools Before Working

Considering that GayHub probably has many diligent learners just like me who have already uploaded their tools for downloading study materials, I searched and tried several, but all of them ended in failure. The reasons were probably varied — my guess is that their projects didn’t keep up with the updates of the Douyin website.

Forced by circumstances, I had to try writing one myself. Unfortunately, my front-end knowledge is extremely limited, so I couldn’t write a crawler program.

I remembered writing an auto campus-network login program with RPA-Python two years ago. This library is simple and practical — it operates the browser directly, so you don’t need to understand reverse engineering or any such nonsense; if you can click and download, that’s enough. So I wanted to continue using this library. But it requires special installation in Linux environments, so forget it.

Because, in fact, for the need of controlling a browser, the more famous library is selenium.

Installing selenium on Ubuntu 22

Installing selenium should be roughly the same on different operating systems. But I also stepped on some pitfalls during the installation; here I record the successful installation process.

1. Install selenium

   pip install selenium

2. Install the Chrome browser

   I have a GUI environment here, so I just download the deb package from the official website and install it.

3. Download chromedriver

   Before downloading chromedriver, first check the Chrome version. You can check it in the browser’s About page, or run google-chrome --version in the terminal.

   The output is as follows:

   wgx@wgx-Lenovo:~$ google-chrome --version
   Google Chrome 114.0.5735.198

   Next, open the chromedriver download page: https://chromedriver.chromium.org/downloads

   Choose the version matching your browser to download. My browser version was too new here — there was no chromedriver that matched it exactly, so I downloaded version 114.0.5735.90. Only the last minor version number differs, which should not be a problem, and it indeed turned out to be the case.

   

   After the download is complete, copy chromedriver to the /usr/local/bin/ directory.

   Run the code below to test; if the browser launches successfully, the installation was successful.

   from selenium import webdriver
   from selenium.webdriver.chrome.options import Options
   
   options = Options()
   options.add_argument("--disable-dev-shm-usage")
   options.add_argument("--no-sandbox")
   
   driver = webdriver.Chrome(options=options)
   input()
   driver.quit()

Downloading Douyin Videos in the Browser

The Douyin website does not provide a download link; you need to press F12 and find the video address in the video tag.

Take Sister Zhou’s video as an example:

As for the difference between these 3 video addresses, I haven’t figured it out yet. Any of them can download the video.

I thought there might be a difference in resolution, but I downloaded the videos from all three addresses and their sizes were all the same.

Now that I’ve learned to download manually, I can use selenium to control the browser to download automatically.

Basic Selenium Operations

Launching the Browser and Opening a Web Page

from selenium.webdriver.chrome.options import Options
from selenium import webdriver
options = Options()
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")

driver = webdriver.Chrome(options=options)
driver.get("https://www.douyin.com")

Getting Elements in the Page

Getting elements in the page mainly uses the find_element or find_elements methods — the former returns an element object, the latter returns a list of element objects.

To find elements, you need the locating strategies provided by the By class. Here are some commonly used By class methods and their explanations:

1. By.ID: locates an element by its unique identifier id.

2. By.NAME: locates an element by its name attribute.

3. By.XPATH: locates an element by an XPath expression. XPath is a language for navigating and locating elements in XML or HTML.

4. By.CSS_SELECTOR: locates an element by a CSS selector. A CSS selector is a way to match elements through style rules.

5. By.CLASS_NAME: locates an element by its class attribute.

6. By.TAG_NAME: locates an element by its tag name.

7. By.LINK_TEXT: locates an anchor element <a> by its complete text content.

8. By.PARTIAL_LINK_TEXT: locates an anchor element <a> by part of its text content.

   Here is a basic example:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 使用 By.ID 定位元素
element_by_id = driver.find_element(By.ID, "element_id")

# 使用 By.XPATH 定位元素
element_by_xpath = driver.find_element(By.XPATH, "//div[@class='example']")

# 使用 By.NAME 定位元素
element_by_name = driver.find_element(By.NAME, "element_name")

# 使用 By.CSS_SELECTOR 定位元素
element_by_css = driver.find_element(By.CSS_SELECTOR, ".example-class")

# 使用 By.CLASS_NAME 定位元素
element_by_class = driver.find_element(By.CLASS_NAME, "example-class")

# 使用 By.TAG_NAME 定位元素
element_by_tag = driver.find_element(By.TAG_NAME, "div")

# 使用 By.LINK_TEXT 定位超链接元素
element_by_link_text = driver.find_element(By.LINK_TEXT, "Click here")

# 使用 By.PARTIAL_LINK_TEXT 定位超链接元素
element_by_partial_link_text = driver.find_element(By.PARTIAL_LINK_TEXT, "here")

Pressing Keys in a Web Page

Pressing keys in a web page relies on the Keys class. The common keys are as follows:

1. Keys.ENTER: simulates the Enter key.
2. Keys.RETURN: same as Keys.ENTER, simulates the Enter key.
3. Keys.TAB: simulates the Tab key.
4. Keys.ESCAPE: simulates the Escape key.
5. Keys.SPACE: simulates the Space key.
6. Keys.BACK_SPACE: simulates the Backspace key.
7. Keys.DELETE: simulates the Delete key.
8. Keys.UP: simulates the Up arrow key.
9. Keys.DOWN: simulates the Down arrow key.
10. Keys.LEFT: simulates the Left arrow key.
11. Keys.RIGHT: simulates the Right arrow key.
12. Keys.HOME: simulates the Home key.
13. Keys.END: simulates the End key.
14. Keys.PAGE_UP: simulates the Page Up key.
15. Keys.PAGE_DOWN: simulates the Page Down key.
16. Keys.CONTROL: simulates the Control key.
17. Keys.SHIFT: simulates the Shift key.
18. Keys.ALT: simulates the Alt key.

The Keys class is used through the send_keys() method, which inputs the specified keys into a particular web page element. For example:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 使用 By.ID 定位元素
element_by_id = driver.find_element(By.ID, "element_id")

# 按下 Enter 键
element_by_id.send_keys(Keys.ENTER)

Mouse Operations

In Selenium, to click an element with the mouse, you need to use the ActionChains class. The ActionChains class provides a set of methods for simulating mouse operations, including clicking, double-clicking, dragging and dropping, and so on.

Here is an example showing how to use the ActionChains class to click an element with the mouse:

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get("https://www.example.com")

# 找到需要点击的元素
element_to_click = driver.find_element(By.ID,"element_id")

# 创建 ActionChains 对象
actions = ActionChains(driver)

# 执行鼠标点击操作
actions.click(element_to_click).perform()

Getting Element Attributes

With all of the above, you can perform basic browser operations. The last step to achieving my study goal is how to get the attributes of a tag — that is, getting the src attribute under the source tag.

Actually, it only takes one simple method: get_attribute()

For example:

source.get_attribute("src")

Full Code

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import requests
import os
import time 

options = Options()
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")

driver = webdriver.Chrome(options=options)


def download(config):
    user_count = 1
    login = False

    for user in config['user_list']:
        # 打开主页
        driver.get("https://www.douyin.com/user/"+user)
        driver.implicitly_wait(10)
        time.sleep(5)
        if not login:
            input("登陆后请关闭提示框，然后按回车键：")
            login = True

        ## 创建存储目录
        dir_path = f"{config['save_to']}/{user}"
        if not os.path.exists(dir_path):
            os.mkdir(dir_path)

        # 点击第一个视频
        first = driver.find_element(By.XPATH,'/html/body/div[2]/div[1]/div[2]/div[3]/div/div/div[3]/div[2]/div/div[2]/ul/li[1]/div/a/div')
        first.click()
        driver.implicitly_wait(5)
        time.sleep(2)

        # 跳过向导
        if user_count == 1:
            next_btn = driver.find_element(By.TAG_NAME,"body")
            next_btn.send_keys(Keys.DOWN)

        # 打开列表
        xpath = '//*[@id="sliderVideo"]/div[1]/div[2]/button'
        open_list = driver.find_element(By.XPATH,xpath)
        open_list.click()

        if user_count == 1:
            input("即将开始正式下载，手动选择清晰度后，按回车键：")

        video_name = 0 
        last_src = ""
        while True:
            driver.implicitly_wait(2)
            ul = driver.find_element(By.XPATH,'//*[@id="relatedVideoCard"]/div/div[3]/div/div/div[2]/div/div[1]/div/ul')
            lis = ul.find_elements(By.TAG_NAME,'li')
            li_size = len(lis)
            if(video_name == li_size):
                ul = driver.find_element(By.XPATH,'//*[@id="relatedVideoCard"]/div/div[3]/div/div/div[2]/div/div[1]/div/ul')
                lis = ul.find_elements(By.TAG_NAME,'li')
                li_size = len(lis)
                if(video_name == li_size):
                    print(f"第{user_count}个用户下载完毕，共下载{li_size}个视频")
                    break
            lis[video_name].click()
            video = driver.find_element(By.XPATH,'//*[@id="slideMode"]/div[1]/div[1]/div/xg-video-container/video')
            try:
                source = video.find_element(By.TAG_NAME,"source")
                if source == last_src:
                    driver.refresh()
                    driver.implicitly_wait(5)
                    xpath = '//*[@id="sliderVideo"]/div[1]/div[2]/button'
                    open_list = driver.find_element(By.XPATH,xpath)
                    open_list.click()
                    continue
                else:
                    last_src = source
            except Exception as e:
                print(e)
                driver.refresh()
                driver.implicitly_wait(5)
                xpath = '//*[@id="sliderVideo"]/div[1]/div[2]/button'
                open_list = driver.find_element(By.XPATH,xpath)
                open_list.click()
                continue
            response = requests.get(source.get_attribute("src"))
            if response.status_code == 200:
                content = response.content
                with open(f"{dir_path}/{video_name+1}.mp4", "wb") as f:
                    f.write(content)
                    video_name += 1
                    print(f"第{video_name}个下载完成！")
            else:
                print(f"第{video_name}个下载失败！")
        user_count += 1

if __name__ == '__main__':

    config = {
        'save_to': ".",
        'user_list': [
            'MS4wLjABAAAAYIN1c9oIG_oCZuDe2rYbRdfXs86P9hMhoFW4pMq3FZU',
            'MS4wLjABAAAAbgCnupO_NGaTAmzWnXSivCeHWrOe0wC2ZcpNvVoQfEk'
        ]
    }

    download(config)

    # 关闭浏览器
    driver.quit()

How to Use

The video at the beginning of this article already demonstrated how to use it; here I’ll add illustrated explanations.

First, the two fields of the config dict in the code represent the save path and the user list respectively; what you fill into the user list are the links to the Douyin homepages of the users to download.

After modifying, run the code.

After the browser launches, you’ll first encounter the captcha interstitial page — verify it manually here.

Then comes the login page — log in by scanning the QR code manually here as well.

After logging in, sometimes you’ll see the prompt box below, which needs to be closed manually.

This could actually be automated, but it didn’t appear every time while I was debugging, so I didn’t write it into the code. Another solution here is to simply sleep for 5 seconds.

Next, go back to the terminal and press Enter to let the program continue.

Next, manually choose the resolution, then press Enter in the terminal again.

OK! Study materials are already flowing continuously into my local disk 😋.