Adding subtitles to videos has become quite convenient with the help of speech recognition technology (ASR). In particular, OpenAI's Whisper series of models excel in converting speech to text, making automatic subtitle generation possible.

However, extracting hard subtitles (subtitles embedded in the video frames) from videos still presents many challenges.

Videos are essentially composed of consecutive image frames. A common frame rate is 30fps (30 frames per second), meaning a 1-hour video contains 108,000 images. For high-definition videos, the number of frames is even higher. This massive amount of data poses a significant challenge to OCR processing capabilities.

Google's Gemini-2.0-flash model not only supports text generation but also video and image recognition and processing, with a generous daily free quota, making it suitable as an OCR tool.

Domestically, Zhipu AI's glm-4v-flash model is not only free but also has strong image understanding capabilities, making it suitable as an OCR tool. Although it currently only supports Chinese and English recognition, it is sufficient for most scenarios.

Developed a hard subtitle extraction software based on Gemini and Zhipu AI

Download Link: GVS Chinese-English Video Hard Subtitle Extraction Software (640MB)

Baidu Netdisk Download: https://pan.baidu.com/s/1SDKm5tWsr6dkajhsf8T5Ew?pwd=95i4

Github Download: https://github.com/jianchang512/stt/releases/download/0.0/GVS-v0.2-AI.7z

Software Usage Guide

Here are the steps to use the software:

1. Download and Extract: Download the software compressed package, extract it, and double-click app.exe to run.

2. Select Video: Click the button at the top of the software interface to select the video file from which you want to extract subtitles. Ensure the video contains hard subtitles.

3. Select Subtitle Position: Choose the position of the subtitles in the video. The default is "Bottom," but you can also select "Top," "Middle," or "All" areas.

4. Enter API Key:

   You can enter the API key for Zhipu AI, which is free domestically.

   Alternatively, you can enter the API key for Gemini AI, which offers 1500 free calls per day. However, using it domestically requires a VPN. Multiple keys can be entered, separated by commas.

   Zhipu AI platform allows free registration and API Key acquisition: https://bigmodel.cn/usercenter/proj-mgmt/apikeys

   

   Gemini API Key can be obtained from this page: https://aistudio.google.com/app/apikey

5. Select Model: Zhipu AI supports the free GLM-4V-FLASH model. Gemini AI supports the gemini-2.0-flash-exp and gemini-1.5-flash models.

6. If using Gemini AI, enter the proxy IP and port, or enable system proxy in the VPN software.

7. Start Extraction: Click the "Start" button, and the text box at the bottom of the software will display progress and log information. After extraction is complete, an SRT subtitle file with the same name as the video file will be generated in the video file's directory.

Technical Principles

The core steps of the software for extracting hard subtitles are as follows:

1. Video Frame Extraction: First, use the FFmpeg tool to split the video into image frames at 1-second intervals. Choosing 1-second intervals instead of extracting every frame significantly reduces the number of images to be recognized. Additionally, since subtitles typically last at least 1 second, extracting too many frames would increase the difficulty of deduplication.
2. OCR Recognition: Send the extracted image frames to the AI model for OCR recognition to extract text from the images.
3. Subtitle Deduplication: Since consecutive image frames may contain the same subtitle content, to avoid repetition, we use the sentence-transformers model to calculate the similarity between the currently recognized subtitle and the previous one. If the similarity exceeds 60%, the two subtitles are considered identical and deduplicated.
4. Generate Subtitle File: Finally, the deduplicated subtitle text is assembled according to the corresponding timestamps and saved as an SRT format subtitle file.