Adding subtitles to videos has become quite convenient thanks to Automatic Speech Recognition (ASR) technology. In particular, OpenAI's Whisper series models excel at speech-to-text conversion, making automatic subtitle generation possible.

However, extracting existing hardcoded subtitles (subtitles embedded in the video frame) from videos still presents several challenges.

A video is essentially composed of a series of image frames. A common video frame rate is 30fps (30 frames per second), which means that a 1-hour video contains 108,000 images. For high-definition videos, the number of frames will be even higher. This massive amount of data poses a severe challenge to OCR processing capabilities.

Google's Gemini-2.0-flash model not only supports text generation but also supports video and image recognition and processing. Moreover, it offers a large daily free quota, making it useful as an OCR tool.

China's Zhipu AI glm-4v-flash model is not only free but also has powerful image understanding capabilities and can be used as an OCR tool. Although it currently only supports Chinese and English recognition, it is sufficient for most scenarios.

A hardcoded subtitle extraction software has been developed based on Gemini and Zhipu AI.

Download Address GVS Chinese and English Video Hardcoded Subtitle Extraction Software (640MB)

Baidu Netdisk Download: https://pan.baidu.com/s/1SDKm5tWsr6dkajhsf8T5Ew?pwd=95i4

Github Download: https://github.com/jianchang512/stt/releases/download/0.0/GVS-v0.2-AI.7z

Software Usage Guide

The following are the steps to use the software:

1. Download and Extract: Download the software package, extract it, and double-click app.exe to run it.

2. Select Video: Click the button at the top of the software interface to select the video file for which you want to extract subtitles. Make sure the video has hardcoded subtitles.

3. Select Subtitle Position: Select the position of the subtitles in the video. The default is "Bottom", but you can also select "Top", "Middle", or "All" areas.

4. Fill in API Key:

   You can fill in the api key of Zhipu AI, which is free in China.

   You can also fill in the api key of Gemini AI, which has a free daily quota of 1500 calls, but you need a proxy to use it in China. You can fill in multiple keys separated by commas.

   You can register for free and get an API Key on the Zhipu AI platform: https://bigmodel.cn/usercenter/proj-mgmt/apikeys

   

   You can get Gemini API key at: https://aistudio.google.com/app/apikey

5. Select Model: Zhipu AI supports the free GLM-4V-FLASH model. GeminiAI supports the gemini-2.0-flash-exp and gemini-1.5-flash models.

6. If you are using GeminiAI, you need to fill in the proxy IP and port, or enable the system proxy in the VPN software.

7. Start Extraction: Click the "Start" button, and the progress and log information will be displayed in the text box below the software. After the extraction is complete, an SRT subtitle file with the same name as the video file will be generated in the video file directory.

Technical Principle

The core steps of the software to extract hardcoded subtitles are as follows:

1. Video Frame Cutting: First, use the FFmpeg tool to cut the video into image frames at 1-second intervals. Selecting 1-second intervals instead of frame-by-frame extraction can significantly reduce the number of images to be recognized. Additionally, considering that subtitles usually last for no less than 1 second, too many frames will increase the difficulty of deduplication.
2. OCR Recognition: Send the segmented image frames to the AI model for OCR recognition to extract the text in the image.
3. Subtitle Deduplication: Since consecutive image frames may contain the same subtitle content, in order to avoid duplication, we use the sentence-transformers model to calculate the similarity between the currently recognized subtitle and the previous subtitle. If the similarity exceeds 60%, it is considered that the two subtitles have the same content, and deduplication is performed.
4. Generate Subtitle File: Finally, the deduplicated subtitle text is spliced according to the corresponding timestamps and saved as an SRT format subtitle file.