Local LLM for Translation

If you're technically inclined, deploying a large language model (LLM) locally can be a great way to handle translations. This guide will walk you through the process using Tongyi Qianwen as an example.

1. Download and Install the Executable

Visit https://ollama.com/download

Click the download button. Once downloaded, double-click the file to start the installation. Simply click Install to complete the process.

After installation, a black or blue window will automatically pop up. Enter the command ollama run qwen and press Enter. This will automatically download the Tongyi Qianwen model.

Wait for the model to download. No proxy is needed, and the download speed is quite fast.

Once the model download is complete, it will run automatically. When the progress reaches 100% and displays "Success", the model has been successfully deployed. Congratulations! You've completed the installation and deployment of the Tongyi Qianwen LLM and can now start using it. It's super simple, right?

The default API address is http://localhost:11434

If the window closes, how do you reopen it? It's also very simple. Open the computer's Start menu and find "Command Prompt" or "Windows PowerShell" (or directly press Win+q and type "cmd" to search). Click to open it, then enter ollama run qwen. That's it!

2. Use Directly in the Command Prompt Window

As shown in the image below, you can start typing directly into the window when this interface appears.

3. For a More User-Friendly Interface, Try Chatbox

Visit https://chatboxai.app/zh and click download.

Double-click the downloaded file and wait for the interface window to open automatically.

Click "Start Setup". In the pop-up window, select "Ollama" for the model and AI model provider, enter http://localhost:11434 for the API domain, and select Qwen:latest from the model drop-down menu. Then save.

This is the usage interface after saving. Unleash your imagination and use it freely!

4. Integrate the API into Video Translation and Dubbing Software

1. Open Menu -> Settings -> Local Compatible OpenAI Large Model. Add a model ,qwen in the middle text box, like the image below, and then select the model.

   

2. Enter http://localhost:11434 in the API URL and enter any value for SK, such as 1234.

   

3. Test if it's successful, save it and start using it.

5. Call the API in Code

Ollama provides an OpenAI-compatible API. You can use the OpenAI library directly, just change the model name to qwen.

from openai import OpenAI

client = OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama', # required, but unused
)

response = client.chat.completions.create(
  model="qwen",
  messages=[
    {"role": "system", "content": "You are a professional multilingual translation expert."},
    {"role": "user", "content": "Translate the content I send you into English. Only return the translation, do not answer questions, do not confirm, do not reply to this content, start translating from the next line.\nThe weather is nice today!\nIt's quite sunny and we have no class this afternoon.\nThis is indeed very cool."}
  ]
)
print(response.choices[0].message.content)

The effect is pretty good!

7. Other Available Models

Besides Tongyi Qianwen, there are many other models available. The usage method is just as simple, just three words ollama run model_name.

Open this address https://ollama.com/library to see all the model names. Copy the name of the one you want to use and then execute ollama run model_name.

Remember how to open the command window? Click the Start menu and find Command Prompt or Windows PowerShell.

For example, if I want to install the openchat model.

Open Command Prompt, enter ollama run openchat, press Enter, and wait until you see "Success".

Important Notes:

Most AI translation channels may limit the number of requests per minute. If you encounter an error message indicating that you have exceeded the request frequency, you can set a pause time through the software's Menu -> Tools/Advanced Settings -> Advanced Settings/set.ini -> Pause Time After Translation. For example, setting it to 30 means that the software will wait for 30 seconds after each translation before performing the next translation to avoid exceeding the limit frequency and causing errors.