Mastering Shell Commands in Python: From Basics to Advanced Applications (Windows and Linux)

Why This Article?

If you're familiar with Python 3, you might know that subprocess is a great tool for running shell commands. It allows Python to call system commands with powerful functionality.

However, it has many options and complex details like pipes, input/output, error handling, and encoding issues, which can often be confusing. Many times, we just copy code snippets that work without understanding why they're written that way or when to use a different approach.

Today, based on the official documentation, I revisited subprocess to gain a comprehensive and thorough understanding, knowing how to use it in various scenarios and how to flexibly combine options!

---

What is subprocess and Why Use It?

subprocess is a Python module for running system commands. For example:

• Run ls -l on Linux to list directories.
• Run dir on Windows to view files.

Why not use shell scripts? Python code is clearer, easier to maintain, and allows adding logical processing. subprocess.run (introduced in Python 3.5) is the most user-friendly function, and we'll focus on it today.

---

Overview of Common Parameters in subprocess.run

subprocess.run has many parameters. Let's introduce them first, then dive into usage in specific scenarios. This way, you'll get an overall idea of what each option does.

Parameter	Purpose	Common Values	Notes
args	Command to run	List (e.g., ["ls", "-l"]) or string (e.g., "ls -l")	Use list for no shell, string for shell=True
shell	Whether to use shell execution	True / False (default)	True is less efficient; required for Windows built-in commands
capture_output	Whether to capture stdout and stderr	True / False (default)	Equivalent to stdout=PIPE, stderr=PIPE
stdout	Destination for stdout	PIPE / None / STDOUT / file object	PIPE captures, None displays in terminal
stderr	Destination for stderr	PIPE / None / STDOUT / file object	STDOUT merges with stdout
text	Whether to return strings directly	True / False (default)	True requires encoding, avoids decoding
encoding	Output encoding	"utf-8" / "gbk" etc.	Recommended "utf-8", note system encoding on Windows
errors	Error handling for decoding	"strict" / "ignore" / "replace"	Only effective with text=True
input	Input data for command	String (e.g., "data")	Use string with text=True, bytes otherwise
check	Check return code, raise exception on failure	True / False (default)	Raises CalledProcessError

Return object (cp or exception e):

• cp.returncode: Return code (0 for success, non-zero for failure).
• cp.stdout: Standard output.
• cp.stderr: Error output (or logs, e.g., FFmpeg).
• e.stdout / e.stderr: Output and error when exception occurs.

---

Core Scenarios and Usage: From Simple to Complex

Let's use these parameters to explore common scenarios step by step.

Scenario 1: Run Simple Commands and Capture Output

Linux: Run echo

import subprocess

cp = subprocess.run(
    args=["echo", "hello world"],
    capture_output=True,    # Capture stdout and stderr
    text=True,              # Return strings directly
    encoding="utf-8"        # UTF-8 encoding
)
print(cp.stdout)  # Output: hello world

Windows: Run dir

import subprocess

cp = subprocess.run(
    args="dir",
    shell=True,             # Windows built-in commands need shell
    capture_output=True,
    text=True,
    encoding="utf-8"
)
print(cp.stdout)  # Output: Directory list

Key Points:

• capture_output=True simplifies capturing.
• Use list on Linux, shell=True for Windows built-in commands.

When to Use?

• Run simple commands and get results.

---

Scenario 2: Run Complex Commands (with Pipes |)

Linux: Run ls -l | grep file

import subprocess

cp = subprocess.run(
    args="ls -l | grep file",
    shell=True,
    capture_output=True,
    text=True,
    encoding="utf-8"
)
print(cp.stdout)  # Output: Filtered file list

Windows: Run dir | find "txt"

import subprocess

cp = subprocess.run(
    args='dir | find "txt"',
    shell=True,
    capture_output=True,
    text=True,
    encoding="utf-8"
)
print(cp.stdout)  # Output: Lines containing "txt"

Key Points:

• shell=True supports pipes.

When to Use?

• Combine commands to filter output.

---

Scenario 3: Capture Output and Error Separately

Linux: Run ls on Non-existent File

import subprocess

cp = subprocess.run(
    args=["ls", "nope"],
    stdout=subprocess.PIPE,  # Capture output separately
    stderr=subprocess.PIPE,  # Capture error separately
    text=True,
    encoding="utf-8"
)
print(f"Output: {cp.stdout}")
print(f"Error: {cp.stderr}")  # Output: ls: cannot access 'nope'

Windows: Run dir on Non-existent File

import subprocess

cp = subprocess.run(
    args="dir nope",
    shell=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
    encoding="utf-8"
)
print(f"Output: {cp.stdout}")
print(f"Error: {cp.stderr}")  # Output: File not found

Key Points:

• stdout=PIPE, stderr=PIPE capture separately.

When to Use?

• Distinguish between output and error.

---

Scenario 4: Check Results and Handle Exceptions

Linux: Check Return Code vs. Raise Exception

import subprocess

# Method 1: Check return code
cp = subprocess.run(
    args=["ls", "nope"],
    capture_output=True,
    text=True,
    encoding="utf-8"
)
if cp.returncode != 0:
    print(f"Failed! Return code: {cp.returncode}")
    print(f"Error: {cp.stderr}")

# Method 2: Use check=True
try:
    cp = subprocess.run(
        args=["ls", "nope"],
        capture_output=True,
        text=True,
        encoding="utf-8",
        check=True
    )
except subprocess.CalledProcessError as e:
    print(f"Failed! Return code: {e.returncode}")
    print(f"Output: {e.stdout}")
    print(f"Error: {e.stderr}")

Windows: Similar Handling

import subprocess

try:
    cp = subprocess.run(
        args="dir nope",
        shell=True,
        capture_output=True,
        text=True,
        encoding="utf-8",
        check=True
    )
except subprocess.CalledProcessError as e:
    print(f"Failed! Return code: {e.returncode}")
    print(f"Output: {e.stdout}")
    print(f"Error: {e.stderr}")

Key Points:

• capture_output=True only captures.
• check=True raises exception on failure.

When to Use?

• Ensure command success.

---

Scenario 5: Call External Programs (e.g., FFmpeg)

Important: FFmpeg logs normal output to stderr, not stdout

Transcode Video File

Use FFmpeg to convert input.mp4 to output.mp3:

import subprocess

cp = subprocess.run(
    args=["ffmpeg", "-i", "input.mp4", "output.mp3", "-y"],
    capture_output=True,
    text=True,
    encoding="utf-8"
)
if cp.returncode == 0:
    print("Transcode successful!")
    print(f"Logs: {cp.stderr}")  # FFmpeg normal logs in stderr
else:
    print(f"Transcode failed! Return code: {cp.returncode}")
    print(f"Output: {cp.stdout}")
    print(f"Error details: {cp.stderr}")

# Or use check=True
try:
    cp = subprocess.run(
        args=["ffmpeg", "-i", "input.mp4", "output.mp3", "-y"],
        capture_output=True,
        text=True,
        encoding="utf-8",
        check=True
    )
    print("Transcode successful!")
    print(f"Logs: {cp.stderr}")  # FFmpeg normal logs in stderr
except subprocess.CalledProcessError as e:
    print(f"Transcode failed! Return code: {e.returncode}")
    print(f"Output: {e.stdout}")
    print(f"Error details: {e.stderr}")

Key Points:

• Use list to call external programs.
• FFmpeg normal logs and errors are in stderr, stdout is usually empty.
• With check=True, success logs use cp.stderr, failure details use e.stderr.

When to Use?

• Handle audio/video, file conversion.

---

Scenario 6: Input Data to Command

import subprocess

data = "line1\nline2 py\nline3"
cp = subprocess.run(
    args=["grep", "py"],
    input=data,
    capture_output=True,
    text=True,
    encoding="utf-8"
)
print(cp.stdout)  # Output: line2 py

Key Points:

• input passes data to the command.

When to Use?

• Process program-generated data.

---

Detailed Explanation of Option Combinations

1. shell=True or False?

• False: Efficient and secure, use list.
• True: Supports complex commands, use string.

---

2. capture_output=True vs. check=True

• capture_output=True: Capture output and error, don't care about result.
• check=True: Check return code, raise exception on failure.
• Combination:

  cp = subprocess.run(["ls", "nope"], capture_output=True, check=True, text=True, encoding="utf-8")

---

3. stdout and stderr

• PIPE: Capture to program.
• None: Display in terminal.
• STDOUT (only for stderr): Merge with stdout.
• File: Write to file.

---

4. text=True

• Purpose: Return strings, requires encoding.
• Without it: Returns bytes, requires manual decoding.

---

5. encoding and errors

• encoding="utf-8": Recommended.
• errors: Use replace to prevent garbled text.

---

Common Questions

1. Why always use shell=True on Windows?

   • Built-in commands depend on cmd.exe.

2. What if FFmpeg outputs to stderr?

   • Use capture_output=True, normal and error output are in cp.stderr or e.stderr.

3. What if encoding is messed up?

   • Use text=True, encoding="utf-8", errors="replace".

---

How to Use in Different Scenarios?

Scenario	Usage	Option Combination
Simple commands	["cmd", "arg"]	capture_output=True, text=True
Complex commands	`"cmd1	cmd2"`
Separate capture	["cmd", "arg"]	stdout=PIPE, stderr=PIPE
Check results	Add check=True	capture_output=True
External tools	["ffmpeg", "-i", "in", "out"]	capture_output=True, check=True
Input data	Use input	text=True, encoding="utf-8"

Core Tips:

• Use capture_output=True to simplify capturing.
• text=True is convenient, check=True is strict.
• Note stderr for tools like FFmpeg.