Subprocesses — Python documentation
Subprocesses
This section describes high-level async/await asyncio APIs to create and manage subprocesses.
Here’s an example of how asyncio can run a shell command and obtain its result:
import asyncio
async def run(cmd):
proc = await asyncio.create_subprocess_shell(
cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stdout, stderr = await proc.communicate()
print(f'[{cmd!r} exited with {proc.returncode}]')
if stdout:
print(f'[stdout]\n{stdout.decode()}')
if stderr:
print(f'[stderr]\n{stderr.decode()}')
asyncio.run(run('ls /zzz'))
will print:
['ls /zzz' exited with 1]
[stderr]
ls: /zzz: No such file or directory
Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel. It is indeed trivial to modify the above example to run several commands simultaneously:
async def main():
await asyncio.gather(
run('ls /zzz'),
run('sleep 1; echo "hello"'))
asyncio.run(main())
See also the Examples subsection.
Creating Subprocesses
Important
It is the application’s responsibility to ensure that all whitespace and special characters are quoted appropriately to avoid shell injection vulnerabilities. The shlex.quote() function can be used to properly escape whitespace and special shell characters in strings that are going to be used to construct shell commands.
Note
The default asyncio event loop implementation on Windows does not support subprocesses. Subprocesses are available for Windows if a ProactorEventLoop is used. See Subprocess Support on Windows for details.
See also
asyncio also has the following low-level APIs to work with subprocesses: loop.subprocess_exec()
, loop.subprocess_shell()
, loop.connect_read_pipe()
, loop.connect_write_pipe()
, as well as the Subprocess Transports and Subprocess Protocols.
Constants
- asyncio.subprocess.PIPE
Can be passed to the stdin, stdout or stderr parameters.
If PIPE is passed to stdin argument, the Process.stdin attribute will point to a StreamWriter instance.
If PIPE is passed to stdout or stderr arguments, the Process.stdout and Process.stderr attributes will point to StreamReader instances.
- asyncio.subprocess.STDOUT
- Special value that can be used as the stderr argument and indicates that standard error should be redirected into standard output.
- asyncio.subprocess.DEVNULL
- Special value that can be used as the stdin, stdout or stderr argument to process creation functions. It indicates that the special file os.devnull will be used for the corresponding subprocess stream.
Interacting with Subprocesses
Both create_subprocess_exec()
and create_subprocess_shell()
functions return instances of the Process class. Process is a high-level wrapper that allows communicating with subprocesses and watching for their completion.
- class asyncio.subprocess.Process
An object that wraps OS processes created by the
create_subprocess_exec()
andcreate_subprocess_shell()
functions.This class is designed to have a similar API to the subprocess.Popen class, but there are some notable differences:
unlike Popen, Process instances do not have an equivalent to the poll() method;
the
communicate()
andwait()
methods don’t have a timeout parameter: use thewait_for()
function;the
Process.wait()
method is asynchronous, whereas subprocess.Popen.wait() method is implemented as a blocking busy loop;the universal_newlines parameter is not supported.
This class is not thread safe.
See also the Subprocess and Threads section.
- send_signal(signal)
Sends the signal signal to the child process.
Note
On Windows,
SIGTERM
is an alias for terminate().CTRL_C_EVENT
andCTRL_BREAK_EVENT
can be sent to processes started with a creationflags parameter which includesCREATE_NEW_PROCESS_GROUP
.
- terminate()
Stop the child process.
On POSIX systems this method sends signal.SIGTERM to the child process.
On Windows the Win32 API function
TerminateProcess()
is called to stop the child process.
- kill()
Kill the child.
On POSIX systems this method sends
SIGKILL
to the child process.On Windows this method is an alias for terminate().
- stdin
Standard input stream (StreamWriter) or
None
if the process was created withstdin=None
.
- stdout
Standard output stream (StreamReader) or
None
if the process was created withstdout=None
.
- stderr
Standard error stream (StreamReader) or
None
if the process was created withstderr=None
.
Warning
Use the
communicate()
method rather than process.stdin.write(), await process.stdout.read() or await process.stderr.read. This avoids deadlocks due to streams pausing reading or writing and blocking the child process.- pid
Process identification number (PID).
Note that for processes created by the
create_subprocess_shell()
function, this attribute is the PID of the spawned shell.
- returncode
Return code of the process when it exits.
A
None
value indicates that the process has not terminated yet.A negative value
-N
indicates that the child was terminated by signalN
(POSIX only).
Subprocess and Threads
Standard asyncio event loop supports running subprocesses from different threads, but there are limitations:
- An event loop must run in the main thread.
- The child watcher must be instantiated in the main thread before executing subprocesses from other threads. Call the get_child_watcher() function in the main thread to instantiate the child watcher.
Note that alternative event loop implementations might not share the above limitations; please refer to their documentation.
Examples
An example using the Process class to control a subprocess and the StreamReader class to read from its standard output.
The subprocess is created by the create_subprocess_exec()
function:
import asyncio
import sys
async def get_date():
code = 'import datetime; print(datetime.datetime.now())'
# Create the subprocess; redirect the standard output
# into a pipe.
proc = await asyncio.create_subprocess_exec(
sys.executable, '-c', code,
stdout=asyncio.subprocess.PIPE)
# Read one line of output.
data = await proc.stdout.readline()
line = data.decode('ascii').rstrip()
# Wait for the subprocess exit.
await proc.wait()
return line
if sys.platform == "win32":
asyncio.set_event_loop_policy(
asyncio.WindowsProactorEventLoopPolicy())
date = asyncio.run(get_date())
print(f"Current date: {date}")
See also the same example written using low-level APIs.