Python Popen doesn't recognize scrapy when python script is run as systemd service

python subprocess get output
python subprocess multiple commands
python subprocess output to variable
python subprocess example windows
python subprocess stdin
python subprocess run in background
python subprocess call(('python script with arguments))
subprocess.popen stdout

I have a python script main.py that needs to start the scrapy executable, so I use Popen to do that (using subprocess.call() has same result). To simplify things I just run the help argument of scrapy.

import subprocess
...
p = subprocess.Popen(['scrapy', '-h'])

The script needs to be run inside a vitualenv which has scrapy installed on it. When I activate the virtualenv and run the script as python main.py the command scrapy -h is executed as expected.

Now, I need this script to be run as a systemd service. I have created the systemd unit for the script as follows:

[Unit]
Description=Scrapy test
After=network.target

[Service]
User={{ scrapy_user }}
WorkingDirectory={{ test_path }}
ExecStart={{ virtualenv_path }}/bin/python {{ test_path }}/main.py

[Install]
WantedBy=multi-user.target

When I enable and start the service with sudo systemctl start scrapy-test the code runs normally until p = subprocess.Popen(['scrapy', '-h']) when I get an exception raised

File "{{ test_path }}/main.py", line 52, in launch_spider
   p = subprocess.Popen(['scrapy', '-h'])
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
   errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
   raise child_exception
OSError: [Errno 2] No such file or directory

I have also tried to change the working directory by adding os.chdir('{{ virtualenv_path }}/bin') just before Popen, but I get the exact same exception. I am sure the script is run through the virtualenv because (1) it is set so in the ExecStart attribute and (2) the script would raise exceptions earlier when importing modules that are not installed in the system's python.

The clue is here:

ExecStart={{ virtualenv_path }}/bin/python {{ test_path }}/main.py

The scrapy command is on $VIRTUAL_ENV/bin, which is not on the systemd $PATH.

You could try:

import os
import subprocess
p = subprocess.Popen(['%s/bin/scrapy' % os.environ['VIRTUAL_ENV'], '-h'])

17.1. subprocess — Subprocess management, The subprocess module allows you to spawn new processes, connect to their but does not suffer from this vulnerability; see the Note in the Popen constructor  The reason that os.popen() doesn’t work from within PythonWin is due to a bug in Microsoft’s C Runtime Library (CRT). The CRT assumes you have a Win32 console attached to the process. You should use the win32pipe module’s popen() instead which doesn’t depend on having an attached Win32 console. Example:

Another issue you may encounter later, you may want to make python to wait for the end of subprocess run with p.wait():

import os
import subprocess
p = subprocess.Popen(['%s/bin/scrapy' % os.environ['VIRTUAL_ENV'], '-h'])
p.wait()

Otherwise systemctl may terminate your processes too early.

17.5. subprocess — Subprocess management, The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain That is to say, Popen does the equivalent of:. The Python documentation recommends the use of Popen in advanced cases, when other methods such like subprocess.call cannot fulfill our needs. This method allows for the execution of a program as a child process.

You must specify the current working directory on your call to Popen to ensure that scrapy is executed from the right directory.

17.5. subprocess — Subprocess management, subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False, but does not suffer from this vulnerability; see the Note in the Popen constructor  Created on 2011-07-04 22:03 by vstinner, last changed 2011-07-06 08:50 by vstinner.This issue is now closed.

17.1. subprocess — Subprocess management, The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain That is to say, Popen does the equivalent of:. Messages (9) msg199976 - Author: Peter Graham (peter0) Date: 2013-10-15 05:43; On Windows, subprocess.Popen requires the executable name and working directory to be ascii.

17.1. subprocess — Subprocess management, The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain That is to say, Popen does the equivalent of:. gungor@gungors-mint ~/Desktop $ python popen_timeout.py ('Gungor 0 Gungor 1 Gungor 2 Gungor 3 Gungor 4 Gungor 5 Gungor 6 Gungor 7 Gungor 8 Gungor 9 ', '') In this example, I ran popen_timeout.py with 25 seconds timeout on an external program (test.py) which runs for 20 seconds and outputs lines of strings to the standard output which

python Popen does not capture program's stdout. Unsure why , Since the buffer is not flushed after the printf() statements, a workaround I've found is to use stdbuf -o0 , forcing the application in question to  Subprocess Overview For a long time I have been using os.system() when dealing with system administration tasks in Python. The main reason for that, was that I thought that was the simplest way of running Linux commands. In the official python documentation we can read that subprocess should be used for accessing system commands. The […]

Comments
  • I have the same problem, but in my case your solution does not work, as it is a third-party lib (pygatt) which calls hcitool or gatttool.
  • Could you move your answer from an edit in your question to a proper, accepted response?