Unable to reliably open subprocesses with Popen() -- OSError [Errno 14] Bad Address

I am attempting to open multiple sub-processes--each running the same pre-compiled binary, but operating on files in unique directories--under Python (2.7) using subprocess32.Popen(). Most of the time things work fine, but all too often I get an OSError [Errno 14] Bad Address. Here is the code:

self.gld_stdout_file = open('stdout', 'w+')
self.gld_stderr_file = open('stderr', 'w+')
...
subprocess.Popen(string.join(gld_open_str, " "), shell=True, stderr=self.gld_stderr_file,
                 stdout=self.gld_stdout_file, bufsize=-1, close_fds=ON_POSIX,
                 env={'TEMP':temp_path})

This error occurs about 5-10% of the attempts to use Popen(), while other Popen() calls in the same loop work just fine. Looking around, it seems this could come from an error in lower-level socket calls, that I am not directly interfacing. (e.g. Here or here)

Any ideas on why am I getting this error?

And more importantly:

How I might fix it?

For reference, we are using subprocess32, which supposedly offers improved stability with multiple subprocess calls. Also, if relevant, the entire scheme is wrapped up into a larger MPI-based HPC parallel call, such that multiple compute nodes are attempting to do the same thing at the same time. Fearing there might be some conflict or filesystem challenge with multiple attempts to execute the same file, we are already copying the binary to each of these nodes before execution.

Also, I see the same problem using shell=False as in:

subprocess.Popen(gld_open_list, shell=False, stderr=self.gld_stderr_file,
                 stdout=self.gld_stdout_file, bufsize=-1, close_fds=ON_POSIX,
                 env={'TEMP':temp_path})

That's a bug on python2.6 and fixed on 2.7.

IOError raised due to a read system call returning EINTR from within these methods(read(), readline(), readlines()).

see: https://github.com/python/cpython/commit/736ca00270db72fefa5fb278982c96e5e7139d72

and

https://github.com/python/cpython/blob/2.6/Objects/fileobject.c#L1362.

Upgrade your python and then everything is fun.

importing mxnet causing subprocess to crash · Issue #13875 , importing mxnet causing subprocess to crash #13875. Closed. dabraude opened this issue on Jan 14, 2019 · 23 comments OSError: [Errno 14] Bad address: 'ls' Run : pip install -U mxnet --pre I'd like to note that the website CI pipeline has been intermittently failing with subprocess errors ever since the MKLDNN  706 - Issue #11432: A bug was introduced in subprocess.Popen on posix systems with: 707 3.2.0 where the stdout or stderr file descriptor being the same as the stdin: 708 file descriptor would raise an exception. webbrowser.open would fail. fixed. 709: 710 - Issue #9856: Change object.__format__ with a non-empty format string: 711 to be a

It seems to happen with Windows. No problems with MacOS.

I have found a hacky but practical solution:

while True:
    try:
        proc = subprocess.Popen(cmd, 
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1)
        (output,err)=proc.communicate()
        break
    except:
        log_msg("Exception in spawning subprocess. Retrying ...") 

OSError: [Errno 14] Bad address on website CI tests · Issue #13831 , OSError: [Errno 14] Bad address on website CI tests #13831. Closed 14, 2019. importing mxnet causing subprocess to crash #13875. Closed. Note. This method is intended to aid in implementing protocols that want to delay choosing which data to send until the last moment. E.g., suppose you’re working on an implemention of a remote display server like VNC, and the network connection is currently backed up so that if you call send_all() now then it will sit for 0.5 seconds before actually sending anything.

Changelog, to not return the part before 2nd @ as valid email address. bpo-30730: Prevent environment variables injection in subprocess on Windows. bpo-29110: Fix file object leak in aifc.open() when file is given as a filesystem path and is set to None if the creation of the object failed, instead of raising an OSError exception. Issue #15663: Tcl/Tk 8.5.14 is now included with the OS X 10.6+ 64-/32-bit installer. It is no longer necessary to install a third-party version of Tcl/Tk 8.5 to work around the problems in the Apple-supplied Tcl/Tk 8.5 shipped in OS X 10.6 and later releases.

programs/buck_tool.py - buck, import errno from subprocess import CalledProcessError, check_output. from ng import be able to close() the temporary file without deleting it. So, set except OSError as e: n.b. 'buck --help clean' is *not* currently a help command FATAL_DISK_FULL = 14 with open(os.devnull, "w") as devnull: This is a bad. """Open an LZMA-compressed file in binary or text mode. - filename can be either an actual file name (given as a str or bytes object), - in which case the named file is opened, or it can be an existing file object

Changelog, Issue #21291: subprocess's Popen.wait() is now thread safe so that multiple threads may TextIOWrapper (and hence the open() builtin) now uses the internal codec Issue #20113: os.readv() and os.writev() now raise an OSError exception on error Issue #19088: Fixed incorrect caching of the copyreg module in object. Popen is no able to process a command that runs in shell; OSError: [Errno 2] No such file or directory: '39' Unable to open up prover9/Mace4;

+++++++++++ Python News +++++++++++ What's New in Python , What's New in Python 2.7.14 release candidate 1? Issue #28449: tarfile.open() with mode "r" or "r:" now tries to open a tar file Issue #1703178: Fix the ability to pass the --link-objects option to the distutils Issue #26083: Workaround a subprocess bug that raises an incorrect _cmp, catch IOError as well as os.error. cloud-init git related things importing ubuntu branches. the file import-ubuntu-to-trunk shows the script that I used to create the ubuntu branches so that we can see ubuntu/precise, ubuntu/trusty branches here.

Comments
  • My testing was on 2.7 and still experienced the problem.
  • My testing was under Linux (Red Hat Enterprise Server 6.3)