Why does stdout need explicit flushing when redirected to file?

linux flush stdout
bash flush stdout

The behaviour of printf() seems to depend on the location of stdout.

  1. If stdout is sent to the console, then printf() is line-buffered and is flushed after a newline is printed.
  2. If stdout is redirected to a file, the buffer is not flushed unless fflush() is called.
  3. Moreover, if printf() is used before stdout is redirected to file, subsequent writes (to the file) are line-buffered and are flushed after newline.

When is stdout line-buffered, and when does fflush() need to be called?

Minimal example of each:
void RedirectStdout2File(const char* log_path) {
    int fd = open(log_path, O_RDWR|O_APPEND|O_CREAT,S_IRWXU|S_IRWXG|S_IRWXO);
    dup2(fd,STDOUT_FILENO);
    if (fd != STDOUT_FILENO) close(fd);
}

int main_1(int argc, char* argv[]) {
    /* Case 1: stdout is line-buffered when run from console */
    printf("No redirect; printed immediately\n");
    sleep(10);
}

int main_2a(int argc, char* argv[]) {
    /* Case 2a: stdout is not line-buffered when redirected to file */
    RedirectStdout2File(argv[0]);
    printf("Will not go to file!\n");
    RedirectStdout2File("/dev/null");
}
int main_2b(int argc, char* argv[]) {
    /* Case 2b: flushing stdout does send output to file */
    RedirectStdout2File(argv[0]);
    printf("Will go to file if flushed\n");
    fflush(stdout);
    RedirectStdout2File("/dev/null");
}

int main_3(int argc, char* argv[]) {
    /* Case 3: printf before redirect; printf is line-buffered after */
    printf("Before redirect\n");
    RedirectStdout2File(argv[0]);
    printf("Does go to file!\n");
    RedirectStdout2File("/dev/null");
}

Flushing for stdout is determined by its buffering behaviour. The buffering can be set to three modes: _IOFBF (full buffering: waits until fflush() if possible), _IOLBF (line buffering: newline triggers automatic flush), and _IONBF (direct write always used). "Support for these characteristics is implementation-defined, and may be affected via the setbuf() and setvbuf() functions." [C99:7.19.3.3]

"At program startup, three text streams are predefined and need not be opened explicitly — standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). As initially opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device." [C99:7.19.3.7]

Explanation of observered behaviour

So, what happens is that the implementation does something platform-specific to decide whether stdout is going to be line-buffered. In most libc implementations, this test is done when the stream is first used.

  1. Behaviour #1 is easily explained: when the stream is for an interactive device, it is line-buffered, and the printf() is flushed automatically.
  2. Case #2 is also now expected: when we redirect to a file, the stream is fully buffered and will not be flushed except with fflush(), unless you write gobloads of data to it.
  3. Finally, we understand case #3 too for implementations that only perform the check on the underlying fd once. Because we forced stdout's buffer to be initialised in the first printf(), stdout acquired the line-buffered mode. When we swap out the fd to go to file, it's still line-buffered, so the data is flushed automatically.
Some actual implementations

Each libc has latitude in how it interprets these requirements, since C99 doesn't specify what an "interactive device" is, nor does POSIX's stdio entry extend this (beyond requiring stderr to be open for reading).

  1. Glibc. See filedoalloc.c:L111. Here we use stat() to test if the fd is a tty, and set the buffering mode accordingly. (This is called from fileops.c.) stdout initially has a null buffer, and it's allocated on the first use of the stream based on the characteristics of fd 1.

  2. BSD libc. Very similar, but much cleaner code to follow! See this line in makebuf.c

Why do you want to avoid flushing stdout?, This is typically fixed by explicitly putting a “flush” call in the code, e.g. you may also want to use this when redirecting grep output to a file and  How often does python flush to a file? Why does C's printf format string have both %c and %s? Why does stdout need explicit flushing when redirected to file?

Your are wrongly combining buffered and unbuffered IO functions. Such a combination must be done very carefully especially when the code has to be portable. (and it is bad to write unportable code...) It is certainly best to avoid combining buffered and unbuffered IO on the same file descriptor.

Buffered IO: fprintf(), fopen(), fclose(), freopen()...

Unbuffered IO: write(), open(), close(), dup()...

When you use dup2() to redirect stdout. The function is not aware of the buffer which was filled by fprintf(). So when dup2() closes the old descriptor 1 it does not flush the buffer and the content could be flushed to a different output. In your case 2a it was sent to /dev/null.

The solution

In your case it is best to use freopen() instead of dup2(). This solves all your problems:

  1. It flushes the buffers of the original FILE stream. (case 2a)
  2. It sets the buffering mode according to the newly opened file. (case 3)

Here is the correct implementation of your function:

void RedirectStdout2File(const char* log_path) {
    if(freopen(log_path, "a+", stdout) == NULL) err(EXIT_FAILURE, NULL);
}

Unfortunately with buffered IO you cannot directly set permissions of a newly created file. You have to use other calls to change the permissions or you can use unportable glibc extensions. See the fopen() man page.

Stdout Buffering, stdout in a portable way. In Python 3, it is only useful to flush at each line, even if the output is not a console, but a pipe (e.g. output redirected  The behaviour of printf() seems to depend on the location of stdout. If stdout is sent to the console, then printf() is line-buffered and is flushed after a newline is printed. If stdout is redirected to a file, the buffer is not flushed unless fflush() is called.

You should not close the file descriptor, so remove close(fd) and close stdout_bak_fd if you want the message to be print only in the file.

Issue 11633: Document that print may need explicit flushing, In the UNIX tradition, use of standard input and output is customary for for instance, by redirecting standard output to a file while allowing standard error to print on You can explicitly flush the buffer by calling the following: fflush (stdout​); In  If stdout is redirected to a file, the buffer is not flushed unless fflush() is called. Moreover, if printf() is used before stdout is redirected to file, subsequent writes (to the file) are line-buffered and are flushed after newline. Source: Why does stdout need explicit flushing when redirected to file?

Advanced Linux Programming, As I understand it — the output has an implicit write buffer (as it's to the output too quickly and when the output file/terminal does not manage to to be able to explicitly exit the process and have stdout/stderr flushed before exiting. or redirecting in my shell (either node --trace_gc foo.js > bar.log or node  Hence the inconsistency between the immediate output when your program is writing to the terminal and the delayed output when it is writing to a pipe or file. Programmers that don't want their application's output buffered can either: Ask for an explicit buffer flush when appropriate.

stdout/stderr buffering considerations · Issue #6379 · nodejs/node , When a watched process output stream is redirected to a file, any buffered data in sys.stdout.flush() print "no explicit flush given". Already have an account? You use &1 to reference the value of the file descriptor 1 ( stdout ). So when you use 2>&1 you are basically saying “Redirect the stderr to the same place we are redirecting the stdout ”. And that’s why we can do something like this to redirect both stdout and stderr to the same place: $ cat foo.txt > output.txt 2>&1 $ cat output.txt foo

Circus loses buffered stream data from watched processes · Issue , Source: Why does stdout need explicit flushing when redirected to file? I'm not well versed with C programming, but apparently you need to add  Is there a way to redirect a dos command (regedit) which requires a file as output to redirect the output to stdout instead? I.e. trick it to write to the screen or clipboard and not to a file at all. I do not wish to create and delete a file to extract some basic reg data. July 28, 2009 at 10:22 PM

Comments
  • You should check for errors from all system calls.
  • @Joachim Thanks for your suggestion. I just remove the checking code here to make code more readable. Error checking dose not fix my problem.
  • So I guess none of the functions actually returns an error?
  • Thanks for your suggestion on error checking and leak fixing. But I still don't understand why I need to fflush here. And why printing some message before redirecting can fix the problem.
  • @Patrick Ah, I see. In that case, dupe of this issue: stackoverflow.com/questions/1716296/…
  • Thanks for your answer. But I have "\n" in my code, should it trigger flush?
  • Ok, last answer! fsync, fstat, ftruncate etc are one family of functions (in man section 2); totally unrelated to fopen, fprintf, fflush (section 3). fsync is in the kernel, and knows nothing about libc's stdio buffers, which are in your application.
  • I get it now. close is a system function, it has no knowledge of libc buffer either.