Why are my server sent events arriving as a batch?

sse events
eventsource

I have a Java 8 / Spring4-based web application that is reporting the progress of a long-running process using Server Sent Events (SSEs) to a browser-based client running some Javascript and updating a progress bar. In my development environment and on our development server, the SSEs arrive in near-real-time at the client. I can see them arriving (along with their timestamps) using Chrome dev tools and the progress bar updates smoothly.

However, when I deploy to our production environment, I observe different behaviour. The events do not arrive at the browser until the long-running process completes. Then they all arrive in a burst (the events all have the timestamps within a few hundred milliseconds of each other according to dev tools). The progress bar is stuck at 0% for the duration and then skips to 100% really quickly. Meanwhile, my server logs tell me the events were generated and sent at regular intervals.

Here's the relevant server side code:

public class LongRunningProcess extends Thread {
    private SseEmitter emitter;
    public LongRunningProcess(SseEmitter emitter) {
        this.emitter = emitter;
    }
    public void run() {
        ...
        // Sample event, representing 10% progress
        SseEventBuilder event = SseEmitter.event();
        event.name("progress");
        event.data("{ \"progress\": 10 }"); // Hand-coded JSON
        emitter.send(event);
        ...
    }
}

@RestController
public class UploadController {
    @GetMapping("/start")
    public SseEmitter start() {
        SseEmitter emitter = new SseEmitter();
        LongRunningProcess process = new LongRunningProcess(emitter);
        process.start();
        return emitter;
    }
}

Here's the relevant client-side Javascript:

EventSource src = new EventSource("https://www.example.com/app/start");
src.addEventListener('progress', function(event) {
    // Process event.data and update progress bar accordingly
});

I believe my code is fairly typical and it works just fine in DEV. However if anyone can see an issue let me know.

The issue could be related to the configuration of our production servers. DEV and PROD are all running the same version of Tomcat. However, some of them are accessed via a load balancer (F5 in out case). Almost all of them are behind a CDN (Akamai in our case). Could there be some part of this setup that causes the SSEs to be buffered (or queued or cached) that might produce what I'm seeing?

Following up on the infrastructure configuration idea, I've observed the following in the response headers. In the development environment, my browser receives:

Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: Keep-Alive
Content-Type: text/event-stream;charset=UTF-8
Keep-Alive: timeout=15, max=99
Pragma: no-cache
Server: Apache
Transfer-Encoding: chunked
Via: 1.1 example.com

This is what I'd expect for an event stream. A chunked response of an unknown content length. In the production environment, my browser receives something different:

Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: keep-alive
Content-Type: text/event-stream;charset=UTF-8
Content-Encoding: gzip
Content-Length: 318
Pragma: no-cache
Vary: Accept-Encoding

Here the returned content has a known length and is compressed. I don't think this should happen for an event stream. It would appear that something is converting my event stream into single file. Any thoughts on how I can figure out what's doing this?

It took a significant amount of investigation to determine that the cause of the issue was the elements in our network path. So the code above is correct and safe to use. If you find SSE buffering you will most likely want to check the configuration of key networking elements.

In my case, it was Akamai as our CDN and the use of an F5 device as a load balancer. Indeed it was the fact that both can introduce buffering that made it quite difficult to diagnose the issue.

Akamai Edge servers buffer event streams by default. This can be disabled through the use of Akamai's advanced metadata and controlled via custom behaviours. At this time, this cannot be controlled directly through Amakai's portal, so you will need to get their engineers to do some of the work for you.

F5 devices appear to default to buffering response data as well. Fortunately, this is quite simple to change and can be done yourself via the device's configuration portal. For the virtual device in question, go to Profile : Services : HTTP and change the configuration of Response Chunking to Preserve (in our case it had defaulted to Selective).

Once I made these changes, I began to receive SSEs in near real-time from our PROD servers (and not just our DEV servers).

A Look at Server-Sent Events - Conectric Networks, What is the difference between server sent events and Websockets in html5? Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion.

Have you tried alternative browsers? I'm trying to debug a similar problem in which SSE works on an iPhone client but not on MacOS/Safari or Firefox.

There may be a work-around for your issue - if the server sends "Connection: close" instead of keep-alive, or even closes the connection itself, the client should re-connect in a few seconds and the server will send the current progress bar event.

I'm guessing that closing the connection will flush whatever buffer is causing the problem.

HTML5 - Server Sent Events, data to the browser and receive data from the browser. SSE connections can only push data to the browser. This can happen in our client if the client things the poll times out but the server has actually sent a message (it just got lost). When the client opens a second poll, the server thinks it has sent some messages so those messages are dropped. Ordering should preserved, but there can be dropped messages.

This is not a solution to this question exactly, but related to SSE, Spring and use of compression.

In my case I had ziplet CompressionFilter configured in my Spring application and it was closing the Http Response and causing SSE to fail. This seems to be related to an open issue in the ziplet project. I disabled the filter and enabled Tomcat compression in application.properties (server.compression.enabled=true) and it solved the SSE issue.

Note that I did not change the default compressionMinSize setting, which may have something to do with SSE traffic not getting compressed and passing through.

WebSockets vs. Server-Sent events/EventSource, To get started, we simply need to specify the URL of the SSE event stream logic is handled for us: the connection is negotiated on our behalf, received data is data format is what allows us to offload the bulk of the work to the browser. resulting packet is sent out over a 155 Mbps link. Ignore flow control and congestion control so A can pump out the segments back to back and continuously. a) There are 2^32 = 4,294,967,296 possible sequence numbers. The sequence number does not increment by one with each segment. Rather, it increments by the number of bytes of data sent.

Using server-sent events, Batch Processing and Server-Sent Events using Spring. Enterprise The source data may arrive in bulks or batches. The application needs to  Then they all arrive in a burst (the events all have the timestamps within a few hundred milliseconds of each other according to dev tools). The progress bar is stuck at 0% for the duration and then skips to 100% really quickly. Meanwhile, my server logs tell me the events were generated and sent at regular intervals.

Browser APIs and Protocols: Server-Sent Events (SSE), To get started, we simply need to specify the URL of the SSE event stream the full received response until the connection is dropped, an SSE connection can a well-defined data format is what allows us to offload the bulk of the work to the  The Windows Task Scheduler can automatically send email at a specific time or in response to a specific event, but its integrated email feature won’t work very well for most users. Instead of using the Task Scheduler’s email feature to send emails, you can use the SendEmail utility.

Batch Processing and Server-Sent Events , Server-sent events allow web servers to push real-time event notifications to the before we've received the entire content from the origin server. containing a batch of events that happened over some earlier period in time. Let’s empirically measure how the throughput and latency is impacted by the number of EH partitions, batch and prefetch sizes. Since it will require quite a few iterations, let’s automate the process via Terraform. For every iteration let’s create a 20 TU Standard Event Hub and an EH-triggered Azure Function.

Comments
  • Did you find a solution this problem? I'm facing the same issue. My investigation led me to the same point. After disabling CompressingFilter which is enabled by default, sse works as expected. I tried bypassing the filter for "text/event-stream" but it doesn't help.
  • Are you using CompressingFilter by any chance?
  • @Mustafa No, I'm not using CompressingFilter. I have not yet found a solution, however I am following up a possible lead at the moment. I'll add some more info if this leads to a solution.
  • I swicthed to tomcat compression and no longer see this issue with SSE.
  • @Mustafa You should document what you did as an answer in case someone else finds that useful.