How to iterate a big list to make it smaller for a REST call using Java streams?

Related searches

I have logic something like this to process in chunks.

List<String> bigList = getList(); // Returns a big list of 1000 recs
int startIndex=0;
boolean entireListNotProcessed=false;
while(entireListNotProcessed) {
   int endIndex = startIndex + 100;
   if(endIndex > myList.size()-1) {
      endIndex=myList.size()-1;
   }
   List<String> subList= bigList.subList(startIndex, endIndex);
   //call Rest API with subList and update entireListNotProcessed flag...
}

Is there a better way of doing this iteration using java streams?

You can do similar to this answer, by creating range with stepping and then subList:

int step = 100;
IntStream
  .iterate(0, o -> o < bigList.size(), o -> o + step)
  .mapToObj(i -> bigList.subList(i, Math.min(i + step, bigList.size()))
  .forEach(subList -> callRestApi(subList));

Or you can extract method:

private static <T>  Stream<List<T>> partition(List<T> list, int step) {
    return IntStream
        .iterate(0, o -> o < list.size(), o -> o + step)
        .mapToObj(i -> list
            .subList(i, Math.min(i + step, list.size()))
        );
}

and then

partition(bigList, 100).forEach(subList -> callRestApi(subList));

Partition a List in Java, How to Partition a List using Guava or Apache Commons REST. The guides on building REST APIs with Spring Learn how to shuffle various collections in Java. List<Integer> lastPartition = subSets.get(2); <Integer> newArrayList(7, 8); List<Integer> firstPartition = subSets.iterator().next();. I'm relatively new to REST calls and am probably a beginner to intermediate in C#. I am trying to make a REST call to Qualys to obtain appliance information. My problem is that when I do a foreach loop to iterate through the List, the console app fails by just exiting after I input my credentials.

You could use ListUtils from apache commons to do the splitting for you (if you really need to since 1000 is not that muuch)

List<List<Integer>> output = ListUtils.partition(largeList, targetSize);

Filtering a Java Collection by a List, Learn different ways to filter a Collection in Java based on the values of Filtering a Collection by a List is a common business logic scenario. We'll begin with the most classic syntax, a for-each loop. If we call the size of the employee list n, then nameFilter will be on Using Streams and List#contains. 5 Checking list of enums with Java 8 Stream API Feb 11 4 Java Stream: Filter with multiple ranges Jan 29 4 How to iterate a big list to make it smaller for a REST call using Java streams?

You can use Guava's Iterables.partition or Lists.partition

Lists.partition(bigList, 100).forEach(subList -> callRestApi(subList));

Faster parallel processing in Java using Streams and a fixed-batch , I can't remember a single for-loop that I had to write with Java 8 (well, except next big thing (seamless multicore computing), concise lambda syntax with A bit of explanation is present in this message by Paul Sandoz on the lambda-dev list. framework decides to consume a spliterator in full, it can make a single call to� PayPal Java SDK Complete Example – How to Invoke PayPal Authorization REST API using Java Client? How to Iterate Through Map and List in Java? Example attached (Total 5 Different Ways) How to Iterate through LinkedList Instance in Java? In Java8 – How to Convert Array to Stream using Arrays.stream() and Stream.of() Operations

You can batch your processing into chucks of a fixed size (or more advance dynamically computed batch size).

 final List<Integer> numbers = Arrays.asList(1,2,3,4,5,6,7,); // any range of items.
    final int batchSize = 3;
    final AtomicInteger counter = new AtomicInteger();

    final Collection<List<Integer>> result = numbers.stream()
        .collect(Collectors.groupingBy(it -> counter.getAndIncrement() / batchSize))
        .values();

You have now a collection of List of N element each (N=3)

Java 8 streams API and parallelism, Prior to the feature updates made available in Java 8, running bulky processing due to the inherent explicit iteration that was being used. of fetching out the biggest trade from a set collection using Java 7: Now that we have all the pieces in play, a client can invoke the filterTrades method as follows:. Few Java 8 examples to execute streams in parallel. 1. BaseStream.parallel() A simple parallel example to print 1 to 10.

10 Examples of forEach() method in Java 8, From Java 8 onward, you can iterate over a List or any Collection without using any If you not started with Java 8 yet then you should make it one of your new year once calling forEach() method on stream, you cannot call another method. In Python, list is a type of container in Data Structures, which is used to store multiple data at the same time. Unlike Sets, the list in Python are ordered and have a definite count. There are multiple ways to iterate over a list in Python. Let’s see all different ways to iterate over a list in Python, and a performance comparison between them.

We created an infinite stream using an iterate() method. Then we called a limit() transformation and a collect() terminal operation. Then in our resulting List, we will have first 10 elements of an infinite sequence due to a laziness of a Stream. 4. Infinite Stream of a Custom Type of Elements

Java Streams Improvements In Java 9. Java 8 brought Java streams to the world. However, the following version of the language also contributed to the feature. So, we’ll now give a brief overview of the improvements that Java 9 brought to the Streams API. Let’s do it. takeWhile. The takeWhile method is one of the new additions to the Streams

Comments
  • "Returns a big list of 1000 recs" - 1000 elements is like nothing. Big would be in milions of elements.
  • Anyway, your solution looks fine to me.
  • Why don’t you use a straight-forward counting loop? for(int startIndex = 0, endIndex, size = bigList.size(); startIndex < size; startIndex = endIndex) { List<String> subList = bigList.subList(startIndex, endIndex = Math.min(startIndex + 100, size)); /* processing of subList */ }. The entireListNotProcessed flag is obsolete.
  • I could do that. I'm just wondering if there is any better way than traditional for loops.
  • Instead of (i + step >= bigList.size()) ? bigList.size() : i + step, I’d use Math.min(i + step, bigList.size()).
  • A good proof that "using the Stream API" does not imply "better".
  • Would be happy to have some insights
  • a) an external counter is defeating the intent of the Stream API, as it makes the code rely on the processing order, which is not guaranteed. b) AtomicInteger adds the expenses of thread safe updates, while the code is not thread safe due to the dependency on the processing order mentioned in a. c) subList is a copying free operation, as it creates a view to a part of the original List. In contrast, your solution creates new lists containing all elements, plus an additional HashMap holding all of them (all lists being fully materialized in memory even before the processing starts).