Why element order is not preserved when converting a list of pairs to Map in Scala?
spark rdd get value by key
scala map
scala convert map to listmap
spark pair rdd
list.map scala
reduce by key spark
spark groupbykey
Example:
val occ = List(('l', 2), ('n', 1), ('r', 1), ('u', 2), ('x', 1)) occ.toMap // Map(x -> 1, n -> 1, u -> 2, l -> 2, r -> 1)
The elements are no longer sorted in alphabetical order. Why does this happen?
Map
s are not sorted and the keys may be in any order depending on implementation.
However a ListMap
preserves the order of addition. You can build a ListMap
using the normal Map
operations, or you can create one from a List
like this:
ListMap(occ:_*)
Immutable maps have inconsistent behaviour on update · Issue , Why element order is not preserved when converting a list of pairs to Map in Scala? - scala. A Map is an Iterable consisting of pairs of keys and values (also named mappings or associations).Scala’s Predef object offers an implicit conversion that lets you write key -> value as an alternate syntax for the pair (key, value).
Edit: there's a LinkedMap
coming in 2.13 with different trade-offs.
Edit: the question is really that you might expect list.toMap
to produce a ListMap
. Why doesn't it?
The doc says "inserting or removing entries, are also O(n), which makes this collection suitable only for a small number of elements."
So that's another sensitivity to size. Often you create a collection as pairs resulting from other operations and you just want to dump them into a map, often a large one.
--
There is sensitivity to size due to specialization. There are custom maps for the smallest sizes. This isn't guaranteed, but is confusing if you notice the pattern.
scala $ ~/scala-2.12.6/bin/scala -Dscala.repl.info Welcome to Scala 2.12.6 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144). Type in expressions for evaluation. Or try :help. [info] started at Wed Jun 20 00:17:33 PDT 2018 scala 2.12.6> val m = List(1->10,2->20,3->30,4->40).toMap m: scala.collection.immutable.Map[Int,Int] = Map(1 -> 10, 2 -> 20, 3 -> 30, 4 -> 40) scala 2.12.6> val m = List(1->10,2->20,3->30,4->40,5->50).toMap m: scala.collection.immutable.Map[Int,Int] = Map(5 -> 50, 1 -> 10, 2 -> 20, 3 -> 30, 4 -> 40) scala 2.12.6> :quit scala $ ~/scala-2.13.0-M4/bin/scala -Dscala.repl.info [info] started at Wed Jun 20 00:18:41 PDT 2018 Welcome to Scala 2.13.0-M4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144). Type in expressions for evaluation. Or try :help. scala 2.13.0-M4> val m = List(1->10,2->20,3->30,4->40).toMap m: scala.collection.immutable.Map[Int,Int] = ChampHashMap(1 -> 10, 2 -> 20, 3 -> 30, 4 -> 40) scala 2.13.0-M4> val m = List(1->10,2->20,3->30,4->40,5->50).toMap m: scala.collection.immutable.Map[Int,Int] = ChampHashMap(5 -> 50, 1 -> 10, 2 -> 20, 3 -> 30, 4 -> 40)
Learning Scala Programming: Object-oriented programming meets , I would be happy to work on this change after confirmation of the semantics required List map iterators and; traversal methods visit key-value pairs in the order whey were first inserted. I personally think that the scaladoc should not include the ordering Existing key elements must not be preserved. Program to convert Java list to Set in Scala Scala | reduce() Function The reduce() method is a higher-order function that takes all the elements in a collection (Array, List, etc) and combines them using a binary operation to produce a single value.
You can use ListMap instead of Map, it keeps items in the order they were inserted. To convert List
to ListMap
:
val occ = List(('l', 2), ('n', 1), ('r', 1), ('u', 2), ('x', 1)) ListMap(occ: _*)
In case you don't know what _*
means - repeated parameters
scala.collection.immutable.ListMap, Almost every collection has higher order functions to operate with. In the majority of situations, we come up with Map, Set, List, ArrayBuffer, or a Vector. is of no importance, and ListMap when we want to store the key-value pairs in a sequence. elements, so to remove duplicates we may choose to use a Set, or convert * This function takes the first two elements of the list `trees` and combines * them into a single `Fork` node. This node is then added back into the * remaining elements of `trees` at a position such that the ordering by weights * is preserved. * * If `trees` is a list of less than two elements, that list should be returned * unchanged. */
4. Working with Key/Value Pairs - Learning Spark [Book], [use case] Returns a new immutable list map containing the elements from A syntactic sugar for out of order folding. The order of the elements is preserved. Returns a new map with all key/value pairs for which the predicate p returns true . and must not change the result (e.g., Nil for list concatenation, 0 for addition, Scala Lists are quite similar to arrays which means, all the elements of a list have the same type but there are two important differences. First, lists are immutable, which means elements of a list cannot be changed by assignment. Second, lists represent a linked list whereas arrays are flat. The type of a list that has elements of type T is
How to choose a collection class in Scala, We can do this by running a map() function that returns key/value pairs. An implicit conversion on RDDs of tuples exists to provide the additional key/value functions. When creating a pair RDD from an in-memory collection in Scala and Python, as the aggregation function (appending to a list) does not save any space. a pair of lists: the list of all elements which satisfy p and the list of all elements which do not. The relative order of the elements in the sub-lists is the same as in the original list. The relative order of the elements in the sub-lists is the same as in the original list.
Scala List class examples: range, fill, tabulate, appending, foreach , A map contains a collection of key/value pairs, like a Java Map , Ruby Hash , or Python dictionary. A set is a collection that contains no duplicate elements. The documentation states, “If you plan to convert the buffer to a list, use Map classes, a SortedMap trait to keep elements in sorted order by key, Iterating over a Scala Map - Summary. In summary, I hope these examples of iterating a Scala Map have been helpful. As you can see, there are many ways to loop over a Map, using for, foreach, tuples, and key/value approaches.
Comments
- @CannedMoose if we use sorted map then only it will sort the map by key it's not the case always.
- Possible duplicate of Scala Map implementation keeping entries in insertion order?
- The question asked about a 'toMap' method in List, I see it is different with ListMap. And I do not find an answer in the question recommended.
- I know there are ways to preserve the order. I am just curious why converting to map do not preserve the order. What is reason/concern in implementation the "toMap" method.
- @EmmaLiu
toMap
just converts theList
to aMap
. It is the implementation ofMap
that matters. This is typically done with a table indexed by a hash of the key value (henceHashMap
). The order of the keys will be the order of the hashed values, not the order of the keys themselves. - @EmmaLiu: A map has no order. It doesn't preserve the order because there is no order to preserve.
- @Tim Your explanation answers my question. Thanks a lot.
- Yes, I do noticed this. For very small maps, the conversion keeps the element in order. But for slightly larger map, the order is not longer preserved, but determined.