java.lang.NoSuchFieldError: DECIMAL128 mongoDB spark

spark mongodb java example
how to read data from mongodb in spark python
spark read from mongodb
spark mongodb query
mongodb spark connector authentication
databricks spark mongodb connector
could not initialize class com/mongodb spark config readconfig
bson spark

I'm writing a spark job using pyspark; I should only read from mongoDB collection and print the content on the screen; the code is the following:

import pyspark
from pyspark.sql import SparkSession

my_spark = SparkSession.builder.appName("myApp").config("spark.mongodb.input.uri", "mongodb://127.0.0.1/marco.weather_test").config("spark.mongodb.output.uri", "mongodb://127.0.0.1/marco.out").getOrCreate()

df = my_spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri", "mongodb://127.0.0.1/marco.weather_test").load()

#df.printSchema()
df.show()

the problem is that when I want to print the schema the job works, but when I want to print the content of the dataFrame with show() function I get back the error:

#java.lang.NoSuchFieldError: DECIMAL128

the command I use is:

#bin/spark-submit --packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.3 /home/erca/Scrivania/proveTesi/test_batch.py

I got the same error due to wrong jar is used for mongo-java-driver. The NoSuchFieldError is raised from

org.bson.BsonType.Decimal128

, and the field Decimal128 is added in class BsonType after mongo-java-driver 3.4。While the

org.mongodb.spark:mongo-spark-connector_2.11:2.2.4

contains mongo-java-driver 3.6.2, a existing jar "mongo-java-driver" with version 3.2.1 is located in driverExtraClassPath.

Just start the spark-shell with verbose:

spark-shell --verbose

i.e. the output:

***
***
Parsed arguments:
  master                  local[*]
  deployMode              null
  executorMemory          1024m
  executorCores           1
  totalExecutorCores      null
  propertiesFile          /etc/ecm/spark-conf/spark-defaults.conf
  driverMemory            1024m
  driverCores             null
  driverExtraClassPath    /opt/apps/extra-jars/*
  driverExtraLibraryPath  /usr/lib/hadoop-current/lib/native
***
***

and pay attention to the driverExtraClassPath, driverExtraLibraryPath.

Check these paths, and remove mongo-java-driver if it exists within these paths.

java.lang.NoSuchFieldError：DECIMAL128 mongoDB spark, Pass a JavaSparkContext to MongoSpark.load() to read from MongoDB into a JavaMongoRDD . package com.mongodb.spark_examples; import java.util. Returns an instance of decimal.Decimal for this Decimal128. bson.decimal128.create_decimal128_context ¶ Returns an instance of decimal.Context appropriate for working with IEEE-754 128-bit decimal floating point values.

Importing the library explicitly might solve the issue or give you a better error message for debugging.

import org.bson.types.Decimal128

python, Not able to Show/Write from spark DF read using mongo spark at java.util.concurrent. runWorker(ThreadPoolExecutor.java:1142). Finally Able to debug and fix the issue. The Issue was with the installation as one of the data nodes are having older version of parquet Jars(5.2 cdh distribution).

To BSON jar I a dded a version major and it worked

(In gradle)

compile group: 'org.mongodb', name: 'bson', version: '3.10.2',  'force': "true"

mongo-spark/1-sparkSQL.md at master · mongodb/mongo-spark , java.lang.Object. org.bson.types.Decimal128. All Implemented Interfaces: Serializable A constant holding a Not-a-Number (NaN) value of type Decimal128 . The Mongo Spark Connector provides the com.mongodb.spark.sql.DefaultSource class that creates Datasets from MongoDB. However, the easiest way to create a DataFrame is by using the MongoSpark helper: val df = MongoSpark .load(sparkSession) // Uses the SparkConf df.printSchema()

Read from MongoDB, Я подозреваю, что у вас есть несколько версий mongo-java-driver jar в вашем classpath. java.lang.NoSuchFieldError: ACKNOWLEDGED at com.mongodb. java.lang.NoSuchFieldError: DECIMAL128 mongoDB Искра. Я пишу задание spark с использованием pyspark ; я должен только читать из коллекции mongo-spark / src / main / scala / com / mongodb / spark / sql / MapFunctions.scala Find file Copy path rozza Added WriteConfig.extendedBsonTypes setting 807ae92 Jun 6, 2019

[SPARK-172] Not able to Show/Write from spark , Java decimal128. lang. double has a precision of 15 or 16 significant decimal digits. AutoCloseable NoSuchFieldError: DECIMAL128 mongoDB spark. Python Spark Shell. This tutorial uses the pyspark shell, but the code works with self-contained Python applications as well. When starting the pyspark shell, you can specify: the --packages option to download the MongoDB Spark Connector package. The following package is available: mongo-spark-connector_2.11 for use with Scala 2.11.x

Decimal128, Here, you're using your own build of Spark against an older version of Hive than what's in CDH. That might mostly work, but you're seeing the problems in compiling and running vs different versions.