admin管理员组

文章数量:1355522

I'm trying to run Apache Spark using Scala 2.11.12 and Java 21.0.6, but I keep running into an error related to accessing internal fields in the java.util.ArrayList class. Specifically, when I try to load a text file using Spark, I get the following error:

    25/03/29 22:00:19 WARN BlockManager: Putting block broadcast_0 failed due to exception java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @2462cb01.
25/03/29 22:00:19 WARN BlockManager: Block broadcast_0 could not be removed as it was not found on disk or in memory
java.lang.reflect.InaccessibleObjectException: Unable to make field transient java.lang.Object[] java.util.ArrayList.elementData accessible: module java.base does not "opens java.util" to unnamed module @2462cb01
  at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException(AccessibleObject.java:391)
  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:367)
  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:315)
  at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:183)
  at java.base/java.lang.reflect.Field.setAccessible(Field.java:177)
  at .apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:336)
  at .apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:330)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at .apache.spark.util.SizeEstimator$.getClassInfo(SizeEstimator.scala:330)
  at .apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:222)
  at .apache.spark.util.SizeEstimator$$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:201)
  at .apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:69)
  at .apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
  at .apache.spark.util.collection.SizeTracker$class.afterUpdate(SizeTracker.scala:70)
  at .apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31)
  at .apache.spark.storage.memory.DeserializedValuesHolder.storeValue(MemoryStore.scala:665)
  at .apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
  at .apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
  at .apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
  at .apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
  at .apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
  at .apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
  at .apache.spark.storage.BlockManager.putIterator(BlockManager.scala:914)
  at .apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1481)
  at .apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:123)
  at .apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
  at .apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
  at .apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
  at .apache.spark.SparkContext.broadcast(SparkContext.scala:1489)
  at .apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1035)
  at .apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1027)
  at .apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at .apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at .apache.spark.SparkContext.withScope(SparkContext.scala:699)
  at .apache.spark.SparkContext.hadoopFile(SparkContext.scala:1027)
  at .apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:830)
  at .apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:828)
  at .apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at .apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at .apache.spark.SparkContext.withScope(SparkContext.scala:699)
  at .apache.spark.SparkContext.textFile(SparkContext.scala:828)
  ... 45 elided

Here is the code I am using to load a text file:

val lines = sc.textFile("/root/input/file1.txt")

Here is the file1.txt:

Hello Spark Wordcount!
Hello everybody else!

Environment:

Spark Version: 2.4.5

Scala Version: 2.11.12

Java Version: 21.0.6 (OpenJDK 64-Bit Server VM)

What I've tried:

Added JVM Options: I've tried adding the following JVM options to bypass the module system restrictions, but it didn't solve the issue:

--conf "spark.driver.extraJavaOptions=--add-opens java.base/java.util=ALL-UNNAMED" --conf "spark.executor.extraJavaOptions=--add-opens java.base/java.util=ALL-UNNAMED"

本文标签： InaccessibleObjectException when using Spark with Java 21 and Scala 21112Stack Overflow

版权声明：本文标题：InaccessibleObjectException when using Spark with Java 21 and Scala 2.11.12 - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1743999477a2573595.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。