admin管理员组

文章数量:1287290

I have:

  • Apple Macbook Pro M4 with Silicon Chip
  • Docker Desktop Installed
  • Docker Resources settings - CPU limit: 12, Memory Limit: 15GB, Swap: 4GB

I need:

  • A multinode spark cluster running on my macbook
  • 1 driver, 2 workers
  • Be able to run spark scala applications from intellij on this cluster
  • proper networking established
  • proper volumes attached between macbook and docker containers
  • the jobs shouldn't just keep hanging without finding necessary resources
  • be able to access application(spark ui, spark history server) urls through google chrome on my mac

I have tried every possible thing from simple to complex docker-compose.yml configuration. Multiple different images but I am just not able to get it working.

My simple spark scala application is to read a json file from local mac, do a filter and write back to mac.

Mostly the application never starts because it is unable to find enough resources. Then I increase the driver memory, driver cpu, executor memory , executor cpu but never seem to get it working.

Can anyone try this on their end and tell me if this works at all?

Your help is much appreciated! Thanks in advance

本文标签： scalaRunning Spark Application on Docker Containers Through Intellij IDEA CEStack Overflow

版权声明：本文标题：scala - Running Spark Application on Docker Containers Through Intellij IDEA CE - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1741307312a2371445.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。