admin管理员组

文章数量:1194000

Is it possible to set field on the UDF instance in driver and use it by the executors when call() is invoked?

public class SomeUDF implements UDF2<String, String, String> {
    private String val = "foo";

    public void init(String st){
        val = st;
    }
    @Override
    public String call(String a, String b) {
        return val;
    }
}

py spark:

jvm_udf = spark._jvm.example.demo.SomeUDF()
jvm_udf.init("bla")

spark.udf.registerJavaFunction("foo", jvm_udf.getClass().getName(), T.StringType())
df_single_row.withColumn("val", expr(f"foo('a','b')"))

output is:

"foo"

I want:

"bla"

本文标签： Set field of java UDF instance in spark driver and use in executorsStack Overflow

版权声明：本文标题：Set field of java UDF instance in spark driver and use in executors - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1738432100a2086467.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。