dataframe - Extract Column values from data frame and pass into SQL pyspark where Clause - Stack Overflow

admin管理员组

文章数量:1387414

I am trying a scenario to extract data from backend into Data frame and just retrieve Column1 list values example "ID" column from that table and pass that list of ID values into SQL query for another data extraction. Tried the below line and it gives me array of response like the one i pasted here:

row_list = df.select('Column_header').collect()

• Response:

  [Row(Column_header='Value1'), Row(Column_header='Value2')........]

What I would like to extract is like this:

[val1,val2,val3.....]

Tried with RDD and map, but still getting syntax errors even on using correct format. Need help here.

Tried RDD, flatmap etc no syntax works.

I am trying a scenario to extract data from backend into Data frame and just retrieve Column1 list values example "ID" column from that table and pass that list of ID values into SQL query for another data extraction. Tried the below line and it gives me array of response like the one i pasted here:

row_list = df.select('Column_header').collect()

• Response:

  [Row(Column_header='Value1'), Row(Column_header='Value2')........]

What I would like to extract is like this:

[val1,val2,val3.....]

Tried with RDD and map, but still getting syntax errors even on using correct format. Need help here.

Tried RDD, flatmap etc no syntax works.

• dataframe
• pyspark
• aws-glue

Share Improve this question Follow edited Mar 17 at 9:36 Yash Mehta 2,00644 gold badges1212 silver badges2121 bronze badges asked Mar 17 at 9:09 Kirthi ShreeKirthi Shree 322 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

There's an extra step missing:

row_list = df.select('Column_header').collect()
result = [row['Column_header'] for row in row_list]

本文标签： dataframeExtract Column values from data frame and pass into SQL pyspark where ClauseStack Overflow