1. 程式人生 > >pyspark AnalysisException: u'Text data source supports only a single column, and you have 5 columns

pyspark AnalysisException: u'Text data source supports only a single column, and you have 5 columns

pyspark dataframe 多列資料當我們想儲存為text格式的時候:

df.coalesce(1).write.format("text").option("header", "false").mode("append").save("<path>")

我們發現報錯:

pyspark.sql.utils.AnalysisException: 'Text data source supports only a single column,

解決辦法就是將多列拼接為一列string字串

dataFrameWithOnlyOneColumn = dataFrame.select(concat(*dataFrame.columns).alias('data'))
dataFrameWithOnlyOneColumn.coalesce(1).write.format("text").option("header", "false").mode("append").save("<path>")