達觀杯文字智慧處理挑戰賽
筆者嘗試了一下達觀杯的比賽,程式碼如下圖所示:
import pandas as pd
import imp
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import CountVectorizer
df_test = pd.read_csv(r'testset.csv')
df_train = pd.read_csv(r'trainset.csv')
df_train.drop(columns=['article','id'],inplace=True)
df_test.drop(columns=['article'],inplace=True)
vectorizer = CountVectorizer(ngram_range=(1, 2),min_df=3,max_df=0.9,max_features=10000)
vectorizer.fit(df_train['word_seg'])
x_train =vectorizer.transform(df_train['word_seg'])
x_test =vectorizer.transform(df_test['word_seg'])
y_train =df_train['class']-1
lg = LogisticRegression(C=4,dual=True)
lg.fit(x_train,y_train)
y_test = lg.predict(x_test)
df_test['class'] = y_test.tolist()
df_test['class'] = df_test['class'] + 1
df_result =df_test.loc[:,['id','class']]
df_result.to_csv('./result.csv',index=False)
print("完成")
執行結果是:
pandas.errors.ParserError: Error tokenizing data. C error: out of memory
這是因為我的電腦記憶體太小,導致了記憶體溢位,因此換一臺電腦就可以得到最終得分為72分的答案了。筆者最終排名位於全國前300名,算是一個個人感覺還不錯的成績了。 ---------------------