【NLP】文本处理基础操作：停用词，去掉杂乱的词（用nltk），pandas遍历和存储成为文件

程序代码以及说明

利用这段程序完成了把一个csv第一行是情感，第二行是评论的数据去掉乱七八糟的字符和完成小写之后，存到了一个txt文件里面
当然整体的程序可能不是很重要
重要的是里面处理的步骤和特定的语法
下面拆开来讲

import pandas as pd
import numpy as np
from nltk.tokenize import RegexpTokenizer

FILE_PATH = 'E:data1_movie.csv'
df = pd.read_csv(FILE_PATH)
tokenizer = RegexpTokenizer(r'w+')
stopwords = [['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers', 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', 'couldn', 'didn', 'doesn', 'hadn', 'hasn', 'haven', 'isn', 'ma', 'mightn', 'mustn', 'needn', 'shan', 'shouldn', 'wasn', 'weren', 'won', 'wouldn']]
for w in ['!',',','.','?','','s','n','t']:
    stopwords.append(w)
for index,row in df.iterrows():
    print(index)
    total_num = 400
    if (index < total_num) and (type(df.iloc[index,1]) == type('1')):
        # deal with the text
        sentence = []
        for word in tokenizer.tokenize(df.iloc[index,1]):
            if len(word) != 1 and (word.lower() not in stopwords):
                sentence.append(word.lower())
        df.iloc[index,1] = ' '.join(sentence)
        # deal with the label
        if df.iloc[index,0]>=7:
            df.iloc[index,0]=1
        elif df.iloc[index,0]<=4:
            df.iloc[index,0]=0
        else:
            df.iloc[index,0]= np.nan
    else:
        df.iloc[index,0] = np.nan
# delete the one with no distinct tendency in emotion
df = df.dropna(axis=0,how='any')
df.to_csv('E:data1_movie.txt', sep='t', index=False,header = False)

遍历dataframe

for index,row in df.iterrows():
    print(df.iloc[index,1])

这样就可以完成遍历了，df.iloc[index,i]表示的是第index样本的第i个属性

完成去掉杂乱词和小写

from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r'w+')
stopwords = [['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers', 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', 'couldn', 'didn', 'doesn', 'hadn', 'hasn', 'haven', 'isn', 'ma', 'mightn', 'mustn', 'needn', 'shan', 'shouldn', 'wasn', 'weren', 'won', 'wouldn']]
for word in tokenizer.tokenize(df.iloc[index,1]):
            if len(word) != 1 and (word.lower() not in stopwords):
                sentence.append(word.lower())

设定好停用词，建立好nltk的tokenizer，进行遍历，并且用lower变成小写

pandas存为文件

df.to_csv('E:data1_movie.txt', sep='t', index=False,header = False)

这样就ok了，存的txt

用dropna去除不想要的数据的小技巧

df.iloc[index,0]= np.nan
# delete the one with no distinct tendency in emotion
df = df.dropna(axis=0,how='any')

我在遍历的时候，把我不想要的，不符合要求的样本都变成了np.nan，之后灵活的使用dropna将这些东西都去除了

最后

以上就是孤独往事为你收集整理的【NLP】文本处理基础操作：停用词，去掉杂乱的词（用nltk），pandas遍历和存储成为txt文件程序代码以及说明遍历dataframe完成去掉杂乱词和小写pandas存为文件用dropna去除不想要的数据的小技巧的全部内容，希望文章能够帮你解决【NLP】文本处理基础操作：停用词，去掉杂乱的词（用nltk），pandas遍历和存储成为txt文件程序代码以及说明遍历dataframe完成去掉杂乱词和小写pandas存为文件用dropna去除不想要的数据的小技巧所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。