python如何读取数据框中的数据,如何在Python数据框中读取大块数据？

I want to read the file f in chunks to a dataframe. Here is part of a code that I used.

for i in range(0, maxline, chunksize):

df = pandas.read_csv(f,sep=',', nrows=chunksize, skiprows=i)

df.to_sql(member, engine, if_exists='append',index= False, index_label=None, chunksize=chunksize)

I get the error:

pandas.io.common.EmptyDataError: No columns to parse from file

The code works only when the chunksize >= maxline (which is total lines in file f). However, in my case, the chunksize<=maxline.

Please advise the fix.

解决方案

I think it is better to use the parameter chunksize in read_csv. Also, use concat with the parameter ignore_index, because of the need to avoid duplicates in index:

chunksize = 5

TextFileReader = pd.read_csv(f, chunksize=chunksize)

df = pd.concat(TextFileReader, ignore_index=True)

See pandas docs.