Q: Pandas ignoring headers while reading large (2GB) csv I'm trying to read a rather large CSV (2 GB) with pandas to do some datatype manipulation and joining with other dataframes I have already loaded before. As I want to be a little careful with memory I decided to read the it in chunks. For the purpose of the questions here is an extract of my CSV layout with dummy data (cant really share the real data, sorry!):
institution_id,person_id,first_name,last_name,confidence,institution_name
1141414141,4141414141,JOHN,SMITH,0.7,TEMP PLACE TOWN
10123131114,4141414141,JOHN,SMITH,0.7,TEMP PLACE CITY
1003131313188,4141414141,JOHN,SMITH,0.7,"TEMP PLACE,TOWN"
18613131314,1473131313,JOHN,SMITH,0.7,OTHER TEMP PLACE
192213131313152,1234242383,JANE,SMITH,0.7,"OTHER TEMP INC, LLC"

My pandas code to read the files:
inst_map = pd.read_csv("data/hugefile.csv",
                       engine="python",
                       chunksize=1000000,
                       index_col=False)

print("processing institution chunks")

chunk_list = []  # append each chunk df here
for chunk in inst_map:
    # perform data filtering
    chunk['person_id'] = chunk['person_id'].progress_apply(zip_check)
    chunk['institution_id'] = chunk['institution_id'].progress_apply(zip_check)

    # Once the data filtering is done, append the chunk to list
    chunk_list.append(chunk)

ins_processed = pd.concat(chunk_list)

The zip check function that I'm applying is basically performing some datatype checks and then converting the value that it gets into an integer.
Whenever I read the CSV it will only ever read the institution_id column and generates an index. The other columns in the CSV are just silently dropped.
When i dont use index_col=False as an option it will just set 1141414141/4141414141/JOHN/SMITH/0.7 (basically the first 5 values in the row) as the index and only institution_id as the header while only reading the institution_name into the dataframe as a value.
I have honestly no clue what is going on here, and after 2 hours of SO / google search I decided to just ask this as a question. Hope someone can help me, thanks!

A: The issue came out to be that something went wrong while transferring the large CSV file to my remote processing server (which sufficient RAM to handle in memory editing). Processing the chunks on my local computer seems to work. 
After reuploading the file it worked fine on the remote server.
