Q: ValueError: Attempted to encode null for non-nullable field "last_review" I am trying to read a CSV as a beam dataframe and convert it into a Pcollection.
pipeline code:
  with beam.Pipeline(options=pipeline_options) as p:
    df = p | read_csv(input_file)
    pcol = to_pcollection(df)

Throws the following error:
ValueError: Attempted to encode null for non-nullable field "last_review". [while running 'Unbatch 'placeholder_DataFrame_140400915519808'/ParDo(_UnbatchNoIndex)']

I understand that this is happening because of NULL values in the field. It does not throw this error if I use a Dataset without NULL values.
Question:
Is there a way to make the fields/columns in the DataFrame Nullable before converting it into a Pcollection, so that it does not throw the error? If not how can I get around this?
I referred to this Post:
Beam Python SDK: pd.merge left join error (valueError: Attempted to encode null for non-nullable field)
Someone Suggested to do this:
to_pcollection(..., yield_elements='pandas')

It works but seems to be loading the pandas dataframe object into the Pcollection. Question: Is the pandas dataframe object loaded into the memory of a single worker node or is it split into multiple worker nodes? If it is loaded into a single worker node's memory it might run into memory issues for Huge files which I don't want it to happen.
Please correct my understanding.

A: I solved this by reading the whole CSV as an object dtype and filling the NaN values with an empty string as below.
df = p | read_csv(input_file,dtype=object)
        pcol = to_pcollection(df.fillna(''))

If you want to retain the original dtypes of the columns, you could identify the columns that are object dtype and then fill the NaN values in them. There are many examples to do this in pandas but I could not get it to work on beam dataframes.
