Q: Writestreaming query not writing data into table in databricks Can someone help me with this issue.
I have a delta table "orders". this table is loaded with 1000 records using the delta file.
Now we are getting the streaming JSON file which is appending the data into this table.
Readstream dataframe **orderInputDF **:
    from pyspark.sql.functions import *
orderInputDF = (spark
                .readStream
                .schema(jsonSchema)
                .option("maxFilesPerTrigger", 1)
                .json(stream_path).
                withColumn("submitted_at",to_timestamp('submittedAt')).
                withColumn("submitted_yyyy_mm", col('submitted_at').substr(0,7)).
                withColumnRenamed("orderId",'order_id').                
                withColumnRenamed("customerId","customer_id").
                withColumnRenamed("salesRepId","sales_rep_id").
                withColumn("shipping_address_attention", col("shippingAddress.attention")).
                withColumn("shipping_address_address", col("shippingAddress.address")).
                withColumn("shipping_address_city", col("shippingAddress.city")).
                withColumn("shipping_address_state", col("shippingAddress.state")).
                withColumn("shipping_address_zip", col("shippingAddress.zip")).
                withColumn("ingest_file_name",lit(files[0][0])).
                withColumn("ingested_at",current_timestamp()).
                drop("products","shippingAddress","submittedAt")
               )

This readstream dataframe is returning 20 records.
Writestream Query is as below
orderOutputDF = (orderInputDF
          .writeStream
          .format("delta")
          .format("memory")
          .queryName(orders_table) 
          .outputMode("append")
          .option("checkpointLocation", orders_checkpoint_path)
          .start()
)

This query is not appending the data. Everytime it is inserting the new records and deleting the old records.
Can someone help me on this.
