Actually to do unit testing on the final and intermediate datasets, need more dataset file formats such as parquet, Avro, sas7bdat, ORC, etc while exporting datasets to the local system for large datasets, as CSV format can't handle more than 1 million records.