I figured it out after 5 hours of research... The original parquet file I had created was corrupted. I did not use the correct code to create it. Below is how you can create a parquet file in a Managed folder (S3 or Azure Blog Container).
output_file_name = 'lab_samples.parquet'
# Create file
output_managed_folder_id = "xxxxx" # Managed Folder
output_folder = dataiku.Folder(output_managed_folder_id)
f = io.BytesIO()
content = f.read()
See the last example on the page:
This is how you read it:
from dataiku import pandasutils as pdu
import pandas as pd
# Open sample file
input_managed_folder_id = "xxxxx"
input_folder = dataiku.Folder(input_managed_folder_id)
input_file_name = 'lab_samples.parquet'
file_stream = input_folder.get_download_stream(input_file_name)
file_bytes = io.BytesIO(file_stream.read())
lab_samples = pd.read_parquet(file_bytes, engine='pyarrow')