http connection to csv file that changes column count

tgb417
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron

I've been playing with the GitHub Coronavirus data at:

https://github.com/CSSEGISandData/COVID-19

Thank You Johns Hopkins CSSE.

I've been using this URL in an HTTP dataset.

https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Confirmed.csv

However, the dataset gets a new column each day. When DSS looks at the new data it does not reflect the Schema that the data had the prior day and throws at least a warning.

Is there a better way to gather such a dataset that expands in this column-wise way without manually refreshing the schema?

--Tom

Best Answer

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron
    Answer ✓

    So, I think that I found the answer.

    tgb417_1-1585087296973.png

Setup Info
    Tags
      Help me…