Importing Splunk data
![Gabriel](https://us.v-cdn.net/6038231/uploads/lithium_attachments/n7853iDE73471EB7557E6B.jpg)
Hi everyone
I try to import Splunk data using the official: plugin.
The problem is, that the plugin seems to assume all my Splunk data is ascii encoded, when in reality it is UTF-8. The plugin has no option that lets me specify the encoding.
Since Splunk data is usually voluminous, I can't really solve the issue there since reindexing data is associated with significant costs.
Following the error message I get when I try to import data:
... 2023-03-21 15:44:56,091 INFO SplunkIndexConnector:Connected to Splunk 2023-03-21 15:44:56,093 INFO Processing task: read_rows 2023-03-21 15:44:57,899 ERROR Connector send fail, storing exception Traceback (most recent call last): File "/.../dataiku-dss-11.0.2/python/dataiku/connector/server.py", line 110, in serve read_rows(connector, schema, partitioning, partition_id, limit, output) File "/.../dataiku-dss-11.0.2/python/dataiku/connector/server.py", line 32, in read_rows for row in connector.generate_rows(schema, partitioning, partition_id, limit): File "/tmp/tmp_folder_nSFiBdol/dku_code.py", line 91, in generate_rows UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 173000: ordinal not in range(128) 2023-03-21 15:44:57,901 INFO Processing task: finish_read_session
Best Answer
-
Hi !
Could you try to convert this plugin into a dev plugin, then edit line 91 of the file named `python-connectors/splunk_import-index/connector.py`, from this:
for sample in content.decode().split("\n"):
into this:
for sample in content.decode("utf-8").split("\n"):
and see if it solves the issue..
Answers
-
Gabriel Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 9 Partner
Hey Alex!
First of all, thanks for your reply and patience. I was held up by a couple other tasks for a while :S
I just tested your solution and it seems to work. However, I got a couple other errors but I think they are related to something else. I will come back to you with a more thorough reply!
Thanks for taking the time to sift through the code and providing a solution. I greatly appreciate your effort!
Regards,
Gabriel
-
Gabriel Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 9 Partner
All right, I tested your solution and it works perfectly.
Thanks again and have a great day!