O365 connectors within Teams will be deprecated and notifications from this service

yfzh
yfzh Registered Posts: 1

We setup a script to send out scenarios checking using get_message_sender() & send() APIs to team channel. Starting today, this message popup:

"Action Required: O365 connectors within Teams will be deprecated and notifications from this service will stop. Learn more about the timing and how the Workflows app provides a more flexible and secure experience. If you want to continue receiving these types of messages, you can use a workflow to post messages from a webhook request. Set up workflow"

It seems that Microsoft will deprecate O365 connectors starting from Aug, 2024 and retire this at Oct, 2024. Is there an easy solution to solve this in v12?

Operating system used: windows 11

Answers

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    Yes, I'm seeing the same thing, since we also useget_message_sender() and send()and I would like to understand how the integration between these APIs and Teams workflows should be configured?

  • Jorge
    Jorge Registered Posts: 2 ✭✭

    We are having the same scenario. Are there any plans to update Dataiku to allow integration with workflows instead of O365 connectors?

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    I have raised this with Dataiku Support and they have confirmed the integration works when using Power Automate Workflows. The steps are pretty simple, I did myself on my organisation although I can't test it since we do not have permissions to run Power Automate Workflows. One thing is that even the integration works with Power Automate Workflows the reporters end in error due to the Power Automate Workflows returning HTTP code 202 rather than 200. Dataiku said they will fix this in an upcoming v13 patch release.

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    Hi @Turribeach ,
    Any chance you could share the steps? I understand we still need to wait for the v13 patch release before it will work properly.

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    So in my organisation Teams you go to a channel, select the 3 dots, select Workflows, then search for webhook and select the "Post to a channel when a webhook request is received" template. Give it a name, click on Next abd then click on Add workflow. You should now get the "Your workflow has been created" message and the webhook URL you can use in Dataiku. To see the Workflows you created you can go to Workflows on the Teams left side bar.

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    Thanks @Turribeach !
    I tried this and used the new webhook in the send() function, eg:
    message_sender.send(message=html_message, sendAsHTML=True, webhookUrl=<new webhook>)
    but I guess it didn't work because of this issue where it returns HTTP code 202 rather than 200 that you mentioned. Will wait for the patch release 🤞

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    What error do you get? Can you post the full stack trace? Are you able to trap the error? Did the message get delivered?

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭
    [2024/07/10-10:14:10.926] [qtp579447973-449766] [DEBUG] [dku.tracing]  - [ct: 513] Done call: /api/tintercom/scenarios/send-message [POST] time=512ms user=stanislava.schaefer@olamagri.com auth=<AC:user:stanislava.schaefer@olamagri.com via: ticket:scenario_python_step:IRONOREFMRETURNSPREDICTION.CAPE_FM_DAILY_RUN,via: scenario-run: scenario=IRONOREFMRETURNSPREDICTION.CAPE_FM_DAILY_RUN runId=2024-07-10-10-14-06-390,via: stanislava.schaefer@olamagri.com> [variablesJson={} projectKey=IRONOREFMRETURNSPREDICTION] thrown=IOException
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  - Traceback (most recent call last):
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -   File "/data/dataiku/datadir/scenarios/IRONOREFMRETURNSPREDICTION/CAPE_FM_DAILY_RUN/2024-07-10-10-14-06-390/custom-step-Send Teams Message/script.py", line 143, in <module>
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -     message_sender.send(message=html_message, sendAsHTML=True,
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -   File "/home/dataiku/dataiku-dss-12.6.1/python/dataiku/scenario/messaging.py", line 35, in send
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -     return backend_json_call("scenarios/send-message", data)
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -   File "/home/dataiku/dataiku-dss-12.6.1/python/dataiku/core/intercom.py", line 550, in backend_json_call
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -     return _handle_json_resp(backend_api_post_call(path, data, **kwargs), err_msg = err_msg)
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -   File "/home/dataiku/dataiku-dss-12.6.1/python/dataiku/core/intercom.py", line 649, in _handle_json_resp
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  -     raise Exception("%s: %s" % (err_msg, _get_error_message(err_data).encode("utf8")))
    [2024/07/10-10:14:10.929] [Exec-450965] [INFO] [process]  - Exception: None: b'API call to post message on Teams failed 202: Accepted'
    

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    Ummm that's not good. OK I just checked the instructions I got from Dataiku Support and they are slightly different. Rather than using an existing Workflow template. Can you test with these instructions instead?

    Step 1 - Create a workflow in Microsoft Teams that posts a card to a channel when a 

    • In Microsoft Teams, open the Workflows apps from the left panel.
    • Click "+ New flow"
    • Click "+ Create from blank"
    • Add a trigger When a Teams webhook request is received
      • Who can trigger the flow: Anyone
    • Click "+ New step"
    • Add an action of type Control named Apply to each
      • Select an output from previous steps: dynamic content "attachments"
    • Click "Add an action"
    • Select Post a card in a chat or channel operation
      • Post as: Flow bot
      • Post in: Channel
      • Team: select a team
      • Chanel: select the destination channel
      • Adaptive Card: dynamic content "content"
    • Save
    • Expand the trigger When a Teams webhook request is received and copy the HTTP POST URL


    Step 2 - Configure a Scenario reporter

    • In DSS, open a scenario and add a Microsoft Teams reporter
      • Webhook URL: Paste the URL you just copied
      • Message: Copy/paste the following snippet 
    {
    "attachments":[
    {
    "contentType":"application/json",
    "content":{
    "$schema":"http://adaptivecards.io/schemas/adaptive-card.json",
    "type":"AdaptiveCard",
    "version":"1.0",
    "body":[
    {
    "type":"Container",
    "items":[
    {
    "type":"TextBlock",
    "text":"${scenarioName} run report",
    "weight":"bolder",
    "size":"medium",
    "style":"heading",
    "wrap":true,
    "color":"attention"
    },
    {
    "type":"TextBlock",
    "wrap":true,
    "text":"[${scenarioName}](${scenarioRunURL}): **${outcome}**",
    "color":"good"
    },
    {
    "type":"FactSet",
    "facts":[
    {
    "title":"Project",
    "value":"${scenarioProjectKey}"
    },
    {
    "title":"Triggered by",
    "value":"${triggerName}"
    }
    ]
    }
    ]
    }
    ]
    }
    }
    ]
    }
    

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    I followed the instructions and tested and I'm seeing the same issue:

    2024/07/10-10:32:52.642] [FT-ScenarioThread-WMRZQVQZ-453044] [WARN] [dip.scenario.run_thread] scenario IRONOREFMRETURNSPREDICTION.CAPE_FM_DAILY_RUN#2024-07-10-10-31-27-126 - Failure during report on end of scenario
    java.io.IOException: API call to post message on Teams failed 202: Accepted
    	at com.dataiku.dip.integrations.im.TeamsIntegrationHandler.sendNotification(TeamsIntegrationHandler.java:210)
    	at com.dataiku.dip.integrations.im.TeamsIntegrationHandler.handleEvent(TeamsIntegrationHandler.java:162)
    	at com.dataiku.dip.scheduler.reports.ScenarioReporterRunner.sendMessage(ScenarioReporterRunner.java:165)
    	at com.dataiku.dip.scheduler.reports.ScenarioReporterRunner.sendOnEnd(ScenarioReporterRunner.java:98)
    	at com.dataiku.dip.scheduler.ScenarioThread.execute(ScenarioThread.java:181)
    	at com.dataiku.dip.futures.FutureThreadBase.run(FutureThreadBase.java:136)
    

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    Hi, you are doing in code using the send() method right? Do you mind trying a Reporter in the Scenario settings? I believe this is what Dataiku Support tested and even though the Scenario Reporter ended in error it should deliver the message.

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭
    edited July 10

    Hi, no in the above I tried exactly as outlined and used a Reporter in the Scenario settings.

  • apichery
    apichery Dataiker, Alpha Tester, Registered, Product Ideas Manager, Moderator Posts: 63 Dataiker

    Dear Microsoft Teams users,

    I confirm that the new webhook generated by Workflows/PowerAutomate is not fully compatible with the DSS/Teams integration in the existing DSS releases. The reason is that they changed the payload of the JSON expected by the Webhook.

    What works (kind of):

    • Scenario reporter with custom JSON message (see the example posted above: { "attachments": […] }). The reporter will be marked as failed because DSS does not recognize HTTP 202 as a success but the message is correctly delivered on the Teams channel
    • Send a message scenario step with custom JSON message (same as above, except if will mark the scenario as failed, even though message will be delivered on the Teams channel)

    What does not work:

    • Scenario reporter with non-JSON message
    • "Send a message" step with non-JSON message
    • Python API "message_sender.send()"

    The good news is that this will be fixed in the upcoming DSS 13.0.2. Stay tuned.

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    Thanks @apichery
    Looking forward to the fix in DSS 13.0.2 - will this be available for cloud platform users as well?

    Regarding:

    • Scenario reporter with custom JSON message (see the example posted above: { "attachments": […] }). The reporter will be marked as failed because DSS does not recognize HTTP 202 as a success but the message is correctly delivered on the Teams channel
      » This did not work for me. I don't know if there are some limitations to posting in private vs public Teams channels however? I tested it for a private channel.

  • me2
    me2 Registered Posts: 54 ✭✭✭✭✭
  • apichery
    apichery Dataiker, Alpha Tester, Registered, Product Ideas Manager, Moderator Posts: 63 Dataiker
  • apichery
    apichery Dataiker, Alpha Tester, Registered, Product Ideas Manager, Moderator Posts: 63 Dataiker

    For the bad news…Workflows in Microsoft Teams are currently unable to send message to private channels.
    So DSS - as any other tool - will be unable to send message to private channels until Microsoft solve this.

    (Search for "private" in the comments on the thread with the retirement announce: https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/ )

  • apichery
    apichery Dataiker, Alpha Tester, Registered, Product Ideas Manager, Moderator Posts: 63 Dataiker

    Starting with DSS 12.6.5 and 13.0.1, you'll have to write the following to send a message through a webhook based on Workflows.

    message_sender.send(message=…, webhookUrl=<new webhook>, webhookType="WORKFLOWS")
    

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    Smashing news, many thanks for backporting this fix. We will try to put pressure on Microsoft to fix the private channel issue.

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    Yeah this is a bummer, and given I doubt Microsoft will change this, I'll need to move to a chat rather than a public channel

  • apichery
    apichery Dataiker, Alpha Tester, Registered, Product Ideas Manager, Moderator Posts: 63 Dataiker

    … and 12.6.5 has been released. Enjoy your new Microsoft Teams webhooks (except for those posting to private channels)

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    For anyone on this thread it might be worth up voting this feedback suggestion to prevent the retirement of Teams Webhooks:

    https://feedbackportal.microsoft.com/feedback/idea/80ed6877-b642-ef11-b4ad-000d3a7aba8b

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,063 Neuron

    Microsoft posted an update on their blog post:

    https://devblogs.microsoft.com/microsoft365dev/retirement-of-office-365-connectors-within-microsoft-teams/

    Basically they now deprecate this functionality at the end of 2025 but you will need to update the Webhooks URL before the end of 2024 and the new URL will be provided 90 days before that. But they will still stop allowing new Teams Webhooks being created from August 15th.

    There are also updates to this post in that all the alternatives provided are not good replacements:

    https://feedbackportal.microsoft.com/feedback/idea/80ed6877-b642-ef11-b4ad-000d3a7aba8b

  • bdforbes
    bdforbes Registered Posts: 5 ✭✭✭✭

    After some struggles I've found that the new Workflows capability is just barely fit for purpose - as long as you're setting the webhook up on a non-private channel, you can configure the Flow to post to the channel as Flow Bot. For my user group and organisation that is sufficient, but I guess in principle some people will need the ability to restrict visibility of the notifications.

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    Thanks @apichery
    I'm able to use message_sender.send()as before. Unfortunately however my HTML formatting is not showing up correctly in the Teams message. Do you have any advice for how to fix this part? Thanks again for the upgrades.

  • Stanislava
    Stanislava Registered Posts: 15 ✭✭✭

    Nevermind. I just discovered that MS Teams does not render HTML in adaptive cards…ugh. I guess the solution is to use Markdown formatting: Text formatting in cards - Teams | Microsoft Learn

Setup Info
    Tags
      Help me…