How do I utilize Hash Based Routing for Multiple Generations on API Deployment

tim-wright
tim-wright Partner, L2 Designer, Snowflake Advanced, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 77 Partner

I was looking through the API-deployer options for simultaneously deploying multiple active models to a single endpoint. I have used the "Random" strategy to randomly route portions of the incoming requests to different versions of the model (as described here).

My question is, how can we utilize the "Hash Based" routing strategy? I assume this will allow us to call specific active versions of the endpoint instead of having requests randomly routed - Is this correct?? I haven't found any specific documentation referencing this capability.

What I have done (just fooling around)

If I use the "entries" format similar to what would be used for "Random" strategy and deploy the model it appears to deploy multiple versions successfully. The probas argument here seems counter to my assumption that for Hash-based routing the models are not routed probabilistically but directly via the API call. That being said, using this "entries" the models appear to deploy. See two images below.

Mulfiple_Generations.PNG

Mulfiple_Generations2.PNG

BUT, when I try to query the endpoint using Postman, I receive a 400 bad request. Saying I need to pass the "dispatch key" in the query.

Mulfiple_Generations3.PNG

Can anyone provide guidance on how hash-based routing works (if it does what I think it does) and if so, how can we leverage it?

Thanks,

Tim

Answers

  • NedM
    NedM Dataiker Posts: 8 Dataiker
    edited July 17

    Hi Tim!

    This is a great question, thanks for asking

    Let's start with the "how", and then move onto the "why". First, you're correct in pointing out that, when querying an API endpoint that uses hash-based routing, the JSON payload of the query must include a "dispatch key".

    The following example shows the expected formatting of the JSON payload; namely, the additional "dispatch" dictionary entry. If, on the other hand, you're querying the API endpoint via the dataikuapi python package, the predict_record() method includes a dispatch_key parameter.

    curl -X POST \
      http://<host>:<port>/public/api/<version>/<service_name>/<endpoint_name>/<endpoint_type&gt; \
      --data '{ "features" : {
        "feature_1": 10,
        "feature_2": "accepted",
        "feature_3": 25
      },
        "dispatch" : {"dispatchKey" : "<dispatch_key_string>" }
      }'

    Second, to address how the hash-based routing method works: is deterministic based on the value of the dispatch key and the endpoint version probabilities. More specifically,

    1. The probabilities assigned to each endpoint version specify the relative proportion of "hash values" assigned to each endpoint.
    2. When DSS receives a query, it first computes the hash of the dispatch key, and then determines which endpoint version to query based on the value of the hashed dispatch key.

    Therefore, the behavior of hash-based routing is slightly different to what you describe. I would still consider it to be "random", as there is no a-priori guarantee as to which endpoint version will be chosen for a given dispatch key; however, the choice of version will be deterministic based on the hash-value of the key.

    As a result, you can ensure that two arbitrary queries utilize the same endpoint version by using the same dispatch key (although you aren't able to chose the version itself). For choosing the precise version, I would probably just set up two different endpoints (i.e. with v1 and v2) and query them directly.

    Hopefully this information helps clarify this topic, although please let me know if you have any other questions! Also, thanks again for asking!

    Ned

  • tim-wright
    tim-wright Partner, L2 Designer, Snowflake Advanced, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 77 Partner

    @NedM
    Thanks for the response. What is the primary use case for Hash based routing? Can you give me an example of how this would be useful in practice. I'm not fully sure I am aware of the use case where routing would need to ensure that all records of a similar dispatch key would need to be routed to the same version, but you would allow which version to be decided randomly.

    Also...is the internal mapping of hashes to endpoint versions deterministic for a specific set of probabilities?

    Thanks again for the detailed response.

    -Tim

Setup Info
    Tags
      Help me…