Configurig Mistral with Kong's AI Semantic Plugin

Questions
,
1

I’ve been having a ton of issues trying to get the AI Semantic plugin to work with Mistral. After setting up the plugin using Kong Konnect I was finally able to enable the plugin to work with the routes having an hybrid gateway manger and a node working on my machine in the same network as the REDIS database that I am using for the same effect. Unfortunatelly, everytime I activate the plugin, all requests get rejected and I get a 400 Bad Requests. I tryed to change parameters and made a ton of experiments, but the only thing I was finally able to do was to simply use the AI proxy plugin to communicate with Mistral and that works like a breeze. Not the AI semantic prompt guard plugin however. So my configuration for Mistral looks like this in terms of the AI proxy:

curl -X POST \
https://eu.api.konghq.com/v2/control-planes/{control_plane_id}/core-entities/services/{service_id}/plugins \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer KONG_API_KEY" \
    --data '{
      "name": "ai-proxy",
      "config": {
        "model": {
          "name": "mistral-medium",
          "provider": "mistral",
          "options": {
            "mistral_format": "openai",
            "upstream_url": "https://api.mistral.ai/v1/chat/completions"
          }
        },
        "auth": {
          "header_name": "Authorization",
          "header_value": "Bearer MISTRAL_API_KEY"
        },
        "route_type": "llm/v1/chat"
      }
    }'

and it looks like this for the AI Semantic Prompt Guard plugin:

curl -X POST \
https://eu.api.konghq.com/v2/control-planes/{control_plane_id}/core-entities/routes/{route_id}/plugins \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer KONG_API_KEY" \
    --data '{
          "name": "ai-semantic-prompt-guard",
          "config": {
            "rules": {
              "match_all_conversation_history": true,
              "allow_prompts": ["Questions about StarWars"],
              "deny_prompts": ["Questions about StarTrek"]
            },
            "embeddings": {
                 "auth": {
                      "header_name": "Authorization",
                      "header_value": "Bearer MISTRAL_API_KEY"
                    },
                 "model": {
                    "provider": "mistral",
                    "name": "mistral-embed",
                    "options": {
                        "upstream_url": "https://api.mistral.ai/v1/embeddings"
                      }
                  }
            },
            "vectordb": {
              "dimensions": 1024,
              "distance_metric": "cosine",
              "strategy": "redis",
              "threshold": 0.1,
              "redis": {
                "host": "redis",
                "port": 6379
              }
            }
          }
        }
    '

I configued the service and the route manually in Kong Konnect after createing a hybrid mode gateway manager and having two containers running locally. One for Kong and the other for Redis. However nothing works yet and the plugin only blocks the requests. The full examples can be found on my repo at GitHub - jesperancinha/kong-test-drives: Kont test project on folder: kong-test-drives/kong-ai at main · jesperancinha/kong-test-drives · GitHub. Can anyone help me with this and could I be missing at this point? Please let me know! Cheers! PS: I dot not want to use OpenAI for this example. I specifically want to use Mistral. Thank you!

2

Hi @jesperancinha thanks for reaching out. When you mention Kong returns 400, what does the error message say?

3

Hello there,
It always says 400 Bad Request and a reference number and that’s it. Nothing else and there are also no logs being recorded of any attempt to connect to Redis and so I’m not even sure of Kong was able to establish a connection with Redis. Both containers have no logs regarding that.

4

Hey @jesperancinha

It might be the same problem I had: you need to run a redis image with FT indexes installed.

That isn’t in Docker Hub “redis” image, you need e.g:

docker run -it --rm --name redis -p 6379:6379 redis/redis-stack-server

You able to check again with this image?

1 Like
5

It is also the wrong model for Mistral embeddings I think. mistral-medium is a chat model.

This is my configuration (Mistral only support maximum 1024 vectors):

    - name: ai-semantic-cache
      config:
        embeddings:
          auth:
            header_name: "Authorization"
            header_value: "{vault://env/MISTRAL_AUTH_HEADER}"
          model:
            provider: mistral
            name: mistral-embed
        vectordb:
          strategy: redis
          distance_metric: euclidean
          dimensions: 1024
          threshold: 0.2
          redis:
            host: redis
            port: 6379
1 Like
6

Hello there @JackGPT ,

Thanks for reaching out. Thanks to your input I was able to connect to Redis, and I can finally see connection logs that confirm that the vector database is being created. However, there seems to be a problem still because I still get a “bad request”. This is my current configuration. The AI proxy-plugin looks like this:

curl -X POST \
"https://eu.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/services/$ROUTE_ID/plugins" \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONG_API_KEY" \
    --data "{
      'name': 'ai-proxy',
      'config': {
        'model': {
          'name': 'mistral-medium',
          'provider': 'mistral',
          'options': {
            'mistral_format': 'openai',
            'upstream_url': 'https://api.mistral.ai/v1/chat/completions'
          }
        },
        'auth': {
          'header_name': 'Authorization',
          'header_value': 'Bearer $MISTRAL_API_KEY'
        },
        'route_type': 'llm/v1/chat'
      }
    }"

Here I have a question. Do I also need the “ai-proxy” to work with “mistral-embed” I couldn’t get it to work like that. I was able to configure that though in the “ai-semantic-prompt-guard” though:

curl -X POST \
"https://eu.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/services/$ROUTE_ID/plugins" \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONG_API_KEY" \
    --data "
{
  'name': 'ai-semantic-prompt-guard',
  'config': {
    'embeddings': {
      'auth': {
        'header_name': 'Authorization',
        'header_value': $MISTRAL_API_KEY
      },
      'model': {
        'provider': 'mistral',
        'name': 'mistral-embed',
           'options': {
            'upstream_url': 'https://api.mistral.ai/v1/embeddings'
          }     }
    },
    'rules': {
      'match_all_conversation_history': true,
      'allow_prompts': [
        'Questions about the sun'
      ]
    },
    'vectordb': {
      'strategy': 'redis',
      'distance_metric': 'euclidean',
      'dimensions': 1024,
      'threshold': 0.2,
      'redis': {
        'host': '$REDIS_HOST',
        'port': 6379
      }
    }
  }
}
"

But still, in both cases, I’m still getting a “bad request” the moment I activate the semantic plugin. Here is an example of an error I get:

 curl -X POST http://localhost:8000/mistral \
>   -H "Content-Type: application/json" \
>   -d '{
quote>   "messages": [
quote>     {
quote>         "role": "system",
quote>         "content": "tell me the color of the sun"
quote>     }
quote>   ]
quote>   }'
{"error":{"message":"bad request"}}%

When I deactivate it, then I get a correct response from Mistral.

Let me know if you see anything in this config that is wrong. It may be that I need to also use embed in the “ai-proxy” but I’m not entirely sure.

7

I think your embeddings auth might be wrong. Maybe?

'header_value': $MISTRAL_API_KEY

Mistral accepts API key like Bearer $MISTRAL_API_KEY

8

I just checked it. It didn’t work.

The ai-semantic-cache plugin does work with Mistral very well actually and kong’s webpage provides enough information to do that. I got it to work with the ai-proxy and the ai-semantic-cache plugin. However, with the same configurartion, it just doesn’t seem to work with the ai-semantic-prompt plugin yet.

Would this have something to do with possible limitations of Mistal’s API Key? It is quite impossible at this point to assess anything because I can’t get any good log information anywhere about why I simply get a Bad Request.

It is, however good to know that the ai-semantica-cache plugin does work and very werll :slight_smile:

Let me know if you have any idea. I’ve updated the repo, this is my current config:

curl -X POST \
"https://eu.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/core-entities/routes/$ROUTE_ID/plugins" \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $KONG_API_KEY" \
    --data '
{
  "name": "ai-semantic-prompt-guard",
  "config": {
    "embeddings": {
      "auth": {
        "header_name": "Authorization",
        "header_value": "Bearer '$MISTRAL_API_KEY'"
      },
      "model": {
        "provider": "mistral",
        "name": "mistral-embed",
           "options": {
            "upstream_url": "https://api.mistral.ai/v1/embeddings"
          }     }
    },
    "rules": {
      "match_all_conversation_history": true,
      "allow_prompts": [
        "Questions about the sun"
      ]
    },
    "vectordb": {
      "strategy": "redis",
      "distance_metric": "euclidean",
      "dimensions": 1024,
      "threshold": 0.2,
      "redis": {
        "host": "'$REDIS_HOST'",
        "port": 6379
      }
    }
  }
}
'```
9

I’m not sure on this one, it works over here, I really think your header value is still wrong; it should be very specific:

{
  "name": "ai-semantic-prompt-guard",
  "config": {
    "embeddings": {
      "auth": {
        "header_name": "Authorization",
        "header_value": "Bearer '$MISTRAL_API_KEY'"
      },

Remove the single quotes around the key:

{
  "name": "ai-semantic-prompt-guard",
  "config": {
    "embeddings": {
      "auth": {
        "header_name": "Authorization",
        "header_value": "Bearer $MISTRAL_API_KEY"
      },

Same as ai-proxy exactly.

Maybe we could just clear this up on the Community Slack / Zoom, I’ll find you.