Q: SSL Encryption for Kafka Pipelines Any [SingleStore] users out there?
For enabling SSL encryption (only) for Kafka pipelines, we created the pipeline like below:
CREATE PIPELINE `kafka_ssl`
AS LOAD DATA KAFKA '<kafka-host>:<kafka-port>/test'
CONFIG '{"security.protocol": "ssl"
"ssl.ca.location": "<path-to-the-ca-certificate>"}''
INTO table <t>;

from https://docs.memsql.com/v7.0/concepts/pipelines/kafka-kerberos-ssl/ 22
Please note that we're not looking for Client authentication (mutual TLS connection). This requires us to manually copy the CA Cert into memsql nodes and provide that path. This works fine for standalone Kafka cluster.
What is the recommended approach if we're using AWS MSK?
Following documentation points to the client configuration for SSL encryption for MSK clients. They use client truststore location as a property which is not an identified property for memsql pipeline config json
https://docs.aws.amazon.com/msk/latest/developerguide/msk-encryption.html 10
https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html
See others  having the same issue in the SingleStore forums. Unable to create a pipeline from Singlestore in a box to AWS MSK.

A: Let's suppose,
This is your (== customer's) service name: com.amazonaws.vpce.ap-southeast-2.vpce-svc-05738b36bf477df03
This is mine (== ss's) dns name for your endpoint: com.amazonaws.vpce.ap-southeast-2.vpce-svc-05738b36bf477df03 vpce-07e59dd12b36925ee-q2lyxq0y.vpce-svc-05738b36bf477df03.ap-southeast-2.vpce.amazonaws.com vpce-svc-06f99678d75724074.ap-southeast-2.vpce.amazonaws.com
DNS == vpce-07e59dd12b36925ee-q2lyxq0y.vpce-svc-05738b36bf477df03.ap-southeast-2.vpce.amazonaws.com Which you use for your __spoof.dns; vpce-svc-06f99678d75724074.ap-southeast-2.vpce..com com.amazonaws.vpce.ap-southeast-2.vpce-svc-06f99678d75724074
CREATE OR REPLACE PIPELINE kafka_insert_thing_event_v0 AS LOAD DATA KAFKA 'DNS:6000/prod_common_v0_event_wrapper_thing_event' CONFIG '{ "__spoof.dns": { "DNS:6000": "DNS:6000", "b-1.events.eg3qf0.c2.kafka.ap-southeast-2.amazonaws.com:9092": "DNS:6001", "b-2.events.eg3qf0.c2.kafka.ap-southeast-2.amazonaws.com:9092": "DNS:6002", "b-3.events.eg3qf0.c2.kafka.ap-southeast-2.amazonaws.com:9092": "DNS:6003" } }' INTO TABLE t ;
I have also attached MSK documentation for integrating with SingleStore for the reference. I hope it will help you.
https://docs.google.com/document/d/17nl9S__oxIo4J30OM1WHTkkqrCSYFqJ44sUcJxKM0QA/edit?usp=sharing&urp=gmail_link
