<p>KEDA can best suit this scenario. Event based scaling can help autoscale the K8s pods accordingly based on the chosen events. KEDA offers mutiple event options to act as scaling factor. Reference: <a href="https://keda.sh/docs/2.0/scalers" rel="nofollow noreferrer">https://keda.sh/docs/2.0/scalers</a>
If no available event triggers can be used, an external scaler can also be used as an API which can be used as HPA for Kuberenetes.
<a href="https://keda.sh/docs/2.0/scalers/external/" rel="nofollow noreferrer">https://keda.sh/docs/2.0/scalers/external/</a></p>

CC BY-SA 4.0
2024-11-06T12:27:25.263Z
2024-11-06T12:27:25.263Z
madhinigm
<p>I have an application which processes a single stream of data. Each stream has a unique identifier and a URL. The number of streams is dynamic, but I only want one instance of the application to process each stream.</p>
<p>I want to build the infrastructure to have a single application to process each stream, and expand dynamically when new streams are added. (You can assume that the list of streams is in a database) If the processor of the stream dies, it needs to be restarted automatically.</p>
<p>I was exploring using Kubernetes StatefulSets for this, since each instance will have its unique identifier, but I don't know how to link these to a unique stream, and how to trigger new instances to be executed when new streams are added (as automatically as possible). Maybe using different namespaces, but I want to be able to do it dynamically.</p>
<p>I am a bit new to Kubernetes, so not too sure if I am missing something obvious. Is there a way to achieve this? Similar to how for example databases have master and slaves with different configurations, but more dynamic since streams can be added or deleted during execution.</p>
<p>How is it best to go about something like this?</p>

<kubernetes>
Kubernetes stateful pods for individual streams
jbx
# Kubernetes stateful pods for individual streams

(Asked by: username_0 on 2024-11-05)

I have an application which processes a single stream of data. Each stream has a unique identifier and a URL. The number of streams is dynamic, but I only want one instance of the application to process each stream.

I want to build the infrastructure to have a single application to process each stream, and expand dynamically when new streams are added. (You can assume that the list of streams is in a database) If the processor of the stream dies, it needs to be restarted automatically.

I was exploring using Kubernetes StatefulSets for this, since each instance will have its unique identifier, but I don't know how to link these to a unique stream, and how to trigger new instances to be executed when new streams are added (as automatically as possible). Maybe using different namespaces, but I want to be able to do it dynamically.

I am a bit new to Kubernetes, so not too sure if I am missing something obvious. Is there a way to achieve this? Similar to how for example databases have master and slaves with different configurations, but more dynamic since streams can be added or deleted during execution.

How is it best to go about something like this?

**Tags:** kubernetes


---

## 1 answers

### Answer by username_1 on 2024-11-06. Score: 0

KEDA can best suit this scenario. Event based scaling can help autoscale the K8s pods accordingly based on the chosen events. KEDA offers mutiple event options to act as scaling factor. Reference: <https://keda.sh/docs/2.0/scalers>
If no available event triggers can be used, an external scaler can also be used as an API which can be used as HPA for Kuberenetes.
<https://keda.sh/docs/2.0/scalers/external/>


---

