<p>The service can be <em>always-on</em> listening on the queue for jobs, or <em>single-job</em> started by the queue only when new work arrives.</p>
<p>The tradeoff is <em>startup time</em> for single-jobs vs <em>cost</em> for always-on service.</p>
<p>Define the requirements - number of jobs per day, max latency, ram/cpu needed, etc and then select the right technology that can deliver it in the most cost effective way.</p>
<p>From AWS Lambda, AWS Step Functions, through K8S Jobs and AWS Fargate to AWS EC2 on spot pricing, and more. Plenty of options, depends on the requirements.</p>
<p>Hope that helps :)</p>

CC BY-SA 4.0
2020-11-18T09:17:44.083Z
2020-11-18T09:17:44.083Z
MLu
<p>We are building a system for data science in AWS and our flow is pretty simple.</p>
<ol>
<li>Get the data - From redis</li>
<li>get the model - From FSx/EBS</li>
<li>run.</li>
</ol>
<p>This process script should take something like 5-10 seconds depends on the size of the data and model.
So far so good.</p>
<p>What we did so far is to build a python application (runs in kubernetes) that listen to a queue and runs the data science flow.
The problem we faced was Pythons memory leak. what ever we did and every approach we took with python a long running application that loads/unloads large objects will grow and grow, So we are trying a new approach , running the script on its own process each time a new process is created and closed when finished (still in kubernetes).</p>
<p>My question is
Is there a better way?
AWS Lambda - could be a good solution but it has a memory/cpu limits that doesn't fit.
Kubernetes Job - Inefficient when the script itself takes only a few seconds to run.</p>
<p>Are there any other solutions to run high load of requests with cpu/memory intensive work? (I'm looking for a resource/infrastructure solution, not a software one.)</p>

CC BY-SA 4.0
2020-11-17T19:01:02.493Z
How often do you run the jobs? I.e. does it warrant an always-on *something* ready to pick up the next job?
MLu
CC BY-SA 4.0
2020-11-18T09:03:22.787Z
The ideal idea from Software engineering POV is to have a service that pulls jobs from a Q and process them.
But working with real life constraints it seems that its hard to scale up in this way.
Amorphis
<amazon-web-services><kubernetes><infrastructure>
Data processing infrastructure
Amorphis
# Data processing infrastructure

(Asked by: username_0 on 2020-11-17)

We are building a system for data science in AWS and our flow is pretty simple.

1. Get the data - From redis
2. get the model - From FSx/EBS
3. run.

This process script should take something like 5-10 seconds depends on the size of the data and model.
So far so good.

What we did so far is to build a python application (runs in kubernetes) that listen to a queue and runs the data science flow.
The problem we faced was Pythons memory leak. what ever we did and every approach we took with python a long running application that loads/unloads large objects will grow and grow, So we are trying a new approach , running the script on its own process each time a new process is created and closed when finished (still in kubernetes).

My question is
Is there a better way?
AWS Lambda - could be a good solution but it has a memory/cpu limits that doesn't fit.
Kubernetes Job - Inefficient when the script itself takes only a few seconds to run.

Are there any other solutions to run high load of requests with cpu/memory intensive work? (I'm looking for a resource/infrastructure solution, not a software one.)

**Tags:** amazon-web-services, kubernetes, infrastructure

> **Comments:**
> **username_1**: How often do you run the jobs? I.e. does it warrant an always-on *something* ready to pick up the next job?  
> **username_0**: The ideal idea from Software engineering POV is to have a service that pulls jobs from a Q and process them. But working with real life constraints it seems that its hard to scale up in this way.  

---

## 1 answers

### Answer by username_1 on 2020-11-18. Score: 1

The service can be *always-on* listening on the queue for jobs, or *single-job* started by the queue only when new work arrives.

The tradeoff is *startup time* for single-jobs vs *cost* for always-on service.

Define the requirements - number of jobs per day, max latency, ram/cpu needed, etc and then select the right technology that can deliver it in the most cost effective way.

From AWS Lambda, AWS Step Functions, through K8S Jobs and AWS Fargate to AWS EC2 on spot pricing, and more. Plenty of options, depends on the requirements.

Hope that helps :)


---

