<p>In your approach I see a few scalability problems - I'm judging from a <a href="https://cloud.google.com/appengine/docs/" rel="nofollow noreferrer">Google App Engine (GAE)</a> context, where scalability is achieved via breakdown of the work in small tasks/work items and strict limitation of the response time for a task execution:</p>

<ul>
<li>you're duplicating some piece of functionality: both scripts need to parse the same page, leading to your requirement #1</li>
<li>the task execution time for script A is fundamentally unbound, it can vary widely depending on the number of links discovered which need to be checked if already crawled</li>
</ul>

<p>I'd approach this a bit differently to avoid these issues. I don't even see a need for separate services for this particular job:</p>

<ul>
<li>the service pulls individual page URLs as separate tasks/work items</li>
<li>it checks if the URL was already crawled and, if so, does nothing, otherwise continues</li>
<li><p>gets the page content and parses it to extract 2 kinds of info:</p>

<ul>
<li>the list of other URLs to crawl, creating and enqueueing new work items for each one of them (it doesn't check here if they were already crawled, which may take a long time depending on the number of URLs)</li>
<li>the other info needed from the page</li>
</ul></li>
</ul>

<p>In this approach the duration of each task execution is reduced to the minimum necessary: one page parsing and only one check if a URL has already been crawled. And each page is only parsed once.</p>

<p>The number of pending work items would be the driver for scaling - if it goes above a level new instances can be started, if it drops below a level and multiple instances are running some may be stopped. If implemented as a GAE app the GAE infra can take care of that for you <a href="https://cloud.google.com/appengine/docs/flexible/ruby/configuring-your-app-with-app-yaml#automatic_scaling" rel="nofollow noreferrer">automatically</a>.</p>

<p>This solution can be implemented using a noSQL database (like <a href="https://cloud.google.com/appengine/docs/flexible/ruby/using-cloud-datastore" rel="nofollow noreferrer">Google Datastore</a>, in a GAE context), which can scale better than a SQL one.</p>

CC BY-SA 3.0
2017-11-22T19:58:13.070Z
Thanks for your answer Dan. Interesting approach! You're right that this can be done with one service. I'm really liking what I see with GAE. I might end up using them.  The only question left is req #3, how easy is it to spin up a new server with a different IP, with the same content?
Tommy Adey
CC BY-SA 3.0
2017-11-23T03:34:19.530Z
In the standard environment GAE an instance is started in just a few seconds, depending on the actual language used and implementation.  For my python apps it's somewhere between 1 and 2 seconds most of the time. It doesn't have an assigned IP, though - it's a PaaS solution, not an IaaS one (do you actually need one, as long as it serves the requests?) But you're using ruby, for which only the GAE flexible environment is available, which I'm not a user of, so I can't tell exactly how much it takes. But, if I understand correctly, it's still faster than booting a traditional IaaS server.
Dan Cornilescu
CC BY-SA 3.0
2017-11-23T03:59:45.927Z
I think my earlier comment might have not answered your question - GAE dynamic instances are automatically managed for you, there's nothing you need to do besides the scaling configuration. There is *some* (small) work for the manually scaling config. See https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed and https://cloud.google.com/appengine/docs/flexible/ruby/how-instances-are-managed
Dan Cornilescu
CC BY-SA 3.0
2017-11-23T06:57:06.620Z
App Engine sounds cool, but I might not be able to run selenium/headless chrome on it. Also is there a way to programmatically launch instances on request through an api?
Tommy Adey
CC BY-SA 3.0
2017-11-24T15:24:48.470Z
Yeah, running selenium/headless chrome won't work in standard env. It *might* work in the flexible one, eventually with a [custom runtime](https://cloud.google.com/appengine/docs/flexible/custom-runtimes/). So you could keep the logic in a simple standard env service which supports task queues and it would push the urls to be parsed as tasks  to a flex env service. Back to 2 services as you wanted :)
Dan Cornilescu
CC BY-SA 3.0
2017-11-25T21:22:28.613Z
I see. Thanks so much for your help Dan. I've learnt a ton from this convo. One last question, why do you prefer GCP to AWS?
Tommy Adey
CC BY-SA 3.0
2017-11-26T03:28:14.333Z
~2 years ago when I was looking for a PaaS solution AWS didn't have a one, GCP had GAE. I didn't look back since.
Dan Cornilescu
CC BY-SA 3.0
2017-11-26T09:23:51.123Z
That makes sense :)
Tommy Adey
CC BY-SA 3.0
2017-11-22T18:50:59.953Z
2017-11-22T18:50:59.953Z
Dan Cornilescu
<p>Is it too late to teach an old dog a new trick?</p>

<p>I've always been the guy with a monolithic application/codebase. </p>

<p>For more than 10 years now, I've gotten good at caching, writing good SQL queries, using Redis to store frequently called read data, using background jobs to perform updates, edits, long-running tasks, e.t.c</p>

<p>Then again, I haven't had to deal with more than 100 million rows in a db before.</p>

<p>Not until now, I've used up all my tricks and alas no significant improvement.</p>

<p>Now I'm coming for advice from you smart devops people. </p>

<p>For my current app, just recently I had a rails app, my DB, and some script in the /lib, Redis, Sidekiq, all on one, you guessed it, server.  </p>

<p>I read up on Docker and now I have Sidekiq, Redis, my app, and my db all on different containers. </p>

<p>Now, I'd like to split some functions on my app itself. This is where I need your help.</p>

<p>For example: </p>

<p>Script A: I have this ruby script(a ruby class) that crawls the web for backlink data. Script A only gathers links and removes links that have been parsed already. </p>

<p>Script B: Another script that gets the links script A collected and retrieves the information I want from that link. </p>

<p>Let's stop at that for now, so it doesn't get confusing. </p>

<p>I'd like to run these scripts on their own servers, and they would connect to my app via api's. </p>

<p>Both scripts are less than 1000 lines of code and are written in Ruby. </p>

<p>Requirements </p>

<ol>
<li>Both scripts require Selenium, Watir, and headless chrome to run. </li>
<li>The server must be able to autoscale as needed.</li>
<li>If the server is rate-limited or stuck in a link blackhole, I should be able to spin up a new server(host), with a different wan IP, with all the content of the script in as little time as possible. </li>
<li>The server should be affordable, as it's these are long running tasks. The script does not require a database as it only sends back important data back to my app via API. </li>
<li>In short, both scripts should be on their own server (host) and it should be easy to spin up a new one in no time. </li>
</ol>

<p>How do you suggest I go about it?</p>

<amazon-web-services><microservices><serverless><servers>
Teaching an old dog new tricks: Finally adopting the micro-service architecture
Tommy Adey
# Teaching an old dog new tricks: Finally adopting the micro-service architecture

(Asked by: username_0 on 2017-11-22)

Is it too late to teach an old dog a new trick?

I've always been the guy with a monolithic application/codebase.

For more than 10 years now, I've gotten good at caching, writing good SQL queries, using Redis to store frequently called read data, using background jobs to perform updates, edits, long-running tasks, e.t.c

Then again, I haven't had to deal with more than 100 million rows in a db before.

Not until now, I've used up all my tricks and alas no significant improvement.

Now I'm coming for advice from you smart devops people.

For my current app, just recently I had a rails app, my DB, and some script in the /lib, Redis, Sidekiq, all on one, you guessed it, server.

I read up on Docker and now I have Sidekiq, Redis, my app, and my db all on different containers.

Now, I'd like to split some functions on my app itself. This is where I need your help.

For example:

Script A: I have this ruby script(a ruby class) that crawls the web for backlink data. Script A only gathers links and removes links that have been parsed already.

Script B: Another script that gets the links script A collected and retrieves the information I want from that link.

Let's stop at that for now, so it doesn't get confusing.

I'd like to run these scripts on their own servers, and they would connect to my app via api's.

Both scripts are less than 1000 lines of code and are written in Ruby.

Requirements

1. Both scripts require Selenium, Watir, and headless chrome to run.
2. The server must be able to autoscale as needed.
3. If the server is rate-limited or stuck in a link blackhole, I should be able to spin up a new server(host), with a different wan IP, with all the content of the script in as little time as possible.
4. The server should be affordable, as it's these are long running tasks. The script does not require a database as it only sends back important data back to my app via API.
5. In short, both scripts should be on their own server (host) and it should be easy to spin up a new one in no time.

How do you suggest I go about it?

**Tags:** amazon-web-services, microservices, serverless, servers


---

## 1 answers

### Answer by username_1 on 2017-11-22. Score: 4 | Accepted answer

In your approach I see a few scalability problems - I'm judging from a [Google App Engine (GAE)](https://cloud.google.com/appengine/docs/) context, where scalability is achieved via breakdown of the work in small tasks/work items and strict limitation of the response time for a task execution:

* you're duplicating some piece of functionality: both scripts need to parse the same page, leading to your requirement #1
* the task execution time for script A is fundamentally unbound, it can vary widely depending on the number of links discovered which need to be checked if already crawled

I'd approach this a bit differently to avoid these issues. I don't even see a need for separate services for this particular job:

* the service pulls individual page URLs as separate tasks/work items
* it checks if the URL was already crawled and, if so, does nothing, otherwise continues
* gets the page content and parses it to extract 2 kinds of info:

  + the list of other URLs to crawl, creating and enqueueing new work items for each one of them (it doesn't check here if they were already crawled, which may take a long time depending on the number of URLs)
  + the other info needed from the page

In this approach the duration of each task execution is reduced to the minimum necessary: one page parsing and only one check if a URL has already been crawled. And each page is only parsed once.

The number of pending work items would be the driver for scaling - if it goes above a level new instances can be started, if it drops below a level and multiple instances are running some may be stopped. If implemented as a GAE app the GAE infra can take care of that for you [automatically](https://cloud.google.com/appengine/docs/flexible/ruby/configuring-your-app-with-app-yaml#automatic_scaling).

This solution can be implemented using a noSQL database (like [Google Datastore](https://cloud.google.com/appengine/docs/flexible/ruby/using-cloud-datastore), in a GAE context), which can scale better than a SQL one.

> **Comments:**
> **username_0**: Thanks for your answer Dan. Interesting approach! You're right that this can be done with one service. I'm really liking what I see with GAE. I might end up using them.  The only question left is req #3, how easy is it to spin up a new server with a different IP, with the same content?  
> **username_1**: In the standard environment GAE an instance is started in just a few seconds, depending on the actual language used and implementation.  For my python apps it's somewhere between 1 and 2 seconds most of the time. It doesn't have an assigned IP, though - it's a PaaS solution, not an IaaS one (do you actually need one, as long as it serves the requests?) But you're using ruby, for which only the GAE flexible environment is available, which I'm not a user of, so I can't tell exactly how much it takes. But, if I understand correctly, it's still faster than booting a traditional IaaS server.  
> **username_1**: I think my earlier comment might have not answered your question - GAE dynamic instances are automatically managed for you, there's nothing you need to do besides the scaling configuration. There is *some* (small) work for the manually scaling config. See https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed and https://cloud.google.com/appengine/docs/flexible/ruby/how-instances-are-managed  
> **username_0**: App Engine sounds cool, but I might not be able to run selenium/headless chrome on it. Also is there a way to programmatically launch instances on request through an api?  
> **username_1**: Yeah, running selenium/headless chrome won't work in standard env. It *might* work in the flexible one, eventually with a [custom runtime](https://cloud.google.com/appengine/docs/flexible/custom-runtimes/). So you could keep the logic in a simple standard env service which supports task queues and it would push the urls to be parsed as tasks  to a flex env service. Back to 2 services as you wanted :)  
> **username_0**: I see. Thanks so much for your help Dan. I've learnt a ton from this convo. One last question, why do you prefer GCP to AWS?  
> **username_1**: ~2 years ago when I was looking for a PaaS solution AWS didn't have a one, GCP had GAE. I didn't look back since.  
> **username_0**: That makes sense :)  

---

