<p>You might want to have a look at <a href="https://www.youtube.com/watch?v=tr2KcekX2kk" rel="nofollow noreferrer">this video</a>:</p>
<blockquote>
<p>&quot;Why We Built Our Own Distributed Column Store&quot; by Sam Stokes</p>
</blockquote>
<p>It illustrates how you could build a highly scalable columnar data store using SSDs rather than loading everything into RAM. They got really high performance out of this solution (try the Honeycomb demo to see it).</p>
<p>It might fit your use case?</p>

CC BY-SA 4.0
2021-10-29T01:11:46.907Z
Thanks for the reference. However, I will not use a traditional database, the index is a custom data structure serialized that I can only read if I load into RAM. Loading even from SSD its orders or magnitude slower than from RAM, so I also need short latency. Im looking for something that can store things in RAM and be horizontally scaled.
Pablo Carneiro Elias
CC BY-SA 4.0
2021-10-29T06:11:18.453Z
If you don't have to perform complex operations - e.g. only count, sum and average - then you could create a distributed system to perform partial operations to be merged into a final result. Otherwise you'll need plenty of RAM.

You could also check out Snowflake and see if their approach fits your use case.
Erik Norman
CC BY-SA 4.0
2021-10-29T08:01:53.527Z
I need to use an specialized algorithm for performing a KNN search. The index is actually a serialized index of a kdtree for binary descriptors (using hamming distance).
Pablo Carneiro Elias
CC BY-SA 4.0
2021-10-28T07:23:59.653Z
2021-10-30T07:47:15.723Z
2021-10-30T07:47:15.723Z
Erik Norman
<p>I am sorry for the newbye question. Im trying to figure out the best architecture / set of technologies to implement some very specific system:</p>
<ol>
<li><p>I have several databases indexes (used for searching data elements) that are broken down into small pieces. Each datababase index is actually a serialized data structure of mine and can be composed by N index file pieces. All pieces of all databases are in a s3 bucket and can be properly identified (pieces of one same database can be found by filename pattern).</p>
</li>
<li><p>For searching into one specific database, we need to load all the N file pieces from s3 into RAM memory for fast search. But all pieces might not fit one machine RAM, so theoretically, each database can / will be distributed over several machines.</p>
</li>
<li><p>Once a query is requested for a specific database, we need to send the query to all machines that contains any piece of the database (some might contain more than one piece, no problem), and then the machines will returns if it was found or not.</p>
</li>
<li><p>Some database might get to be unused for a while and will get unloaded from machines (the database index file pieces will be unloaded from RAM).</p>
</li>
<li><p>As any database index piece is not loaded when a query arrives, it must be loaded into one of the service machines that stores the pieces in memory.</p>
</li>
</ol>
<p>What I think it need to be done:</p>
<p>IDEA</p>
<ol>
<li><p>A master node will contain a map of loaded/unloaded databases and which pieces are loaded in which machine.</p>
</li>
<li><p>A request will arrive to master for a specific database, and it will check if all pieces for that database are loaded, if not, it will request loading sending a command to one of the available machines.</p>
</li>
<li><p>More machines are put available as soon as memory of the whole set of service machines get low (auto scaling)</p>
</li>
<li><p>After all pieces are loaded in some machines, the master updates the map of pieces and send the query to all of the corresponding machines that contains the database pieces.</p>
</li>
<li><p>The master will also have timestamps or a LRU for the pieces so it can unload after some time not being used.</p>
</li>
</ol>
<p>So for this solution we would have a master and service slaves. Slaves contains api to load/unload a piece and also for listing which piece it has loaded.</p>
<p>Drawbacks from this solution:</p>
<p>This solution involves a master node that will be a bottleneck and a context that reflects the service nodes and its stored in master. If master fails, how this context can be restored? Not so trivial. Possible fixes would be having lots of masters and many more service slaves. Also if a master dies it can recover the state by quering the slaves.</p>
<p>Can anyone give me any hint on the technologies I could use? I think amazon ECS can help.</p>
<p>Thanks in advance.</p>

CC BY-SA 4.0
2023-07-26T03:40:33.643Z
Would Elasticsearch do the trick? It does index sharding across multiple machines. You could decide on all-flash storage, RAMDisk, or Tier III storage for the shards and Elasticsearch has jobs to take indexes and prune out old data, unneeded segments of data or just take that data from every 5 minutes, to an hour summary.
James Shewey
<kubernetes><amazon-ecs><distributed-systems>
Proper technologies to create a horizontably scalable index search
Pablo Carneiro Elias
# Proper technologies to create a horizontably scalable index search

(Asked by: username_0 on 2021-10-28)

I am sorry for the newbye question. Im trying to figure out the best architecture / set of technologies to implement some very specific system:

1. I have several databases indexes (used for searching data elements) that are broken down into small pieces. Each datababase index is actually a serialized data structure of mine and can be composed by N index file pieces. All pieces of all databases are in a s3 bucket and can be properly identified (pieces of one same database can be found by filename pattern).
2. For searching into one specific database, we need to load all the N file pieces from s3 into RAM memory for fast search. But all pieces might not fit one machine RAM, so theoretically, each database can / will be distributed over several machines.
3. Once a query is requested for a specific database, we need to send the query to all machines that contains any piece of the database (some might contain more than one piece, no problem), and then the machines will returns if it was found or not.
4. Some database might get to be unused for a while and will get unloaded from machines (the database index file pieces will be unloaded from RAM).
5. As any database index piece is not loaded when a query arrives, it must be loaded into one of the service machines that stores the pieces in memory.

What I think it need to be done:

IDEA

1. A master node will contain a map of loaded/unloaded databases and which pieces are loaded in which machine.
2. A request will arrive to master for a specific database, and it will check if all pieces for that database are loaded, if not, it will request loading sending a command to one of the available machines.
3. More machines are put available as soon as memory of the whole set of service machines get low (auto scaling)
4. After all pieces are loaded in some machines, the master updates the map of pieces and send the query to all of the corresponding machines that contains the database pieces.
5. The master will also have timestamps or a LRU for the pieces so it can unload after some time not being used.

So for this solution we would have a master and service slaves. Slaves contains api to load/unload a piece and also for listing which piece it has loaded.

Drawbacks from this solution:

This solution involves a master node that will be a bottleneck and a context that reflects the service nodes and its stored in master. If master fails, how this context can be restored? Not so trivial. Possible fixes would be having lots of masters and many more service slaves. Also if a master dies it can recover the state by quering the slaves.

Can anyone give me any hint on the technologies I could use? I think amazon ECS can help.

Thanks in advance.

**Tags:** kubernetes, amazon-ecs, distributed-systems

> **Comments:**
> **username_1**: Would Elasticsearch do the trick? It does index sharding across multiple machines. You could decide on all-flash storage, RAMDisk, or Tier III storage for the shards and Elasticsearch has jobs to take indexes and prune out old data, unneeded segments of data or just take that data from every 5 minutes, to an hour summary.  

---

## 1 answers

### Answer by username_2 on 2021-10-28. Score: 0

You might want to have a look at [this video](https://www.youtube.com/watch?v=tr2KcekX2kk):

> "Why We Built Our Own Distributed Column Store" by Sam Stokes

It illustrates how you could build a highly scalable columnar data store using SSDs rather than loading everything into RAM. They got really high performance out of this solution (try the Honeycomb demo to see it).

It might fit your use case?

> **Comments:**
> **username_0**: Thanks for the reference. However, I will not use a traditional database, the index is a custom data structure serialized that I can only read if I load into RAM. Loading even from SSD its orders or magnitude slower than from RAM, so I also need short latency. Im looking for something that can store things in RAM and be horizontally scaled.  
> **username_2**: If you don't have to perform complex operations - e.g. only count, sum and average - then you could create a distributed system to perform partial operations to be merged into a final result. Otherwise you'll need plenty of RAM.  You could also check out Snowflake and see if their approach fits your use case.  
> **username_0**: I need to use an specialized algorithm for performing a KNN search. The index is actually a serialized index of a kdtree for binary descriptors (using hamming distance).  

---

