c++ – Dimensionless self-expanding array as a kind of inexhaustible virtual memory

Question:

I am trying to implement a dimensionless self-expanding array myself, at first on 2Gb, then more. Those. when accessing an array element to check if there is a cluster in the RAM, and the cluster is issued, if not, so that the cluster is loaded from a file. If there are many clusters, the less used ones were unloaded. Those. like some kind of inexhaustible virtual memory. The goal is to cache the result of a selection from any DBMS, for example, if it returned a million rows.

Maybe there is a ready-made library that implements this through the IStream interface for working with files, or something like that? Some DBMS are 32bit, so that the solution can work in both 32bit and 64bit. VirtualAlloc is not suitable (32bit address space is limited).

Ideally something like this

class BigArea{


  operator char& [] (__int64 index) {
      char * tmp_buff;
     //...
      return &tmp_buff[ index & 4095]; // Т.к. блок памяти 4096 байт.
      }
  }

As a source of "inexhaustible" memory. Is there a library that will allow you to cache 2-4-10 Gb of memory by placing the "extra" on the disk, and the most used memory in RAM?

Answer:

Under WinApi, simply unload everything into a temporary file (the FILE_ATTRIBUTE_TEMPORARY flag), then, without closing the file, map the parts that need to be read into memory. Everything!

Windows (at least 7) ​​have a disk cache that caches frequently or recently read chunks of files in RAM. That is, it does exactly what you want to do. The FILE_ATTRIBUTE_TEMPORARY flag tells the file system that you want to create, which is typical, a temporary file, and therefore you need to keep the blocks of this file in the cache. If there is enough memory for the cache, of course.

I do not recommend using RAM for this. There is no controlled swap in WinApi, it is an automatic system-wide process. In other words, the swap starts when there is not enough memory for everyone, that is, when the whole system is already sitting. And pushing out 10 GB can easily lead to this.

Scroll to Top