A few hours ago, Jens Oehlschlägel has announced on the R-help mailing list of the release of a new version of the ff package.
The ff package provides data structures that are stored on disk but behave (almost) as if they were in RAM by transparently mapping only a section (pagesize) in main memory – the effective virtual memory consumption per ff object.
c) applying an order to ff vectors and ffdf dataframes is substantially slower than in pure R because it involves disk-access AND sorting index positions (to avoid random access).
Win64 binaries and version 2.2.1 supporting Sun should appear during the next days on CRAN. For the impatient: checkout from r-forge with revision 67 or higher.
Non-Windows users: please note that you need to set appropriate values for options 'ffbatchbytes' and 'ffmaxbytes' yourself.
Note that virtual window support is deprecated now because it leads to too complex code. Let us know if you urgently need this and why.
Feedback, ideas and contributions appreciated. To those who offered code during the last months: please forgive us that integrating and documenting was not possible with this release.
o New option 'ffmaxbytes' which allows certain ff procedures like sorting using larger limit of RAM than 'ffbatchbytes' in chunked processing. Such higher limit is useful for (single-R-process) sorting compared to some multi-R-process chunked processing. It is a good idea to reduce 'ffmaxbytes' on slaves or avoid ff sorting there completely.
o New generic 'pagesize' with method 'pagesize.ff' which returns the current pagesize as defined on opening the ff object.
o virtual window support is deprecated. Let us know if you urgently need this and why.
Older version changes can be viewed in the package's NEWS/changelog.