Right now the summary encoder hd5 generator generates all the features in memory and concatenates them before dumping them to hdf5. this means all the chunnks are kept in memory at the same time, which could run into bottlenecks.
This section of the code would likely need re-writing to batch the feature generation and I/O so only a subset of the chunks are kept in memory at any time.
Right now the summary encoder hd5 generator generates all the features in memory and concatenates them before dumping them to hdf5. this means all the chunnks are kept in memory at the same time, which could run into bottlenecks.
This section of the code would likely need re-writing to batch the feature generation and I/O so only a subset of the chunks are kept in memory at any time.