I'm having trouble loading the airline data with read.table, even using xmem. R is making tons of tiny allocations.
I'm thinking of something with read.table syntax (like lazy.frame) that:
1 quickly inspects the file for number of lines and variable types
2 allocates a list of big variables
3 fills in the allocated variables by iterating through the file row-wise
this approach should use xmem reasonably well I think.
I'm having trouble loading the airline data with read.table, even using xmem. R is making tons of tiny allocations.
I'm thinking of something with read.table syntax (like lazy.frame) that:
1 quickly inspects the file for number of lines and variable types
2 allocates a list of big variables
3 fills in the allocated variables by iterating through the file row-wise
this approach should use xmem reasonably well I think.