16 comments on “R at 12,000 Cores

  1. This looks like a very interesting and exciting new project. I just pushed this up on r-bloggers.com and I hope you will find more contributors to this project.

    Good luck!


  2. Hi,

    Nice post, and extremely promising packages--as I already said in private mail. Now, could you explain to me how you measure the cost of Rcpp to b e 20mb? On my 64bit Linux system, libRcpp.so is 344kb, or roughly 1/60 of the size you claim. Did you forget to turn off the `-g` switch for debugging symbols?

    Cheers, Dirk

  3. Regarding your spooling up problems. I saw someone have similar issues when running parallel Python stuff. If you skip to about 19:00, you can hear him talk about his issue. I think I heard that they solved their problem, though I don't know how. You might want to get in touch to see if their problem is similar to yours and if their solution can be adapted.


    • Very interesting; I'll check this out later. Thanks.

      A big part of the problem is that the R interpreter is pretty bloated, and so dragging around a bunch of interpreted stuff that you aren't using while running jobs in batch doesn't make a lot of sense. So eventually, we're probably going to have to build a version of R that throws away a bunch of things that are unneeded for us.

      It's kind of funny though to be working at a scale where you have access to terabytes of memory, but 50mb is considered monstrously large.

  4. This is pretty impressive. I'm looking forward to see the modelling functions being broaded up.
    What seems to be supprisingly difficult about a distributed lm, if I may ask?

    By the way, do you also have a distributed data.frame?

  5. I don't understand why rcpp is a bottleneck. Are you using a shared filesystem for executables? Should rcpp not be loaded independently at each MPI node? At most the contention should be equal to P at each node, where P is the number of MPI processes at each node in the cluster.

Leave a Reply

Your email address will not be published. Required fields are marked *

* Copy This Password *

* Type Or Paste Password Here *