To call R from within Matlab, you will need to have R compiled as a shared library. This needs to be done when compiling R from source (or via one of the binary distributions that have support for this.) Download the R source from cran.r-pro