wiki:R-MPI_Problem

Version 13 (modified by rock, 17 years ago) (diff)

--

  • Error: cannot allocate vector of size
    You have, presumably, a 32-bit computer with a 4GB-per-process memory limit. 
    You have hit it (you get less than 4Gb as the OS services need some and there is some fragmentation). 
    The last failed allocation may be small, as you see, if you are allocating lots of smallish pieces.
    
    The only way to overcome that is to use a 64-bit OS and version of R. 
    


  • run in kerrighed, meet out of memory error message -> Guess that Kerrigthed MM module's scheduler not support R's operation
    • Enable Kerrighed USE_REMOTE_MEMORY
      $ R
      # justRMA original limit: 145 CELs
      setwd("/home/mnt/chris_rmpi/cel_data_mixed")
      rm(list=ls())
      dir()
      library(affy)
      CELNames<-dir(".", "[.]CEL")
      CELNames<-CELNames[1:305]
      data <- justRMA(filenames=CELNames)
      
      $ dmesg
      init.orig invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
       [<c013fd53>] out_of_memory+0x7f/0x19c
       [<c014124d>] __alloc_pages+0x208/0x292
       [<c0142742>] __do_page_cache_readahead+0xcf/0x1fc
       [<c03fe349>] io_schedule+0x22/0x2c
       [<c03fe5c4>] __wait_on_bit_lock+0x4b/0x52
       [<c013cf7a>] __lock_page+0x64/0x6b
       [<c0142b7d>] do_page_cache_readahead+0x41/0x4a
       [<c013f1ca>] filemap_nopage+0x159/0x304
       [<c0147ed6>] __handle_mm_fault+0x170/0x85d
       [<c01043af>] common_interrupt+0x23/0x28
       [<c0400e01>] do_page_fault+0x231/0x543
       [<c0108b13>] read_tsc+0x6/0x7
       [<c0400bd0>] do_page_fault+0x0/0x543
       [<c03ff81c>] error_code+0x7c/0x84
       [<c03f0033>] tipc_link_send_long_buf+0x2b/0x1eb
       =======================
      Mem-info:
      DMA per-cpu:
      CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      Normal per-cpu:
      CPU    0: Hot: hi:  186, btch:  31 usd:  57   Cold: hi:   62, btch:  15 usd:  61
      CPU    1: Hot: hi:  186, btch:  31 usd:  44   Cold: hi:   62, btch:  15 usd:  50
      CPU    2: Hot: hi:  186, btch:  31 usd: 156   Cold: hi:   62, btch:  15 usd:  14
      CPU    3: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  25
      HighMem per-cpu:
      CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  14
      CPU    1: Hot: hi:  186, btch:  31 usd:   9   Cold: hi:   62, btch:  15 usd:  17
      CPU    2: Hot: hi:  186, btch:  31 usd:  29   Cold: hi:   62, btch:  15 usd:  43
      CPU    3: Hot: hi:  186, btch:  31 usd: 149   Cold: hi:   62, btch:  15 usd:  56
      Active:493418 inactive:306 dirty:0 writeback:0 unstable:0 free:12001 slab:4191 mapped:2 pagetables:576
      DMA free:8032kB min:68kB low:84kB high:100kB active:2748kB inactive:0kB present:16256kB pages_scanned:13504 all_unreclaimable? yes
      lowmem_reserve[]: 0 873 1991
      Normal free:39508kB min:3744kB low:4680kB high:5616kB active:819348kB inactive:748kB present:894080kB pages_scanned:1380165 all_unreclaimable? yes
      lowmem_reserve[]: 0 0 8950
      HighMem free:464kB min:512kB low:1712kB high:2912kB active:1151464kB inactive:476kB present:1145604kB pages_scanned:3364854 all_unreclaimable? yes
      lowmem_reserve[]: 0 0 0
      DMA: 0*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8032kB
      Normal: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 15*2048kB 2*4096kB = 39508kB
      HighMem: 0*4kB 2*8kB 2*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 464kB
      Swap cache: add 0, delete 0, find 0/0, race 0+0
      Free swap  = 0kB
      Total swap = 0kB
      Free swap:            0kB
      518032 pages of RAM
      288656 pages of HIGHMEM
      5714 reserved pages
      173 pages shared
      0 pages swap cached
      0 pages dirty
      0 pages writeback
      2 pages mapped
      4191 pages slab
      576 pages pagetables
      
    • Don't enable Kerrighed USE_REMOTE_MEMORY
      $ dmesg
      R invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
      Out of memory: kill process 38507 (bash) score 15539 or a child
      Killed process 38515 (R)
      
    • Use swap to slove this problem or wait kerrighed memory stable functions (If your clinet' fs is nfs, must use client's physical disk to be swap parition)
      $ dd if=/dev/zero of=swap.img bs=1M count=8000
      $ mkswap swap.img
      $ swapon swap.img
      


  • Kerrighed's DSM can't support R. R only use local 2G ram(each node has 2G ram, total 7 nodes have 14G), if it need more than 2G memory, R will use swap.
    rock@node101:~$ free
                 total       used       free     shared    buffers     cached
    Mem:      14254464    2425076   11829388          0        792     173036
    -/+ buffers/cache:    2251248   12003216
    Swap:     41937396    2685232   39252164