wiki:R-MPI_Problem

Version 7 (modified by rock, 17 years ago) (diff)

--

  • run in kerrighed, meet out of memory error message -> Guess that Kerrigthed MM module's scheduler not support R's operation
    • Enable Kerrighed USE_REMOTE_MEMORY
      $ R
      # justRMA original limit: 145 CELs
      setwd("/home/mnt/chris_rmpi/cel_data_mixed")
      rm(list=ls())
      dir()
      library(affy)
      CELNames<-dir(".", "[.]CEL")
      CELNames<-CELNames[1:305]
      data <- justRMA(filenames=CELNames)
      
      $ dmesg
      init.orig invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
       [<c013fd53>] out_of_memory+0x7f/0x19c
       [<c014124d>] __alloc_pages+0x208/0x292
       [<c0142742>] __do_page_cache_readahead+0xcf/0x1fc
       [<c03fe349>] io_schedule+0x22/0x2c
       [<c03fe5c4>] __wait_on_bit_lock+0x4b/0x52
       [<c013cf7a>] __lock_page+0x64/0x6b
       [<c0142b7d>] do_page_cache_readahead+0x41/0x4a
       [<c013f1ca>] filemap_nopage+0x159/0x304
       [<c0147ed6>] __handle_mm_fault+0x170/0x85d
       [<c01043af>] common_interrupt+0x23/0x28
       [<c0400e01>] do_page_fault+0x231/0x543
       [<c0108b13>] read_tsc+0x6/0x7
       [<c0400bd0>] do_page_fault+0x0/0x543
       [<c03ff81c>] error_code+0x7c/0x84
       [<c03f0033>] tipc_link_send_long_buf+0x2b/0x1eb
       =======================
      Mem-info:
      DMA per-cpu:
      CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      Normal per-cpu:
      CPU    0: Hot: hi:  186, btch:  31 usd:  57   Cold: hi:   62, btch:  15 usd:  61
      CPU    1: Hot: hi:  186, btch:  31 usd:  44   Cold: hi:   62, btch:  15 usd:  50
      CPU    2: Hot: hi:  186, btch:  31 usd: 156   Cold: hi:   62, btch:  15 usd:  14
      CPU    3: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  25
      HighMem per-cpu:
      CPU    0: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  14
      CPU    1: Hot: hi:  186, btch:  31 usd:   9   Cold: hi:   62, btch:  15 usd:  17
      CPU    2: Hot: hi:  186, btch:  31 usd:  29   Cold: hi:   62, btch:  15 usd:  43
      CPU    3: Hot: hi:  186, btch:  31 usd: 149   Cold: hi:   62, btch:  15 usd:  56
      Active:493418 inactive:306 dirty:0 writeback:0 unstable:0 free:12001 slab:4191 mapped:2 pagetables:576
      DMA free:8032kB min:68kB low:84kB high:100kB active:2748kB inactive:0kB present:16256kB pages_scanned:13504 all_unreclaimable? yes
      lowmem_reserve[]: 0 873 1991
      Normal free:39508kB min:3744kB low:4680kB high:5616kB active:819348kB inactive:748kB present:894080kB pages_scanned:1380165 all_unreclaimable? yes
      lowmem_reserve[]: 0 0 8950
      HighMem free:464kB min:512kB low:1712kB high:2912kB active:1151464kB inactive:476kB present:1145604kB pages_scanned:3364854 all_unreclaimable? yes
      lowmem_reserve[]: 0 0 0
      DMA: 0*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8032kB
      Normal: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 15*2048kB 2*4096kB = 39508kB
      HighMem: 0*4kB 2*8kB 2*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 464kB
      Swap cache: add 0, delete 0, find 0/0, race 0+0
      Free swap  = 0kB
      Total swap = 0kB
      Free swap:            0kB
      518032 pages of RAM
      288656 pages of HIGHMEM
      5714 reserved pages
      173 pages shared
      0 pages swap cached
      0 pages dirty
      0 pages writeback
      2 pages mapped
      4191 pages slab
      576 pages pagetables
      
    • Don't enable Kerrighed USE_REMOTE_MEMORY
      $ dmesg
      R invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
       [<c013fd53>] out_of_memory+0x7f/0x19c
       [<c014124d>] __alloc_pages+0x208/0x292
       [<c0142742>] __do_page_cache_readahead+0xcf/0x1fc
       [<c0115e3b>] kmap_atomic+0x80/0x91
       [<c0115ca4>] kunmap_atomic+0x54/0x61
       [<c0140fbb>] get_page_from_freelist+0x26f/0x2f9
       [<c0142b7d>] do_page_cache_readahead+0x41/0x4a
       [<c013f1ca>] filemap_nopage+0x159/0x304
       [<c0147ed6>] __handle_mm_fault+0x170/0x85d
       [<c014ac3a>] vma_merge+0x168/0x178
       [<c014b026>] do_brk+0x16e/0x202
       [<c0400e01>] do_page_fault+0x231/0x543
       [<c0400bd0>] do_page_fault+0x0/0x543
       [<c03ff81c>] error_code+0x7c/0x84
       [<c03f0033>] tipc_link_send_long_buf+0x2b/0x1eb
       =======================
      Mem-info:
      DMA per-cpu:
      CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
      Normal per-cpu:
      CPU    0: Hot: hi:  186, btch:  31 usd: 160   Cold: hi:   62, btch:  15 usd:  58
      CPU    1: Hot: hi:  186, btch:  31 usd:  28   Cold: hi:   62, btch:  15 usd:  47
      CPU    2: Hot: hi:  186, btch:  31 usd:  24   Cold: hi:   62, btch:  15 usd:  48
      CPU    3: Hot: hi:  186, btch:  31 usd:  11   Cold: hi:   62, btch:  15 usd:  55
      HighMem per-cpu:
      CPU    0: Hot: hi:  186, btch:  31 usd:  25   Cold: hi:   62, btch:  15 usd:  61
      CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:  58
      CPU    2: Hot: hi:  186, btch:  31 usd:  30   Cold: hi:   62, btch:  15 usd:  58
      CPU    3: Hot: hi:  186, btch:  31 usd:  15   Cold: hi:   62, btch:  15 usd:  45
      Active:493781 inactive:44 dirty:12 writeback:0 unstable:0 free:11999 slab:4305 mapped:64 pagetables:573
      DMA free:8032kB min:68kB low:84kB high:100kB active:2748kB inactive:0kB present:16256kB pages_scanned:12864 all_unreclaimable? yes
      lowmem_reserve[]: 0 873 1991
      Normal free:39488kB min:3744kB low:4680kB high:5616kB active:820280kB inactive:128kB present:894080kB pages_scanned:1280802 all_unreclaimable? yes
      lowmem_reserve[]: 0 0 8950
      HighMem free:476kB min:512kB low:1712kB high:2912kB active:1152096kB inactive:48kB present:1145604kB pages_scanned:2358173 all_unreclaimable? yes
      lowmem_reserve[]: 0 0 0
      DMA: 0*4kB 0*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8032kB
      Normal: 2*4kB 1*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 15*2048kB 2*4096kB = 39488kB
      HighMem: 1*4kB 5*8kB 3*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 476kB
      Swap cache: add 0, delete 0, find 0/0, race 0+0
      Free swap  = 0kB
      Total swap = 0kB
      Free swap:            0kB
      518032 pages of RAM
      288656 pages of HIGHMEM
      5714 reserved pages
      76 pages shared
      0 pages swap cached
      12 pages dirty
      0 pages writeback
      64 pages mapped
      4305 pages slab
      573 pages pagetables
      Out of memory: kill process 38507 (bash) score 15539 or a child
      Killed process 38515 (R)
      
    • Use swap to slove this problem or wait kerrighed memory stable functions (If your clinet' fs is nfs, must use client's physical disk to be swap parition)
      $ dd if=/dev/zero of=swap.img bs=1M count=8000
      $ mkswap swap.img
      $ swapon swap.img