2011-05-03

Hadoop

• (2011-04-21 已經關閉)
  • [參考] ​IPv6 addrconf: prefix with wrong length 修改 /etc/sysctl.conf 加入

    net.ipv6.conf.all.accept_ra = 0
    net.ipv6.conf.all.autoconf = 0
    

    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    net.ipv6.conf.lo.disable_ipv6 = 1
    

    • 檢查方法

      cat /proc/sys/net/ipv6/conf/all/disable_ipv6
      
      0-enabled; 1-disabled
      

• 效能調校：Hadoop 目前不支援 IPv6 ，建議關閉。
  • ​Hadoop and IPv6
  • ​Disable IPv6 on Linux

    ~$ echo "options ipv6 disable=1" >> disable-ipv6.conf
    ~$ sudo mv disable-ipv6.conf /etc/modprobe.d/
    

Hadoop & GPU

• ​NVIDIA Powers Movie Magic and the Data Center

  YFrog’s GPU data center
  
  Robert Scoble interviewed YFrog CEO Jack Levin about their NVIDIA powered data center.  In the third video of this series Jack discusses how an experiment in using NVIDIA GPU cards on their Hadoop cluster affected their data analysis speed. It is truely a unique and innovative way to utilize parallelism, Hadopp and GPU’s – to “semantically analyze a lot of text”.  Mr. Levin talks about using the NVIDIA JDK as well as CUDA, their Compute Unified Device Architecture, the same parallel compute technology used by Plume – a fluid movement simulation tool used by Industrial Light and Magic.  Up next for this GPU use – Wall Street statistical analysis?
  

• 這裡提到 YFrog 的 CEO 正在實驗整合 Hadoop 與 GPU。從一些文章看起來，說會有 30 倍的效能提昇。
• 三個影片可以在 ​The best operationally-run web startup: ImageShack/YFrog (plus, first look at new YFrog) 這篇文章看到。
• 這裡提到的影片 - ​http://www.youtube.com/watch?v=UPUJUa4XgUs

Hadoop & CUDA

• ​CUDA on Hadoop @ Apache Hadoop Wiki

Memcache & OpenOnLoad

• ​Solarflare Accelerates Memcached Big Data Applications by 51 Percent
• 關於 OpenOnLoad - ​http://www.openonload.org/
• 從投影片看起來，有點類似 InfiniBand 的東西，可以透過特製網卡的 DMA 做資料讀寫加速。
• 文章中說這樣可以加速 Memcached ，或許是從網路層下手吧～