2012-10-24

Hadoop World 2012 (Keynotes)

• Big Answers - Mike Olson (Cloudera)
  • ​http://cloudera.com/impla - Apache License , Realtime Query Engine
  • Application : Agriculture , Social Security
• The End of the Data Warehouse - Ben Werther (Platfora)
  • Add Datawarehouse between Web Interface and Hadoop
  • In-Memory BI
• Moneyball for New York City - Michael Flowers (NYC Mayor's Office of Policy and Strategic Planning)
  • Introduce "Data Sensing Lab" in the venue. (based on Arduino)
  • Smart City - How to use NYC data more effectively
  • Fire Risk Inspection Analytic - 預測火災的風險有多高 -> 有效地降低了
  • 25 Years old staff , Start on 2005 PCs and MS Excel
  • Use Web (Wall) for collaberation editing
  • Grow and Enable Culture of Data-driven
• Thinking Big Together: Driving the Future of Data Science - Annika Jimenez (EMC Greenplum), Anthony Goldbloom (Kaggle)
  • Data, Technology, Application, Data Science
  • ​http://openchorus.org
  • Kaggle - in short of professional exprt - 每次會議都會說公司在徵才，也代表這個領域的人才缺乏。
  • ​http://greenplum.com/communities
• The Composite Database - Rich Hickey (Datomic)
  • Big Data 雜誌 -
  • Strata - 9:1 -> 投稿:接受
  • breakdown tranditional database : storage, index, query
  • Indexing as a component : Storage -> Indexing -> Ordered Storage
  • Uqery as a component :
  • Coordination as component
  • Other component : notification, memory indexes (liveness), caching
• The Democratization of Big Data: Bringing Hadoop to the Masses - James Markarian (Informatica)
  • 用分子表示是說明 Hadoop Ecosystem 的關聯性
  • coding model work, but hadoop ecosystem need a friendly interface
• Big Data Direct – The Era of Self-driven Big Data Exploration Sharmila - Shahani-Mulligan (ClearStory Data)
  • Ever-increasing complexity of sources - growing open data api - 資料的複雜度(complexity)愈來愈高
  • 混合 private data 與 public data 可以得到許多洞見(insight) - 舉了不同的案例(Ex. 水管, Web Site SEO)
  • $35 billion industry for databases and data containers
  • Future : new era to analyze big data easilly and balance it with judgment and experience
  • Solution Must Aid Human Insight - Big Data + Amplified Human Intelligence
  • Next Generation Visualization - Only One Part of the Answer ( 一張圖也只是為了解釋某個答案的一部份)
• Bringing the 'So What' to Big Data - Tim Estes (Digital Reasoning)
  • Machine Reading problem
  • It's about People understanding data to change and impove their lives
  • Big understanding > Big Data
  • Key : Machine Learning , Bare Metal, Application
  • People > Data
  • Ex. 支持國家安全的分析, 確保自由。
  • Ex. 從廣告找出人蛇集團，拯救未成年少女被綁架。
  • Ex. 降低金融風險
  • Mission > Consumerism
  • Three Key mission: (1) Security and Freedom of World (2) (3)

Hadoop World 2012 (Sessions)

• 10:50am Wednesday, 10/24/2012
• MapReduce Design Patterns, Donald Miner (EMC Greenplum)
  • 這本書感覺上是本不錯的程式設計參考指南，特別是會把該 Pattern 的特徵也用 SQL 或 Pig 語法呈獻。對於跨團隊溝通上有蠻好的幫助。
• 11:40am Wednesday, 10/24/2012
• Analyzing Millions of GitHub Commits: What Makes Developers Happy, Angry, and Everything in Between?, Ilya Grigorik (Google), Brian Doll (GitHub)
  • Ilya 是 Google BigQuery 的開發者(Cool~), Brian 是 Github 的 Marketing
  • (Ilya)
  • 動機：追蹤太多專案，很難虧得全貌！Global Timeline -> only if we can access github archive
  • ​http://githubarchive.org - data start from March 2012
  • (Think: 未來個人在 githube 的活動也許會成為獵人頭公司判斷這個人程式設計能力的參考 -> 程式設計人員自我行銷的方法)
  • Dremel (Paper) -> ​BigQuery

    $ wget http://data.githubarchive.org/2012-04-11-15.json.gz
    $ ruby flatten.rb 2012-04-11-15.json.gz > flat.csv.gz
    $ bg load github.timeline flat.csv.gz
    

  • bigquery is a SQL-like syntax
  • (Think: 可以將 BigQuery 用在 OpenData.TW 的一些應用上)
  • GitHub + BigQuery + MailChimp -> 每天用 crontab 跑 BigQuery
  • GitHub Data Challenge (Brian)
  • Ex. ​http://octoboard.com
  • (Think: 分析 github 的整體行為是否可以代表全球 Open Source 活動的特徵？ Ex. Private vs Public)
  • comment emotional -> 寫在註解中的情緒語言
  • 蠻有趣的分析 - Programming language associations
  • 關聯分析 - Github 熱門語言 與 StackOverflow 的問題個數
  • activity by country - commits per 100k people (提送數 / 人口數)
• 1:40pm Wednesday, 10/24/2012
• Facebook’s Large Scale Monitoring System Built on HBase - Liyin Tang (Facebook), Vinod Venkataraman (Facebook), Charles Thayer (Facebook)
  • Facebook ODS
    • Problem : (1) MySQL table size limitations (2) Sharding scheme created hotspots (3) Data growth
    • Lesson Learned - Locality : Spliting HBase and HDFS is not good!
  • Pitfalls
  • HBase 效能調校 - Takeaway 1~5
• 2:30pm Wednesday, 10/24/2012
• Bringing Real-Time, End-to-End Analytics into Everyday Use, Greg Khairallah (Intel), Vin Sharma (Intel)