1.Limitation of Running OpenMP in Kerrighed

• Features removed temporarily (Form 2.0.0 version)
  • KerFS
  • Distributed threads
  • Checkpoint
  • Dynamic streams.
• Solution:
  • If we want to use OpenMP in Kerrighed, I think we must...

2.Kerrighed modules - Source Code Structure Analysis

2.1 Kerrighed 0.70 Totally 7 Modules

• Kerrighed extension of Linux Kernel
  • Process & load balancing - Aragorn
  • Containers - Gandalf
  • Synchronization - Elrond
  • Ghosts - Nazgul
  • Communication - Gimli, Gloin
  • Tools - lluvatar

2.1.1 Probably the Key modules for this project

• Gandalf module: Global memory management -> (For Container)
  • /kerrighed-2.3.0/libs/include/kerrighed_tools.h
  • /kerrighed-2.3.0/modules/tools/krg_services.h
  • /kerrighed-2.3.0/modules/mm/mmap_handler.h
    

/* Memory Part -> Distributed Shared Memory (DSM) , MPI&OpenMP */

• Aragon module: Global process management -> (For Migration)
  • /kerrighed-2.3.0/modules/proc/distant_syscalls.c
  • /kerrighed-2.3.0/modules/scheduler/mosix_probe.c
  • /kerrighed-2.3.0/modules/tools/krg_arch.h
  • /kerrighed-2.3.0/modules/rpc/rpcid.h
  • /kerrighed-2.3.0/modules/epm/fork_delay.c
  • /kerrighed-2.3.0/modules/epm/migration.c
  • /kerrighed-2.3.0/modules/epm/procfs.c
    

/* Process & Load balancing Part -> MPI , OpenMP */

Process migration involves two key phases:

(1) migration of the process execution context
(2) maintenance of links between the migrated process and the resources

First task: Nazgul module ---> second task: Gandalf module -> container & linker

• Elrond module: Synchronization tools for parallel applications -> (For Synchronization)
  • /kerrighed-2.3.0/modules/epm/ghost_process_api.h
    

/* Synchronization Part -> MPI , OpenMP */

  Elrond provides distributed locks, barriers, semaphores, and wait condition to processes in a implementation that    
  supports process checkpointing as well as migration between nodes.

• Gimli:
  • /kerrighed-2.3.0/libs/libkerrighed/libcomm.c
  • /kerrighed-2.3.0/libs/include/comm.h
  • /kerrighed-2.3.0/libs/include/kerrighed_tools.h
  • /kerrighed-2.3.0/modules/arch/um/krg_um.c
  • /kerrighed-2.3.0/tools/krg_services.h
  • /kerrighed-2.3.0/tools/krg_arch.h
  • /kerrighed-2.3.0/modules/fs/kerfs/dir_io_linker.c
  • /kerrighed-2.3.0/modules/fs/kerfs/inode_io_linker.c
  • /kerrighed-2.3.0/modules/fs/kerfs/file_io_linker.c
    

   All these distributed services are based on Gimli for their communications. Gimli is a service providing high 
   performance communication inside the cluster and a complete abstraction of the communication system to kerrighed that 
   makes it portable on various interconnection networks.

2.2 Kerrighed 2.3.0 Totally 14 Modules

RPC
Hotplug
KDDM
Capability
ProcFS
Sync
IPC
FS
MM
Dynamic Streams
Ghost
Proc
EPM
Global Scheduler

• Load Kerrighed Moudles Messages

  $ dmesg
  Start loading Kerrighed...
  Init Kerrighed worker(s)...Init Kerrighed low-level framework...
  Init kerrighed syscall mechanism
  Kerrighed tools - init module
  TIPC: Started in network mode
  TIPC: Own node address <1.1.2>, network identity 1
  Try to enable bearer on lo:<5>TIPC: Enabled bearer <eth:lo>, discovery domain <1.1.0>, priority 10
  ok
  Try to enable bearer on eth0:<5>TIPC: Enabled bearer <eth:eth0>, discovery domain <1.1.0>, priority 10
  ok
  Try to enable bearer on eth1:<4>TIPC: Bearer <eth:eth1> priority adjustment required 10->9
  TIPC: Enabled bearer <eth:eth1>, discovery domain <1.1.0>, priority 9
  ok
  RPC initialisation done
  Init Kerrighed low-level framework (nodeid 1) : done
  Init Kerrighed distributed services...
  KDDM initialisation : start
  KDDM set init
  KDDM set init : done
  KDDM initialisation done
  KerMM initialisation : start
  KerMM initialisation done
  DVFS initialisation : start
  FAF: initialisation : start
  FAF: initialisation : done
  DVFS initialisation done
  KerIPC initialisation : start
  KerIPC initialisation done
  Proc initialisation: start
  Proc initialisation: done
  EPM initialisation: start
  EPM initialisation: done
  Init Kerrighed distributed services: done
  Kerrighed... loaded!
  TIPC: Established link <1.1.2:eth0-1.1.3:eth0> on network plane B
  Kerrighed is running on 2 nodes
  
  

2.3 Private Variables

• mmap (map pages of memory - by adding a MAP_LOCAL flag)
  • /kerrighed-2.3.0/libs/include/kerrighed_tools.h
  • /kerrighed-2.3.0/modules/tools/krg_services.h
  • /kerrighed-2.3.0/modules/mm/mman.h

   It is possible to ensure that memory will be allocated in the virtual address space of each thread of the process 
   without linking it to a container, thus ensuring that no coherence will be maintained between the different copies of  
   that memory segment across the different threads.

2.4 OpenMP on Kerrighed

• Standard Pthread library replaced by krgthread library proving the Posix interface on Kerrighed Clusters.
• Containers to support shared variables in the cluster.
• A thread using kerrighed synchronization primitives can migrate and be checkpointed at any time, and except when blocked in the OS.