[[PageOutline]]
== 1.Limitation of Running OpenMP in Kerrighed ==
 * Features removed temporarily (Form 2.0.0 version)
    * KerFS
    * Distributed threads
    * Checkpoint
    * Dynamic streams. 
 * Solution:
    * If we want to use OpenMP in Kerrighed, I think we must...

[[BR]]
== 2.Kerrighed modules - Source Code Structure Analysis ==
=== 2.1 Kerrighed 0.70 Totally 7 Modules ===
[[Image(krg0.7arch.png)]]
 * Kerrighed extension of Linux Kernel
   * Process & load balancing - '''Aragorn'''
   * Containers - '''Gandalf'''
   * Synchronization - '''Elrond'''
   * Ghosts - '''Nazgul'''
   * Communication - '''Gimli''', '''Gloin'''
   * Tools - '''lluvatar'''

==== 2.1.1 Probably the Key modules for this project ====

 * '''Gandalf module''': Global memory management -> ('''For Container''')
   * /kerrighed-2.3.0/libs/include/kerrighed_tools.h
   * /kerrighed-2.3.0/modules/tools/krg_services.h           
   * /kerrighed-2.3.0/modules/mm/mmap_handler.h[[BR]]

   '''/* Memory Part -> Distributed Shared Memory (DSM) , MPI&OpenMP */'''

 * '''Aragon module''': Global process management -> ('''For Migration''')
   * /kerrighed-2.3.0/modules/proc/distant_syscalls.c
   * /kerrighed-2.3.0/modules/scheduler/mosix_probe.c
   * /kerrighed-2.3.0/modules/tools/krg_arch.h
   * /kerrighed-2.3.0/modules/rpc/rpcid.h                    
   * /kerrighed-2.3.0/modules/epm/fork_delay.c
   * /kerrighed-2.3.0/modules/epm/migration.c
   * /kerrighed-2.3.0/modules/epm/procfs.c[[BR]]

   '''/* Process & Load balancing Part -> MPI , OpenMP */''' [[BR]]

   '''Process migration involves two key phases:''' [[BR]]
     (1) migration of the process execution context [[BR]]
     (2) maintenance of links between the migrated process and the resources [[BR]]
      
   First task: '''Nazgul module''' ---> second task: '''Gandalf module''' -> container & linker

 * '''Elrond module''': Synchronization tools for parallel applications -> ('''For Synchronization''')
   * /kerrighed-2.3.0/modules/epm/ghost_process_api.h[[BR]]
   
   '''/* Synchronization Part -> MPI , OpenMP */''' [[BR]]

{{{
  Elrond provides distributed locks, barriers, semaphores, and wait condition to processes in a implementation that    
  supports process checkpointing as well as migration between nodes.
}}}

 * '''Gimli''': 
   * /kerrighed-2.3.0/libs/libkerrighed/libcomm.c
   * /kerrighed-2.3.0/libs/include/comm.h
   * /kerrighed-2.3.0/libs/include/kerrighed_tools.h
   * /kerrighed-2.3.0/modules/arch/um/krg_um.c
   * /kerrighed-2.3.0/tools/krg_services.h
   * /kerrighed-2.3.0/tools/krg_arch.h
   * /kerrighed-2.3.0/modules/fs/kerfs/dir_io_linker.c
   * /kerrighed-2.3.0/modules/fs/kerfs/inode_io_linker.c
   * /kerrighed-2.3.0/modules/fs/kerfs/file_io_linker.c [[BR]]




{{{  
   All these distributed services are based on Gimli for their communications. Gimli is a service providing high 
   performance communication inside the cluster and a complete abstraction of the communication system to kerrighed that 
   makes it portable on various interconnection networks.
}}}



[[BR]]
=== 2.2 Kerrighed 2.3.0 Totally 14 Modules ===
[[Image(KerrighedArch.png)]]
|| RPC               ||               ||
|| Hotplug           ||               ||
|| KDDM              ||               ||
|| Capability        ||               ||
|| ProcFS            ||               ||
|| Sync              ||               ||
|| IPC               ||               ||
|| FS                ||               ||
|| MM                ||               ||
|| Dynamic Streams   ||               ||
|| Ghost             ||               ||
|| Proc              ||               ||
|| EPM               ||               ||
|| Global Scheduler  ||               ||
 * Load Kerrighed Moudles Messages
{{{
$ dmesg
Start loading Kerrighed...
Init Kerrighed worker(s)...Init Kerrighed low-level framework...
Init kerrighed syscall mechanism
Kerrighed tools - init module
TIPC: Started in network mode
TIPC: Own node address <1.1.2>, network identity 1
Try to enable bearer on lo:<5>TIPC: Enabled bearer <eth:lo>, discovery domain <1.1.0>, priority 10
ok
Try to enable bearer on eth0:<5>TIPC: Enabled bearer <eth:eth0>, discovery domain <1.1.0>, priority 10
ok
Try to enable bearer on eth1:<4>TIPC: Bearer <eth:eth1> priority adjustment required 10->9
TIPC: Enabled bearer <eth:eth1>, discovery domain <1.1.0>, priority 9
ok
RPC initialisation done
Init Kerrighed low-level framework (nodeid 1) : done
Init Kerrighed distributed services...
KDDM initialisation : start
KDDM set init
KDDM set init : done
KDDM initialisation done
KerMM initialisation : start
KerMM initialisation done
DVFS initialisation : start
FAF: initialisation : start
FAF: initialisation : done
DVFS initialisation done
KerIPC initialisation : start
KerIPC initialisation done
Proc initialisation: start
Proc initialisation: done
EPM initialisation: start
EPM initialisation: done
Init Kerrighed distributed services: done
Kerrighed... loaded!
TIPC: Established link <1.1.2:eth0-1.1.3:eth0> on network plane B
Kerrighed is running on 2 nodes

}}}
[[BR]]
=== 2.3 Private Variables ===
 * '''mmap''' (map pages of memory - by adding a MAP_LOCAL flag) 
   * /kerrighed-2.3.0/libs/include/kerrighed_tools.h
   * /kerrighed-2.3.0/modules/tools/krg_services.h
   * /kerrighed-2.3.0/modules/mm/mman.h

{{{
   It is possible to ensure that memory will be allocated in the virtual address space of each thread of the process 
   without linking it to a container, thus ensuring that no coherence will be maintained between the different copies of  
   that memory segment across the different threads.
}}}

[[BR]]
=== 2.4 OpenMP on Kerrighed ===
 * Standard '''Pthread''' library replaced by '''krgthread''' library proving the Posix interface on Kerrighed Clusters.
 * Containers to support shared variables in the cluster.
 * A thread using kerrighed synchronization primitives can migrate and be checkpointed at any time, and except when 
   blocked in the OS.