wiki:Osaka_Krg

Version 2 (modified by rock, 16 years ago) (diff)

--

1.Limitation of Running OpenMP in Kerrighed

  • Features removed temporarily (Form 2.0.0 version)
    • KerFS
    • Distributed threads
    • Checkpoint
    • Dynamic streams.
  • Solution:
    • If we want to use OpenMP in Kerrighed, I think we must...


2.Kerrighed modules - Source Code Structure Analysis

2.1 Kerrighed 0.70 Totally 7 Modules

  • Kerrighed extension of Linux Kernel
    • Process & load balancing - Aragorn
    • Containers - Gandalf
    • Synchronization - Elrond
    • Ghosts - Nazgul
    • Communication - Gimli, Gloin
    • Tools - lluvatar

2.1.1 Probably the Key modules for this project

  • Gandalf module: Global memory management -> (For Container)
    • /kerrighed-2.3.0/libs/include/kerrighed_tools.h
    • /kerrighed-2.3.0/modules/tools/krg_services.h
    • /kerrighed-2.3.0/modules/mm/mmap_handler.h

/* Memory Part -> Distributed Shared Memory (DSM) , MPI&OpenMP */

  • Aragon module: Global process management -> (For Migration)
    • /kerrighed-2.3.0/modules/proc/distant_syscalls.c
    • /kerrighed-2.3.0/modules/scheduler/mosix_probe.c
    • /kerrighed-2.3.0/modules/tools/krg_arch.h
    • /kerrighed-2.3.0/modules/rpc/rpcid.h
    • /kerrighed-2.3.0/modules/epm/fork_delay.c
    • /kerrighed-2.3.0/modules/epm/migration.c
    • /kerrighed-2.3.0/modules/epm/procfs.c

/* Process & Load balancing Part -> MPI , OpenMP */

Process migration involves two key phases:

(1) migration of the process execution context
(2) maintenance of links between the migrated process and the resources

First task: Nazgul module ---> second task: Gandalf module -> container & linker

  • Elrond module: Synchronization tools for parallel applications -> (For Synchronization)
    • /kerrighed-2.3.0/modules/epm/ghost_process_api.h

/* Synchronization Part -> MPI , OpenMP */

  Elrond provides distributed locks, barriers, semaphores, and wait condition to processes in a implementation that    
  supports process checkpointing as well as migration between nodes.
  • Gimli:
    • /kerrighed-2.3.0/libs/libkerrighed/libcomm.c
    • /kerrighed-2.3.0/libs/include/comm.h
    • /kerrighed-2.3.0/libs/include/kerrighed_tools.h
    • /kerrighed-2.3.0/modules/arch/um/krg_um.c
    • /kerrighed-2.3.0/tools/krg_services.h
    • /kerrighed-2.3.0/tools/krg_arch.h
    • /kerrighed-2.3.0/modules/fs/kerfs/dir_io_linker.c
    • /kerrighed-2.3.0/modules/fs/kerfs/inode_io_linker.c
    • /kerrighed-2.3.0/modules/fs/kerfs/file_io_linker.c
   All these distributed services are based on Gimli for their communications. Gimli is a service providing high 
   performance communication inside the cluster and a complete abstraction of the communication system to kerrighed that 
   makes it portable on various interconnection networks.


2.2 Kerrighed 2.3.0 Totally 14 Modules

RPC
Hotplug
KDDM
Capability
ProcFS
Sync
IPC
FS
MM
Dynamic Streams
Ghost
Proc
EPM
Global Scheduler
  • Load Kerrighed Moudles Messages
    $ dmesg
    Start loading Kerrighed...
    Init Kerrighed worker(s)...Init Kerrighed low-level framework...
    Init kerrighed syscall mechanism
    Kerrighed tools - init module
    TIPC: Started in network mode
    TIPC: Own node address <1.1.2>, network identity 1
    Try to enable bearer on lo:<5>TIPC: Enabled bearer <eth:lo>, discovery domain <1.1.0>, priority 10
    ok
    Try to enable bearer on eth0:<5>TIPC: Enabled bearer <eth:eth0>, discovery domain <1.1.0>, priority 10
    ok
    Try to enable bearer on eth1:<4>TIPC: Bearer <eth:eth1> priority adjustment required 10->9
    TIPC: Enabled bearer <eth:eth1>, discovery domain <1.1.0>, priority 9
    ok
    RPC initialisation done
    Init Kerrighed low-level framework (nodeid 1) : done
    Init Kerrighed distributed services...
    KDDM initialisation : start
    KDDM set init
    KDDM set init : done
    KDDM initialisation done
    KerMM initialisation : start
    KerMM initialisation done
    DVFS initialisation : start
    FAF: initialisation : start
    FAF: initialisation : done
    DVFS initialisation done
    KerIPC initialisation : start
    KerIPC initialisation done
    Proc initialisation: start
    Proc initialisation: done
    EPM initialisation: start
    EPM initialisation: done
    Init Kerrighed distributed services: done
    Kerrighed... loaded!
    TIPC: Established link <1.1.2:eth0-1.1.3:eth0> on network plane B
    Kerrighed is running on 2 nodes
    
    


2.3 Private Variables

  • mmap (map pages of memory - by adding a MAP_LOCAL flag)
    • /kerrighed-2.3.0/libs/include/kerrighed_tools.h
    • /kerrighed-2.3.0/modules/tools/krg_services.h
    • /kerrighed-2.3.0/modules/mm/mman.h
   It is possible to ensure that memory will be allocated in the virtual address space of each thread of the process 
   without linking it to a container, thus ensuring that no coherence will be maintained between the different copies of  
   that memory segment across the different threads.


2.4 OpenMP on Kerrighed

  • Standard Pthread library replaced by krgthread library proving the Posix interface on Kerrighed Clusters.
  • Containers to support shared variables in the cluster.
  • A thread using kerrighed synchronization primitives can migrate and be checkpointed at any time, and except when blocked in the OS.

Attachments (2)

Download all attachments as: .zip