1.Limitation of Running OpenMP in Kerrighed
- Features removed temporarily (Form 2.0.0 version)
- KerFS
- Distributed threads
- Checkpoint
- Dynamic streams.
- Solution:
- If we want to use OpenMP in Kerrighed, I think we must...
2.Kerrighed modules - Source Code Structure Analysis
2.1 Kerrighed 0.70 Totally 7 Modules
- Kerrighed extension of Linux Kernel
- Process & load balancing - Aragorn
- Containers - Gandalf
- Synchronization - Elrond
- Ghosts - Nazgul
- Communication - Gimli, Gloin
- Tools - lluvatar
2.1.1 Probably the Key modules for this project
- Gandalf module: Global memory management -> (For Container)
- /kerrighed-2.3.0/libs/include/kerrighed_tools.h
- /kerrighed-2.3.0/modules/tools/krg_services.h
- /kerrighed-2.3.0/modules/mm/mmap_handler.h
/* Memory Part -> Distributed Shared Memory (DSM) , MPI&OpenMP */
- Aragon module: Global process management -> (For Migration)
- /kerrighed-2.3.0/modules/proc/distant_syscalls.c
- /kerrighed-2.3.0/modules/scheduler/mosix_probe.c
- /kerrighed-2.3.0/modules/tools/krg_arch.h
- /kerrighed-2.3.0/modules/rpc/rpcid.h
- /kerrighed-2.3.0/modules/epm/fork_delay.c
- /kerrighed-2.3.0/modules/epm/migration.c
- /kerrighed-2.3.0/modules/epm/procfs.c
/* Process & Load balancing Part -> MPI , OpenMP */
Process migration involves two key phases:
(1) migration of the process execution context
(2) maintenance of links between the migrated process and the resources
First task: Nazgul module ---> second task: Gandalf module -> container & linker
- Elrond module: Synchronization tools for parallel applications -> (For Synchronization)
- /kerrighed-2.3.0/modules/epm/ghost_process_api.h
- /kerrighed-2.3.0/modules/epm/ghost_process_api.h
/* Synchronization Part -> MPI , OpenMP */
Elrond provides distributed locks, barriers, semaphores, and wait condition to processes in a implementation that supports process checkpointing as well as migration between nodes.
- Gimli:
- /kerrighed-2.3.0/libs/libkerrighed/libcomm.c
- /kerrighed-2.3.0/libs/include/comm.h
- /kerrighed-2.3.0/libs/include/kerrighed_tools.h
- /kerrighed-2.3.0/modules/arch/um/krg_um.c
- /kerrighed-2.3.0/tools/krg_services.h
- /kerrighed-2.3.0/tools/krg_arch.h
- /kerrighed-2.3.0/modules/fs/kerfs/dir_io_linker.c
- /kerrighed-2.3.0/modules/fs/kerfs/inode_io_linker.c
- /kerrighed-2.3.0/modules/fs/kerfs/file_io_linker.c
All these distributed services are based on Gimli for their communications. Gimli is a service providing high performance communication inside the cluster and a complete abstraction of the communication system to kerrighed that makes it portable on various interconnection networks.
2.2 Kerrighed 2.3.0 Totally 14 Modules
RPC | |
Hotplug | |
KDDM | |
Capability | |
ProcFS | |
Sync | |
IPC | |
FS | |
MM | |
Dynamic Streams | |
Ghost | |
Proc | |
EPM | |
Global Scheduler |
- Load Kerrighed Moudles Messages
$ dmesg Start loading Kerrighed... Init Kerrighed worker(s)...Init Kerrighed low-level framework... Init kerrighed syscall mechanism Kerrighed tools - init module TIPC: Started in network mode TIPC: Own node address <1.1.2>, network identity 1 Try to enable bearer on lo:<5>TIPC: Enabled bearer <eth:lo>, discovery domain <1.1.0>, priority 10 ok Try to enable bearer on eth0:<5>TIPC: Enabled bearer <eth:eth0>, discovery domain <1.1.0>, priority 10 ok Try to enable bearer on eth1:<4>TIPC: Bearer <eth:eth1> priority adjustment required 10->9 TIPC: Enabled bearer <eth:eth1>, discovery domain <1.1.0>, priority 9 ok RPC initialisation done Init Kerrighed low-level framework (nodeid 1) : done Init Kerrighed distributed services... KDDM initialisation : start KDDM set init KDDM set init : done KDDM initialisation done KerMM initialisation : start KerMM initialisation done DVFS initialisation : start FAF: initialisation : start FAF: initialisation : done DVFS initialisation done KerIPC initialisation : start KerIPC initialisation done Proc initialisation: start Proc initialisation: done EPM initialisation: start EPM initialisation: done Init Kerrighed distributed services: done Kerrighed... loaded! TIPC: Established link <1.1.2:eth0-1.1.3:eth0> on network plane B Kerrighed is running on 2 nodes
2.3 Private Variables
- mmap (map pages of memory - by adding a MAP_LOCAL flag)
- /kerrighed-2.3.0/libs/include/kerrighed_tools.h
- /kerrighed-2.3.0/modules/tools/krg_services.h
- /kerrighed-2.3.0/modules/mm/mman.h
It is possible to ensure that memory will be allocated in the virtual address space of each thread of the process without linking it to a container, thus ensuring that no coherence will be maintained between the different copies of that memory segment across the different threads.
2.4 OpenMP on Kerrighed
- Standard Pthread library replaced by krgthread library proving the Posix interface on Kerrighed Clusters.
- Containers to support shared variables in the cluster.
- A thread using kerrighed synchronization primitives can migrate and be checkpointed at any time, and except when blocked in the OS.
Last modified 16 years ago
Last modified on Sep 4, 2008, 10:04:59 AM
Attachments (2)
- KerrighedArch.png (18.6 KB) - added by rock 16 years ago.
- krg0.7arch.png (17.9 KB) - added by rock 16 years ago.
Download all attachments as: .zip