Message Passing Interface (MPI), SHMEM, and Unified Parallel C (UPC) high level APIs and programming models that are implemented and supported by SGI that support access to GRU functionality. For more information, see mpi(1), shmem(3), or sgiupc(1) man pages and the Message Passing Toolkit (MPT) User's Guide and Unified Parallel C (UPC) User's Guide.

The Direct GRU Access API has four components, as follows:

The UV global reference unit (GRU) has control block (CB) and data segment (DSEG) resources associated with it. User applications need to allocate CB resources and usually DSEG resources for use in GRU memory access functions.

There are two categories of GRU resources used by any thread: temporarily and permanently allocated. A program starts running with all the available GRU resources being in the temporary pool until some resources are allocated permanently via the gru_pallocate() function.

The preferred way to get access to all the GRU temporary CBs and DSEG is through the use of the lightweight gru_temp_reserve() and gru_temp_release() functions. These functions should wrap any use of the GRU memory access fuctions, with an exception to be described later.

#include <gru_alloc.h>

void gru_temp_reserve(gru_alloc_thdata_t *gat);

typedef struct {
      gru_segment_t       *gruseg;
      gru_control_block_t *cbp;
      void                *dsegp;
      int                 cb_cnt;
      int                 dseg_size;
} gru_alloc_thdata_t;

The gru_alloc_thdata_t structure returned from this function will describe the GRU resources available for use until the next call to gru_temp_release().

The following code example shows a GRU memory access function gru_gamirr() being called after which the gru_temp_reserve() function reserves the GRU resources, and before the gru_wait_abort() function waits for completion of the operation. Then, followed by a call to gru_temp_release() to release the temporary GRU resources.

gru_alloc_thdata_t gat;
gru_temp_reserve(&gat);
gru_gamirr( gat.cbp, EOP_IRR_DECZ, address, XTYPE_DW, IMA_CB_DELAY);
gru_wait_abort(gat.cbp);
gru_temp_release();

The effect of the gru_temp_reserve() andgru_temp_release() functions is thread-private, so related POSIX threads or OpenMP threads could be executing the above sequence, concurrently.

An alternative allocation scheme is permanent allocation. The gru_pallocate() function returns CB and DSEG resources that can be used at any time thereafter. This can simplify the allocation strategy but it has the disadvantage of reducing the number of GRU resources that can be used by other software. An example would be a call to gru_bcopy() which allows you to pass a DSEG work buffer of any size. The achieved bandwidth for gru_bcopy() is higher with larger DSEG work buffers.

You can find more detailed information in the following man pages:

Use the man(1) command to view these man pages online. For your convenience, copies of the GRU-related man pages are included in the following section.

This section contains GRU-related man pages.

#include <gru_alloc.h>

void gru_temp_reserve(gru_alloc_thdata_t *gat);
int gru_temp_reserve_try(gru_alloc_thdata_t *gat);
void gru_temp_release(void);

typedef struct {
      gru_segment_t       *gruseg;
      gru_control_block_t *cbp;
      void                *dsegp;
      int                 cb_cnt;
      int                 dseg_size;
} gru_alloc_thdata_t;

#include <gru_alloc.h>

int gru_pallocate(int num_cbs, int dseg_sz, gru_segment_t **gruseg,
      int *cbnum, void **dseg);

int gru_pallocate_dseg_granularity(void);

The GRU memory access functions perform GRU operations that include memory read, memory write, memory-to-memory copies, and atomic memory operations, and so on. These functions use an ordinary virtual address or a GRU-mapped virtual address to reference the remote memory.

The interfaces to these functions are viewable in the uv/gru/gru_instructions.h header file installed by the gru-devel RPM. Use a hardware reference manual to get functional descriptions of these operations. TBD? What hardware reference manual?

The following code example of a GRU memory access function illustrates the basic call structure.

static inline 
void gru_vload(gru_control_block_t *cb, void *mem_addr,
         unsigned int tri0, unsigned char xtype, unsigned long nelem,
         unsigned long stride, unsigned long hints);

Arguments are:
	cb	 - pointer to CB
	mem_addr - address of targeted memory
	tri0     - index to DSEG buffer.  Compute it
		   using gru_get_tri().
	xtype	 - log2 of data type byte size (XTYPE_B ...)
	nelem	 - number of elements to transfer
	stride   - memory stride, scaled in elements
	hints    - IMA_CB_DELAY is commonly used

All memory access operations are asynchronous. The wait functions, such as, gru_wait_abort(), specify the CB handle and are used to wait to completion.

The XPMEM interface can map a virtual address range in one process into the GRU-mapped virtual address in another process. The XPMEM interface was designed to meet the needs of MPI and SHMEM implementations and provide ways to map any data region. As a GRU API user, you need to find a way to map the needed memory regions into the processes or threads involved. The Linux operating sytems offers many options for doing this, as follows:

These methods are the likely first choice for most potential GRU users.

The sn/xpmem.h header file installed by the xpmem-devel-noship RPM has interface definitions for all the XPMEM functions.

The following example shows the main XPMEM functions:

extern __s64 xpmem_make_2(void *, size_t, int, void *);
extern int xpmem_remove_2(__s64);
extern __s64 xpmem_get_2(__s64, int, int, void *);
extern int xpmem_release_2(__s64);
extern void *xpmem_attach_2(__s64, off_t, size_t, void *);
extern void *xpmem_attach_high_2(__s64, off_t, size_t, void *);
extern int xpmem_detach_2(void *, size_t size);
extern void *xpmem_reserve_high_2(size_t, size_t);
extern int xpmem_unreserve_high_2(void *, size_t);

For more information on using XPMEM, see the SGI internal XPMEM API document. TBD

The MPT libmpi library uses XPMEM to cross-map virtual memory between all the processes in an MPI job. Several functions are available to lookup mapped virtual addresses that are pre-attached in the virtual address space of a process by MPI. The addresses returned by the lookups may be passed to the GRU library functions.

Not all GRU API users can require their code to execute in an MPI job, but if you do, you may find the MPT address mapping functions are a convenient way to reference remote data arrays and objects.

The MPT address mapping functions are shown below. They reference ordinary virtual addresses or addresses of symmetric data objects. Symmetric data is static data or array-defined in the intro_shmem(3) man page.

The following example shows an MPI_SGI_gam_type:

#include <mpi_ext.h>
   
int
MPI_SGI_gam_type(int rank, MPI_Comm comm)
      
Return value is the XPMEM accessibility of the specified rank.
    
  MPI_GAM_NONE       - not referenceable by load/store or GRU
  MPI_GAM_CPU_NONCOH - Altix 3700 noncoherent
  MPI_GAM_CPU        - if referencable by load/store only
  MPI_GAM_GRU        - if referenceable by GRU only
  MPI_GAM_CPU_PREF   - if referenceable by either load/store
                         or GRU, preferred by load/store
  MPI_GAM_GRU_PREF   - if referenceable by either load/store
                         or GRU, preferred by GRU

The MPT address mapping functions are influenced by the MPI_GSM_NEIGHBORHOOD environment variable. This variable may be used to specify the "neighborhood size" for shared memory accesses. Contigous groups of ranks within a hostcan be considered to be in the same neighborhood. The MPI_GSM_NEIGHBORHOOD variable specifies the size of these neighborhoods, as follows:

When MPI_GSM_NEIGHBORHOOD is not set, the neighborhood size defaults to all ranks in the current host.

#include <mpi_ext.h>

void * MPI_SGI_gam_ptr(void *rem_addr, size_t len, int remote_rank,
  MPI_Comm comm, int acc_mode);

void *MPI_SGI_symmetric_addr(void *local_addr, size_t len,
	    int remote_rank, MPI_Comm comm)

#include <mpp/shmem.h>

       void *shmem_ptr(void *target, int pe);