To: J3 13-231r1 From: Bill Long and John Reid Subject: Coarray teams Date: 2013 February 11 Reference: WG5/N1883, WG5/N1930, 12-201 1. Discussion ------------- This paper proposes syntax and semantics for coarray teams. 2. Requirements (copied from N1930) ----------------------------------- Teams provide a capability to restrict the image set of remote memory references, coarray allocations, and synchronizations to a subset of all the images of the program. This simplifies writing programs that involve segregated activities (parts of a climate model, for example) that might be more easily be written independently or may have already been written as independent programs. Teams also provide a mechanism for subdividing the computation for the sake of better performance (such as within local SMP domains). Finally, teams provide the capability to execute procedures (such as library procedures) that use coarrays internally on a subset of the images of a program. T1: When a block of code is executed on images executing as a team, it should execute on those images as if the program contained no other images. This has the following implications: 1. Image indices shall be relative to the team, starting at 1 and ending with the number of images in the team. 2. Collective activities that would involve all images, such as SYNC ALL, allocation and deallocation of coarrays, collective subroutine execution, and inquiry intrinsics such as THIS_IMAGE and NUM_IMAGES shall be relative to the team. T2: While an image executes a statement it shall be a member of one and only one team. Access to variables on images outside the current team is not permitted. T3: It should be possible to split a team into mutually exclusive subsets that are themselves teams. This should be dynamic in order to allow different groupings of images during different stages of execution. It is desirable to have a compact mechanism for an image to specify which team it wishes to belong. T4: There shall be a construct mechanism for changing the current team, involving the synchronization of all members of the teams at the beginning and end of the construct. The construct shall support separate execution blocks based on team membership. The construct shall make apparent (both to the system and the programmer) where team execution begins and ends. T5: There shall be a type for variables identifying a team collection (probably an opaque derived type defined in the intrinsic module ISO_FORTRAN_ENV). T6: There needs to be a mechanism to find the image index relative to the set of an ancestor team. This might best be done by adding an optional argument to IMAGE_INDEX that specifies the ancestor team. T7: An allocatable coarray that is allocated within a team construct shall be deallocated before execution of the team construct terminates. An coarray that was allocated in a parent team shall not be deallocated within an child team construct. T8: The restriction that standard input is attached only to image 1 is unchanged, and the designated image is image 1 of the original set of images present at program startup. 3. Modifications to the requirements ------------------------------------ We have kept to the requirements except as follows T4. We do not provide explicit syntax for separate execution blocks based on team membership because this can easily be programmed with existing constructs, as is illustrated in an example. T6. An extension of THIS_IMAGE is the natural way to achieve this aim. T9. We have added facilities for the support of failed images. This addition needs consideration by WG5. 4. Syntax and Semantics -------------------- The derived type TEAM_TYPE is defined in the ISO_FORTRAN_ENV intrinsic module. Its components are private; no component is allocatable or a pointer. It is an extensible type with no type parameters. A variable of type TEAM_TYPE is a team variable. A team variable shall not appear in a variable definition context except as an argument to the intrinsic subroutine FORM_SUBTEAM or as an in an ALLOCATE statement that does not have a SOURCE= specifier. Every image of the program begins execution as part of the root team. At any point during execution of the program an image shall be part of one and only one team. That team is the current team. The result of the NUM_IMAGES() intrinsic function is the number of images in the current team. The image index values for images of the current team are in the set 1..NUM_IMAGES(). An intrinsic collective subroutine called FORM_SUBTEAM is provided. The intrinsic function SUBTEAM_COLOR provides the value of the argument COLOR in the call from the executing image of the subroutine FORM_SUBTEAM that formed the current team. The current team to which the execution image belongs is changed by execution of a CHANGE TEAM construct. A new named constant, STAT_FAILED_IMAGE, is defined in the ISO_FORTRAN_ENV intrinsic module. It shall have a nonzero value that is different from the value of STAT_STOPPED_IMAGE, STAT_LOCKED, STAT_LOCKED_OTHER_IMAGE, or STAT_UNLOCKED. The value STAT_FAILED_IMAGE becomes the value of the variable in a STAT= specifier, or STAT argument, in an image control statement if execution of the statement involves synchronization with, or accessing a variable on an image of the current team that has failed. A failed image is one for which references or definitions of data fail when that data should be accessible, or the image fails to respond as part of a collective activity. What constitutes a failed image is processor dependent, but usually is associated with a hardware failure of the processor, memory system, or interconnection network. The intrinsic function FAILED_IMAGES() provides the values of the image indices for the images in the current team that have failed. The intrinsic function THIS_IMAGE() is extended to have an optional argument TEAM. The TEAM argument shall be a team variable for an ancestor team of the current team. The result value is the image index of the executing image in that team. The intrinsic function THIS_IMAGE(COARRAY [, DIM]) is not extended. The intrinsic function NUM_IMAGES() is extended to have optional arguments TEAM and FAILED. The TEAM argument shall be a team variable for an ancestor team of the current team. The result value is the number of images in that team. If FAILED is absent, the total number of images is returned. If FAILED is present with the value true, the number of failed images is returned. If FAILED is present with the value false, the total number of images that have not failed is returned. Access to the standard input unit is available only on the image that has image index 1 in the root team. 5. Edits to J3/12-201: ---------------------- ------------------------------------ [5:7+] Add 3.2 <> scalar variable of type TEAM_TYPE (5.2) from the intrinsic module ISO_FORTRAN_ENV. 3.3 <> set of images that access each other's data (5.1). 3.3.1 <> the team that includes the executing image (5.1). 3.3.2 <> team from which the current team was formed by a call to FORM_SUBTEAM (7.3.13). 3.3.3 <> the current team when the program began execution (5.1). 3.3.4 <> a subset of the set of images in a team (5.1). ------------------------------------ [9] Replace this page by the following 5. Teams of images 5.1 Introduction A team of images is a set of images that access each other's data. The current team is the team that includes the executing image. All image indices are relative to the current team and data on images outside this team are inaccessible. Synchronization is possible only with other images of the team. Initially, the current team consists of all the images and this is known as the root team. A team is divided into subteams by invoking the intrinsic subroutine FORM_SUBTEAM. Each subteam is identified by an integer value known as its color. Information about the team to which current image belongs is stored in its team variable. Execution of a CHANGE TEAM statement changes the current team to the subteam for the executing image as determined by the specified team variable. Execution of an END TEAM statement changes the current team to the parent team. CHANGE TEAM constructs may be nested to any depth to give finer subdivisions. 5.2 TEAM_TYPE The derived type TEAM_TYPE is an extensible type with no type parameters. Its components are private; no component is allocatable or a pointer. A scalar of this type describes a team that includes the executing image. TEAM_TYPE is defined in the intrinsic module ISO_FORTRAN_ENV. A variable of type TEAM_TYPE is a team variable. A team variable shall not appear in a variable definition context except as an argument to the intrinsic subroutine FORM_SUBTEAM or as an in an ALLOCATE statement that does not have a SOURCE= specifier. 5.3 CHANGE TEAM construct The CHANGE TEAM construct changes the current team to which the executing image belongs. R501 change-team-construct is change-team-stmt block end-change-team-stmt R502 change-team-stmt is [team-construct-name:] [] [] CHANGE TEAM (team-variable) R503 end-change-team-stmt is END TEAM [team-construct-name] C501 (R501) A branch within a CHANGE TEAM construct shall not have a branch target that is outside the construct. C502 (R501) If the change-team-stmt of a change-team-construct specifies a team-construct-name, the corresponding end-change-team-stmt shall specify the same team-construct-name. If the change-team-stmt of a change-team-construct does not specify a team-construct-name, the corresponding end-change-team-stmt shall not specify a team-construct-name. C503 (R502) The team-variable shall be a scalar of the type TEAM_TYPE defined in the ISO_FORTRAN_ENV intrinsic module and shall not be a coarray. The value of the team-variable shall have been defined by invoking the intrinsic subroutine FORM_SUBTEAM. The team executing the change-team-stmt shall be the team that formed the team variable. The current team for the statements of the change-team block is the subteam that was specified for the executing image by the invokation FORM_SUBTEAM. An allocatable coarray that was allocated when execution of a change-team construct began shall not be deallocated during the execution of the construct. An allocatable coarray that is allocated when execution of a change-team construct completes is deallocated if it was not allocated when execution of the construct began. Both the CHANGE TEAM and END TEAM statements are image control statements. When a CHANGE TEAM statement is executed, there is an implicit synchronization of all images of the current team. On each image, execution of the segment following the statement is delayed until all the other images have executed the same statement the same number of times. When execution of a change-team block finishes, there is an implicit synchronization of all images of the parent team. On each image, execution of the segment following the END TEAM statement is delayed until all the other images have executed the same block the same number of times. Note: The deallocation of an allocatable coarray that was not alloacted at the beginning of a CHANGE TEAM construct, but was allocated at the end of the construct, occurs even for allocatable coarrays with the SAVE attribute. 5.4 STAT_FAILED_IMAGE The value of the default integer scalar constant STAT_FAILED_IMAGE is different from the value of STAT_STOPPED_IMAGE, STAT_LOCKED, STAT_LOCKED_OTHER_IMAGE, or STAT_UNLOCKED. Its value is assigned to the variable specified in a STAT=specifier or STAT argument in an image control statement if execution of the statement involves synchronization with an image of the current team that has failed or accessing a variable on an image of the current team that has failed. A failed image is one for which references or definitions of data fail when that data should be accessible, or the image fails to respond as part of a collective activity. What constitutes a failed image is processor dependent. NOTE 5.1 A failed image is usually associated with a hardware failure of the processor, memory system, or interconnection network. ------------------------------------ [13:1] Change "New intrinsic" to "Intrinsic". ------------------------------------ [13:3-5] Change "subroutines" to "procedures" twice and add these names to the list of intrinsics: SUBTEAM_COLOR, FAILED_IMAGES, FORM_SUBTEAM. ------------------------------------ [13:7+] Add a new paragraph to 7.1 General: "The intrinsic procedures THIS_IMAGE and NUM_IMAGES described in clause 13 of ISO/IEC 1539-1:2010 are extended as described in 7.4." ------------------------------------ [13:17] Change "subroutines" to "procedures". ------------------------------------ [15:12+] Add 7.3.6a SUBTEAM_COLOR ([TEAM]) Description. Subteam color. Class. Transformational function. Argument. TEAM (optional) shall be a scalar team variable. It shall describe an ancestor of the current team. It is an INTENT(IN) argument. Result Characteristics. Default integer scalar. Result Value. If TEAM is present, the value of the result is the value of the argument COLOR in the call of the subroutine FORM_SUBTEAM that gave TEAM its current value. If TEAM is absent and the current team is not the root team, the value of the result is the value of the argument COLOR in the call of the subroutine FORM_SUBTEAM that formed the current team. Otherwise, the value of the result is 1. Example. The following code illustrates the use of SUBTEAM_COLOR to control which code is executed. TYPE(TEAM_TYPE) :: ODD_EVEN : ME = THIS_IMAGE() CALL FORM_SUBTEAM ( 1+MOD(ME,2), ODD_EVEN ) CHANGE TEAM (ODD_EVEN) SELECT CASE (SUBTEAM_COLOR()) CASE (1) : ! Code for odd images in parent team CASE (2) : ! Code for even images in parent team END SELECT END TEAM ------------------------------------ [17:44+] Add 7.3.12 FAILED_IMAGES ([KIND]) Description. Indices of failed images. Class. Transformational function. Argument. KIND (optional) shall be a scalar integer constant expression. Result Characteristics. Integer. If KIND is present, the kind type parameter is that specified by the value of KIND; otherwise, the kind type parameter is that of default integer type. The result is an array of rank one whose size is equal to the number of failed images. Result. The elements of the result are the values of the image indices of the failed images in the current team, in numerically increasing order. Examples. If image 3 is the only failed image in the current team FAILED_IMAGES() has the value [3]. If there are no failed images in the current team FAILED_IMAGES() is a zero-sized array. 7.3.13 FORM_SUBTEAM (COLOR, SUBTEAM [,INDEX]) Description. Subdivide the current team into subteams. Class. Collective subroutine. Arguments. COLOR shall be scalar and of type default integer. It is an INTENT(IN) argument. Its value shall lie between 1 and the number of subteams and specifies the subteam to which the executing image will belong. SUBTEAM shall be a scalar team variable. It shall not describe the current team or an ancestor of the current team. It is an INTENT(OUT) argument. After successful execution of the subroutine, SUBTEAM describes the the subteam specified by COLOR. INDEX (optional) shall be scalar and of type integer. It is an INTENT(IN) argument. If INDEX is present, it shall be present on all images of the current team. Its value shall lie between 1 and the number of images in the subteam of the executing image and specifies the image index of the executing image in its subteam. No two images of the subteam shall specify the same value of INDEX. If INDEX is not present, the image indices of the images in the subteam are in the order of their image indices in the current team. On successful execution of FORM_SUBTEAM, the images of the current team will have been grouped into subteams and the SUBTEAM on each image will represent the subteam of which the executing image is a member. The subteam consists of all the images of the current team that invoked the subroutine with the same value of COLOR. Examples. CALL FORM_SUBTEAM ( 1+MOD(ME,2), ODD_EVEN ), with ME an integer with value IMAGE_INDEX() and ODD_EVEN of type TEAM_TYPE, divides the current team into two subteams according to whether the image index is even or odd. CALL FORM_SUBTEAM ( 1+MOD(ME,2), ODD_EVEN, (1+ME)/2) does this too, ensuring that the image indices of the subteams are ordered as in the current team. ------------------------------------ [17:44++] At the end of clause 7 add a new subclause: 7.4 Modified intrinsic procedures 7.4.1 NUM_IMAGES The description of the intrinsic function NUM_IMAGES in ISO/IEC 1539-1:2010 is changed by adding two optional arguments TEAM and FAILED and a modified result if either is present. The TEAM argument shall be a scalar team variable that describes an ancestor of the current team. The FAILED argument shall be a scalar LOGICAL argument. Its value determines whether the result is the number of failed images or the number of nonfailed images. The Result value is modified to have four cases as follows: Case(i) The result of NUM_IMAGES ( ) is the total number of images in the current team. Case (ii) The result of NUM_IMAGES (TEAM) is the total number of images in TEAM. Case(iii) If FAILED has the value true, the result of NUM_IMAGES (FAILED=FAILED) is the number of failed images in the current team and the result of NUM_IMAGES (TEAM,FAILED) is the number of failed images in TEAM. Case(iv) If FAILED has the value false, the result of NUM_IMAGES (FAILED=FAILED) is the number of working images in the current team and the result of NUM_IMAGES (TEAM,FAILED) is the number of working images in TEAM. " 7.4.2 THIS_IMAGE The description of the intrinsic function THIS_IMAGE( ) in ISO/IEC 1539-1:2010 is changed by adding an optional argument SUBTEAM_LEVEL and a modified result if SUBTEAM_LEVEL is present. The SUBTEAM_LEVEL argument shall be a scalar integer. It shall be nonnegative. If SUBTEAM_LEVEL is not present the result value is the image index of the invoking image in the current team. If SUBTEAM_LEVEL is present with a value less than or equal to the current nesting level of CHANGE TEAM construct executions, the result has the value of the image index that the invoking image had at the nesting level of CHANGE TEAM construct executions equal to SUBTEAM_LEVEL. Otherwise, the result has the value of the image index that the invoking image had in the root team. ------------------------------------ [19:9] Replace "Include clauses as needed" by {In paragraph 1 of the Introduction} After "informally known as Fortran 2008, plus the facilities defined in ISO/IEC TS 29113:2012" add "and ISO/IEC TS 18508:2014". {After paragraph 3 of the Introduction and after the paragraph added by ISO/IEC TS 29113:2012, insert new paragraph} ISO/IEC TS 18508 provides additional facilities for parallel programming: o teams provide a capability to restrict the image set of remote memory references, coarray allocations, and synchronizations to a subset of all the images of the program; o collective subroutines perform computations based on values on all the images, offering the possibility of efficient execution of reduction operations; o atomic memory operations provide powerful low-level primitives for synchronization of activities among images; o tagged events allow one-sided ordering of execution segments; and o features for the support of continued execution after one or more images have failed. ------------------------------------ [19:9+] Add a three new subclauses "8.2a Edits to clause 1 {In 1.3 Terms and definitions insert new terms} [Text for the terms in clause 3 of the TS go here, with new subclause numbers and corrected references.] 8.2b Edits to clause 2 {At the end of 2.3.4 Program execution insert two new paragraphs} [Text from 5.1 of this TS here, edited for style as needed] 8.2c Edits to clause 8 {In 8.1.1 paragraph 1, in the list of constructs} Following the BLOCK construct entry, insert " CHANGE TEAM construct;" {Following 8.1.4 BLOCK construct} [Insert 5.3 from this TS with rule and constraint numbers modified and references modified.] {In 8.5.1 Image control statements, paragraph 2} Add extra bullet point: " o CHANGE TEAM and END TEAM" " {In 8.5.7 STAT= and ERRMSG= specifiers in image control statements} Replace paragraphs 1-2 by "The appearance of a STAT= or ERRMSG= specifier in a CHANGE TEAM statement is treated as an appearance both there and in the corresponding END TEAM statement. If the STAT= specifier appears, successful invocation of the collective subroutine FORM_TEAMS or successful execution of a CHANGE TEAM, END TEAM, LOCK, SYNC ALL, SYNC IMAGES, SYNC MEMORY, or UNLOCK statement causes the specified variable to become defined with the value zero. If the STAT= specifier appears in an invocation of the collective subroutine FORM_TEAMS or in a CHANGE TEAM, END TEAM, SYNC ALL, or SYNC IMAGES statement and its execution involves synchronization with an image that has initiated termination or failed, the variable becomes defined with the value of the constant STAT_STOPPED_IMAGE (13.8.2.24) or STAT_FAILED_IMAGE, respectively, in the intrinsic module ISO FORTRAN_ENV (13.8.2), and the effect of executing the statement is otherwise the same as that of executing the SYNC MEMORY statement. If any other error condition occurs during execution of one of these, the variable becomes defined with a processor-dependent positive integer value that is different from the value of STAT_STOPPED_IMAGE or STAT_FAILED_IMAGE." Replace paragraphs 4-5 by "If the STAT= specifier does not appear and an error condition occurs during an invocation of the collective subroutine FORM_TEAMS or execution of a CHANGE TEAM, END TEAM, LOCK, SYNC ALL, SYNC IMAGES, SYNC MEMORY, or UNLOCK statement, error termination is initiated. If an ERRMSG= specifier appears in an invocation of the collective subroutine FORM_TEAMS or in a CHANGE TEAM, END TEAM, SYNC ALL, or SYNC IMAGES statement, and an error condition occurs during its execution, the processor shall assign an explanatory message to the specified variable. If no such condition occurs, the processor shall not change the value of the variable." ------------------------------------ [19:29+] Add to the existing edit for Table 13.1: "FAILED_IMAGES ([KIND]) T Indices of failed images. FORM_SUBTEAM (COLOR, SUBTEAM [,INDEX]) C Subdivide current team. SUBTEAM_COLOR ([TEAM]) T Subteam color." ------------------------------------ [19:29+] Add an edit for Table 13.1: {In 13.5 Standard generic intrinsic procedures, Table 13.1} In the entry for NUM_IMAGES(), change "()" to "([TEAM] [,FAILED])" In the entry for THIS_IMAGE(), change "()" to "([SUBTEAM_LEVEL])" ------------------------------------ [20.1] In the edit instructions change "7.3.10" to "7.3.13". ------------------------------------ [20:2+] Add new edits to clause 13 {In 13.7.126 NUM_IMAGES ( )} In the title, change "( )" to "([TEAM] [,FAILED])" Change paragraph 3 Argument to "Arguments. TEAM (optional) shall be a scalar team variable. It shall describe an ancestor of the current team. It is an INTENT(IN) argument. FAILED (optional) shall be a scalar LOGICAL variable. It is an INTENT(IN) argument." Change paragraph 5 Result Value to "Result Value. Case(i) The result of NUM_IMAGES ( ) is the total number of images in the current team. Case (ii) The result of NUM_IMAGES (TEAM) is the total number of images in TEAM. Case(iii) If FAILED has the value true, the result of NUM_IMAGES (FAILED=FAILED) is the number of failed images in the current team and the result of NUM_IMAGES (TEAM,FAILED) is the number of failed images in TEAM. Case(iv) If FAILED has the value false, the result of NUM_IMAGES (FAILED=FAILED) is the number of working images in the current team and the result of NUM_IMAGES (TEAM,FAILED) is the number of working images in TEAM. " {In 13.7.165 THIS_IMAGE ( ) or THIS_IMAGE (COARRAY [, DIM])} In the title, change "THIS_IMAGE ( )" to "THIS_IMAGE ([SUBTEAM_LEVEL])" In paragraph 3 Arguments, add new final item "SUBTEAM_LEVEL (optional) shall be a scalar integer. It shall be nonnegtive." In paragraph 5 Result Value, modify the first case to be "Case (i) : The result of THIS IMAGE ( ) is a scalar with a value equal to the index in the current team of the invoking image. and add new item "Case (ia): If the value of SUBTEAM_LEVEL is less than or equal to the current nesting level of CHANGE TEAM construct executions, the result of THIS_IMAGE (SUBTEAM_LEVEL) is a scalar with a value equal to the image index that the invoking image had at the nesting level of CHANGE TEAM construct executions equal to SUBTEAM_LEVEL. Case (ib): If the value of SUBTEAM_LEVEL is greater than the current nesting level of CHANGE TEAM construct executions, the result of THIS_IMAGE (SUBTEAM_LEVEL) is a scalar with a value equal to the image index that the invoking image had in the root team. {In 13.8.2 The ISO_FORTRAN_ENV intrinsic module} Move the paragraph and Note of subclause 5.4 of this Technical Specification to become subclause 13.8.2.21b of F2008. Move the two paragraphs of subclause 5.2 of this Technical Specification to become subclause 13.8.2.25 of F2008. ------------------------------------ [20:5+] In 8.4 Edits to Annex A, add a new processor dependency: "what constitutes a failed image;" ------------------------------------ [21:3+] In Annex A, add a new subclause "A.1 Clause 5 notes" and renumber A.1 to A.2. Example: Compute fluxes over land, sea and ice in different teams based on surface properties. Assumption: Each image deals with areas containing exactly one of the three surface types. SUBROUTINE COMPUTE_FLUXES(FLUX_MOM, FLUX_SENS, FLUX_LAT) USE ISO_FORTRAN_ENV REAL, INTENT(OUT) :: FLUX_MOM(:,:), FLUX_SENS(:,:), FLUX_LAT(:,:) INTEGER, PARAMETER :: LAND=1, SEA=2, ICE=3 CHARACTER(LEN=10) :: SURFACE_TYPE INTEGER :: MY_SURFACE_TYPE, N_IMAGE TYPE(TEAM_TYPE) :: SUBTEAM_SURFACE_TYPE DO N_IMAGE = 1, NUM_IMAGES() CALL GET_SURFACE_TYPE(N_IMAGE, SURFACE_TYPE) ! Surface type SELECT CASE (SURFACE_TYPE) ! of image N_IMAGE CASE ('LAND') MY_SURFACE_TYPE = LAND CASE ('SEA') MY_SURFACE_TYPE = SEA CASE ('ICE') MY_SURFACE_TYPE = ICE CASE DEFAULT ERROR_STOP END SELECT CALL FORM_SUBTEAM(MY_SURFACE_TYPE, SUBTEAM_SURFACE_TYPE) END DO CHANGE TEAM(SUBTEAM_SURFACE_TYPE) SELECT CASE (SUBTEAM_COLOR(SUBTEAM_SURFACE_TYPE)) CASE (LAND) ! Compute fluxes over land surface CALL COMPUTE_FLUXES_LAND(FLUX_MOM, FLUX_SENS, FLUX_LAT) CASE (SEA) ! Compute fluxes over sea surface CALL COMPUTE_FLUXES_SEA(FLUX_MOM, FLUX_SENS, FLUX_LAT) CASE (ICE) ! Compute fluxes over ice surface CALL COMPUTE_FLUXES_ICE(FLUX_MOM, FLUX_SENS, FLUX_LAT) CASE DEFAULT ERROR_STOP END SELECT END TEAM END SUBROUTINE COMPUTE_FLUXES