J3/06-122r1 Date: 16-February-2006 To: J3 From: Bill Long and John Reid Subject: Edits for co-arrays References: Feature UK-001, J3/05-208, J3/05-272r2 ----------------------------------------------------------------------- Begin Preface ============= This paper represents a draft of the edits for feature uk-001, co-arrays. The major changes with respect to 06-122 are: 1) BACKSPACE, REWIND and ENDFILE are not allowed for files connected to more than one image. 2) Added SYNC_TEAM statement, a FORM_TEAM collective subroutine and an opaque derived type IMAGE_TEAM in ISO_FORTRAN_ENV. The TEAM argument to the collective subroutines was changed to be of type IMAGE_TEAM. The major changes with respect to 05-272r2 are: 1) The following replacements of intrinsics by statements has been made: FLUSH_MEMORY by SYNC_MEMORY, SYNC_ALL by SYNC_ALL, SYNC_TEAM by SYNC_IMAGES; NOTIFY_TEAM by NOTIFY; READY_TEAM by QUERY, and WAIT_TEAM by QUERY. The main reason is that they affect the flow of control between images. (Recall that the FLUSH statement in 2003 started out as a subroutine, but was changed to a statement during integration.) The descriptions are more self-contained and the change allows dynamic restrictions to be imposed to avoid race conditions. A QUERY statement is called 'blocking' if it has no READY= specifier or if it has a READY= specifier whose value is .true. . 2) To allow for efficient use of barrier hardware on systems that have it, we have made SYNC_ALL separate from SYNC_IMAGES in the sense that all images are delayed until they have all executed SYNC_ALL an equal number of times. 3) NOTIFY and QUERY are allowed to have a scalar image set. 4) A blocking QUERY requires the number of NOTIFY executions for its image M on each image T of its image set to be at least one greater than the number of blocking QUERY executions for T on M. 5) There is no equivalent to SYNC_TEAM(TEAM,WAIT), but it can be written as a NOTIFY/QUERY pair. 6) The following is not a technical change but it is a major clarification of the description. Any of the statements SYNC_ALL, SYNC_IMAGES, SYNC_MEMORY, NOTIFY, QUERY, ALLOCATE, DEALLOCATE, CRITICAL, END CRITICAL, OPEN, CLOSE, BACKSPACE, ENDFILE, REWIND, END or RETURN may affect the execution ordering between images and are called 'image control' statements. On an image, the sequence of statements executed between the execution of two image control statements (or before the first image control statement or after the last image control statement) is a 'segment'. The processor may optimize the execution of a segment as if it were the only image in execution. 7) The execution ordering between images may also be affected by 'user-defined ordering constructs', usually spin loops. 8) The following is a major clarification, associated with 6). If a non-volatile co-array variable is defined on an image in a segment, it must not be referenced or defined in a segment on another image unless the set of image control statements and user-defined ordering constructs on all the images ensure that one segment is always executed before the other. 9) The collective functions have been changed to subroutines because they involve image synchronization and this is very awkward to define precisely within a statement that involves collective function calls. A call to a collective subroutine is another image control statement. 10) If the ARRAY or MASK argument of one of the collective subroutines has zero size, the result has zero size. Specifications of the result values in this case have been removed. 11) For consistency with the OPEN statement, we now require all images of the connect team to execute the same CLOSE, BACKSPACE, ENDFILE, or REWIND statement. The major changes that were made between J3/05-208 and 05-272r2 were: 1) Instead of overloading the array reduction functions with collective versions, new functions called CO_ALL, etc. were added. This is because the overloaded functions would no longer be pure. The new functions have an optional argument TEAM, whose absence indicates that the team consists of all the images. We introduced a new category of intrinsic function: collective function. 2) For consistency with the ALLOCATE statement, we now require all images of the connect team to execute the same OPEN statement. 3) We added CO_MAXLOC and CO_MINLOC, which we believe were omitted in error in Delft. End Preface =========== Note 1. < and > are used to delimit italic font. << and >> are used to delimit bold font. Note 2. All page and line numbers refer to j3/04-007. EDITS ....................................................................... [xiii] Introduction, the list of new Fortran 2008 features should include Parallel programming support: SPMD parallel programming, co-arrays for data exchange between images, image control statements, and collective procedures. ....................................................................... In section 1.6 Compatibility, retain paragraph 3:5-9 regarding new intrinsic procedure names. There are new intrinsics introduced in this proposal. ....................................................................... [1:18+] Add new section after 1.2 1.2a Images A Fortran program executes as if it were replicated a number of times (which may be one), the number of replications remaining fixed during execution of the program and available from the intrinsic function NUM_IMAGES(). Each copy is called an <> and has its own set of data objects. The images execute asynchronously. The programmer controls the progress of execution in each image through explicit use of Fortran control constructs. The image control statements (8.5) affect the relative progress of execution between images. Co-arrays provide a mechanism for accessing data on one image from another image. Each image is indexed by an <>. These form a sequence of integers running from one to NUM_IMAGES(). The programmer may retrieve the index of the invoking image from the intrinsic function THIS_IMAGE(). NOTE 1.0. A particular implementation may permit the number of images to be chosen at compile time, at link time, or at execute time. The number of images may be the same as the number of physical processors, or it may be more, or it may be less. [end NOTE] NOTE 1.0a. It is usual for the program to be written for any number of images and the actual number for a specific run to be specified when the program is executed. This allows the program to be developed with fewer images than will be used in the eventual production runs. If the system is a uniprocessor, the processor knows that the number of images is 1 and can take advantage of this. Thus the early testing of the program can be performed on a uniprocessor. For compilation for a multiprocessor, the vendor could supply a switch to specify that the number of images is 1, which would permit the optimizer to eliminate overhead associated with parallel execution. [end NOTE] A <> of images is a set of images that are collaborating in a computation. The members of the team are specified by a rank-one integer array that holds the image indices, with no duplications. A team is identified by a scalar of derived type IMAGE_TEAM (13.8.2). Teams can be formed with the intrinsic collective subroutine FORM_TEAM (13.7.25i). ....................................................................... [11:1-] Add to R213 <> ....................................................................... [11: 23+, 27+, 31+] In the list of action statements, R214, add: <> <> <> <> <> ....................................................................... [15:19+] After the second paragraph of 2.3.4 Execution sequence, add Coordination of the execution sequences of different images is discussed in 8.5. ....................................................................... [15:20] Replace the first sentence of the last paragraph of 2.3.4 by Normal termination of execution of the program occurs on all images if a STOP statement or is executed on an image. ....................................................................... [15:25+] Add at the end of 2.3.4 NOTE 2.2a If synchronization is required before all images complete execution, a SYNC_ALL statement (8.5.1) should be employed ahead of the STOP statement or . [end NOTE] ....................................................................... [18:20+] Add new subsection to 2.4.5 2.4.5a Co-array A <> is a data entity declared with a in square brackets. Within an image, it is normally accessed without use of co-subscripts in square brackets and its properties are exactly those that it would have had without the except that its value may be referenced or defined by other images. For example, the rank, bounds, extents, size, and shape are given by the data in parentheses in its declaration or allocation. The part of a co-array stored in the memory of a specified image may be addressed by using <> in square brackets. Each co-subscript shall be a scalar integer expression. An object whose designator includes square brackets is a <>. For each co-array, co-subscripts are mapped to image indices in the same way as Fortran array subscripts are mapped to the position of the array element in array element order. NOTE 2.4a The use of square brackets for array constructors does not conflict with their use for co-arrays because the left square bracket of the co-array syntax is immediately preceded by a designator for a variable. [end NOTE] On each image, the image index is available from the intrinsic function THIS_IMAGE with no arguments, the set of subscript values for a co-array A that correspond to the image are available as THIS_IMAGE(A), and the image index that corresponds to a set of subscript values SUB for a co-array A is available as IMAGE_INDEX(A,SUB). NOTE 2.4b For example, on image 5, for the co-array declared as REAL :: A(10,20)[10,0:9,0:*] THIS_IMAGE() has the value 5 and THIS_IMAGE(A) has the value (/ 5,0,0 /). For the same example on image 213, THIS_IMAGE(A) has the value (/ 3,1,2 /). [end NOTE] NOTE 2.4c On a shared-memory machine, a co-array may be implemented as if it were an array of higher rank. On a distributed-memory machine with separate physical memory for each image, a co-array may be stored at the same virtual address in each physical processor. On any machine, a co-array may be implemented in such a way that each image can calculate the virtual address of an element on another image relative to the array start address on that other image. An implementation might arrange for each co-array to be stored at the same virtual address in each image, but this is not required. [end NOTE] The <> of a co-array is the number of co-subscripts. The <> and <> are given within square brackets in the co-array's declaration or allocation. The <> of a co-array is always equal to the number of images. ....................................................................... [19:5] In the last line of the first paragraph of 2.5.1, after 'array element selectors,' add 'image selectors,'. ....................................................................... [28:4+] Insert 'END CRITICAL' in the table in 3.3.1 ....................................................................... [50:6+] After the second line of R441, add <> DIMENSION [()] ....................................................................... [50:9+] After the first line of R442, add [] ....................................................................... [50:25+] After C440, add C440a (R440) Each shall be a and the component shall be specified with the ALLOCATABLE attribute. ....................................................................... [51:20+] Before NOTE 4.28, add new paragraph: A data component is a co-array if its contains a or its contains a DIMENSION with a . If the contains a , it specifies the co-rank; otherwise, the in the DIMENSION specifies the co-rank. ....................................................................... [71:21+] In R503, after the DIMENSION line, add <> DIMENSION [()] ....................................................................... [72:9] Separate R504 into two lines after '[()]' and insert line: [] ....................................................................... [72:25+] After C510, add C510a (R501) A co-array that has the ALLOCATABLE attribute shall be specified with a that is a . ....................................................................... [72:33] In the second line of C514, after 'an allocatable entity,' add 'a co-array'. ....................................................................... [73:1+] After C517, add C517a (R501) The SAVE attribute shall be specified for a co-array unless it is allocatable, a dummy argument, or declared in the main program. NOTE 5.2a An allocatable co-array is not required to have the SAVE attribute because a recursive procedure may need separate allocatable co-arrays at each level of recursion. [end NOTE] C517b (R501). If an object of derived type has a co-array component at any level of component selection, each ancestor of the co-array component shall be a non-allocatable non-pointer scalar. C517c (R501). The POINTER attribute shall not be specified for a co- array. NOTE 5.2b. A co-array may be of a derived type with pointer or allocatable components. The targets of such components always reside on the local image. If a large array is needed on a subset of images, it is wasteful of memory to specify it directly as a co-array. Instead, it should be specified as an allocatable component of a co-array and allocated only on the images on which it is needed. [end NOTE] ....................................................................... [74:19+] After the definition of an automatic data object, add a new constraint: C538a (R501) An automatic data object shall not be a co-array. ....................................................................... [74:20-] After NOTE 5.3, add a new note: NOTE 5.3a Automatic co-arrays are not permitted; for example, the co-array WORK in the following code fragment is not permitted to be declared thus SUBROUTINE SOLVE3(N,A,B) INTEGER :: N REAL :: A(N)[*], B(N) REAL :: WORK(N)[*] ! Not permitted [end NOTE] ....................................................................... [78:3-9] Replace the first paragraph of 5.1.2.5 by The <> specifies that an entity is an array, a co-array, or both. The appearance of an or a in an specifies the DIMENSION attribute for the entity. Such an entity is an array if an appears and is a co-array if a appears. The DIMENSION attribute alternatively may be specified in the specification statements DIMENSION, ALLOCATABLE, POINTER, TARGET, or COMMON. For an array, the rank or rank and shape is specified by the , if there is one, in the , or by the in the DIMENSION otherwise. For a co-array, the co-rank or co-rank and co-shape are specified by the , if there is one, in the , or by the in the DIMENSION otherwise. ....................................................................... ***Note to Editor: This edit replaces the one for this line from 05-234r2 [78:14] Replace C541 by R510a <> <> C541 (R510 and R510a) The sum of the rank and co-rank shall not exceed fifteen. ....................................................................... [85:6] Replace the last word, "program." of the first sentence of 5.1.2.16 Volatile, with "program, or by execution of statements on another image in the case of a co-array variable." ....................................................................... [87:34+] After R529, add C552a (R525). A shall not contain any image selectors (6.2a). ....................................................................... [90:2-5] Replace the contents of 5.2.6 by R535 <> DIMENSION [::] R535a <> () <> [()] This statement specifies the DIMENSION attribute (5.1.2.5) and the array and co-array properties for each object named. ....................................................................... [91:19+] After C570, add C570a (R544). A common block that contains a co-array shall be specified to have the SAVE attribute in every scoping unit in which it appears. ....................................................................... [92:4-5] Replace R546 by R546 <> TARGET [::] R546a <> [()] [] ....................................................................... [96:12+] After C577, add C577a (R556). An shall not contain an image selector (6.2a). ....................................................................... [98:22+] After C588, add C588a (R558). A shall not contain an image selector (6.2a). C588b (R558). Blank common shall not contain a co-array. ....................................................................... [105:2] Append to the end of R613 '[]' ....................................................................... [105:9+] After C613, add C613a (R613). If appears and is an array, ( shall appear. ....................................................................... [110:2-] Add new section at the end of 6.2. 6.2a Image selectors An <> specifies the image whose data are referenced or defined. R622a <> The value of a subscript in an image selector shall be within the bounds for its co-dimension. An image selector shall select an image with index in the range 1, ..., NUM_IMAGES(). NOTE 6.15a For example, if there are 16 images and the co-array A is declared REAL :: A(10)[5,*] A(:)[1,4] is valid since it has co-subscript order value 16, but A(:)[2,4] is invalid. [end NOTE] ....................................................................... [110:20] In R628, after '[()]' add the new line '[ ]'. ....................................................................... [111:5+] After R632, add 'R632a <> [,] [:]* '. ....................................................................... [111:17] Add at the end of C628 'An shall appear if and only if the is a co-array.' ....................................................................... [111:19] Add at the end of C629 'The number of s in an shall be one less than the co-rank of the .' ....................................................................... [111:25+] After C633, add C633a (R629) An shall not be a co-array subobject. NOTE 6.15b If a co-array is of a derived type that has an allocatable component, the component shall be allocated only on the local image: TYPE(SOMETHING), ALLOCATABLE :: T[:] ... ALLOCATE(T[*]) ! Allowed - implies synchronization ALLOCATE(T%PTR(N)) ! Allowed - local allocation ALLOCATE(T[Q]%PTR(N)) ! Not allowed [end NOTE] ....................................................................... [113:21+] At the end of 6.3.1.1, add The value of each and each in an shall not differ between images. If a appears in an ALLOCATE statement and any specifies a co-array, the values of each length type parameter shall not differ between images. If SOURCE= appears in an ALLOCATE statement and the specifies a co-array, the values of each length type parameter of the shall not differ between images. The dynamic type of an that is a co-array shall not differ between images. There is implicit synchronization of all images in association with each ALLOCATE statement that involves one or more co-arrays. Execution on an image, M, of the segment following the statement is delayed until all other images have executed the same statement as many times as has image M. The segments that executed before the statement on image M precede the segments that execute after the statement on another image T. If the specifies a co-array that is declared in a recursive procedure and does not have the SAVE attribute, the number of active instances of the procedure shall be equal on all images. NOTE 6.19a When an image executes an ALLOCATE statement, communication is not necessarily involved apart from any required for synchronization. The image allocates the local part and records how the corresponding parts on other images are to be addressed. The processor, except perhaps in debug mode, is not required to enforce the rule that the bounds are the same on all images, nor is it responsible for detecting or resolving deadlock problems. [end NOTE] ....................................................................... [116:19-] At the end of 6.3.3.1, add There is implicit synchronization of all images in association with each DEALLOCATE statement that involves one or more co-arrays. All images delay making the deallocations until they are all about to execute the same DEALLOCATE statement. The segments that executed before the statement on image M precede the segments that execute after the statement on another image T. If an object that has a co-array component at any level of component selection is declared in a procedure and does not have the SAVE attribute, there is an implicit deallocation (and associated synchronization) before the procedure in which it is declared is exited by execution of a RETURN or END statement. NOTE 6.25a An allocatable co-array without the SAVE attribute may be required if separate arrays are needed at each level of recursion in a recursive procedure. For allocation of such a co-array, each image must descend to the same level of recursion or deadlock may occur. [end NOTE] ....................................................................... [137:3] Add to the paragraph before note 7.31: "If an expression involves the invocation of more than one function, each image shall invoke the same functions in the same order." ....................................................................... [139:1+] Add to the end of the first paragraph of 7.4.1.2: If is an allocatable co-array, it shall be allocated and the shapes of and shall conform. If is a co-array subobject of a type that has an allocatable component at any level of component selection, the shapes of the corresponding components of and shall conform. C715a (R734) The in an intrinsic assignment statement shall not be a co-array subobject of a derived type that has a pointer subcomponent (6.1.2). NOTE 7.34a The restriction in C715a is needed to avoid a disallowed pointer assignment for a component, such as Z[I] = Z ! Not allowed if Z has a pointer component Z = Z[I] ! Not allowed if Z has a pointer component [end NOTE] ....................................................................... [143:31+] After C723, add C723a (R739) A shall not be a co-array subobject. NOTE 7.42a A pointer and its target always reside on the same image. A co-array may be of a derived type with pointer or allocatable components. For example, if P is a pointer component, Z[I]%P is a reference to the target of component P of Z on image I. This target must reside on image I and must have been established by an allocate statement executed on image I or a pointer assignment executed on image I. [end NOTE] ....................................................................... [143:37+] After C725, add C725a (R741) The in a shall not be a co-array subobject. NOTE 7.42b A co-array is permitted to be of a type that has a procedure pointer component or a type bound procedure. Such a procedure shall not be invoked remotely; for example, the statement CALL A[I]%PROC(X) ! Remote reference not allowed is not permitted. [end NOTE] ....................................................................... [155:2] At the start of 8, after 'The execution sequence' add 'of a single image'. [155:3] In the second line of 8, after 'execution sequence.' add 'This is the subject of 8.1, 8.2, 8.3, and 8.4; image synchronization for exchange of data between images is the subject of 8.5.' ....................................................................... [170:25-26] Change the first two sentences of 8.4 to Execution of a STOP statement on an image causes normal termination (2.3.4) of execution of the program on all images. At the time of termination, the stop code from the image executing the STOP statement, if any, is available in a processor dependent manner. At most one image shall make a stop code available. ....................................................................... ***Note to Editor: This edit should be applied following the edit for [170:29+] from 05-231r4 [170:29+] After 8.4, add new section: 8.5 Image execution control The execution sequence on each image is as specified in 2.3.4. Each of the following statements affect the execution ordering between images and is an <> statement: (1) SYNC_ALL statement, (2) SYNC_TEAM statement, (3) SYNC_IMAGES statement, (4) SYNC_MEMORY statement, (5) NOTIFY statement, (6) QUERY statement, (7) ALLOCATE or DEALLOCATE statement involving a co-array, (8) CRITICAL or END CRITICAL statement, (9) OPEN statement with a TEAM= specifier, (10) CLOSE statement for a file that is open with a TEAM= specifier, (11) END or RETURN statement that involves an implicit deallocation of a co-array, and (12) CALL statement for a collective subroutine (13.1). On each image, the sequence of statements executed between the execution of two image control statements (or before the first image control statement or after the last image control statement) is known as a <>. The segment preceding an image control statement includes the evaluation of all expressions within the statement. By execution of image control statements and user-defined ordering constructs (8.5.5), the program may ensure that the execution of segment S_I on image I is certain to precede or succeed the execution of segment S_J on image J. If the program does not ensure this, segments S_I and S_J are <>; depending on the relative execution speeds of the images, some or all of the execution of S_I may take place at the same time as some or all of the execution of S_J. NOTE 8.22 The set of all segments on all images is partially ordered. The segment S_I is certain to precede segment S_J if and only if there is a sequence of segments starting with S_I and ending with S_J such that each segment of the sequence is certain to precede the next. [end NOTE] A scalar variable that is of type default integer, default logical, default real, or default bits, and has the VOLATILE attribute (5.1.2.16) may be referenced during the execution of a segment that is unordered relative to the execution of a segment in which the variable is defined. Otherwise, (1) if a co-array variable is defined on an image in a segment, it shall not be referenced or defined in a segment on another image unless the segments are ordered. (2) if a procedure that is in execution on image I in segments S_1, S_2, ..., S_K defines a non-co-array dummy argument, the corresponding actual argument shall not be referenced or defined on another image J in a segment S_J unless S_J precedes S_1 or succeeds S_K. NOTE 8.22a Apart from the effects of volatile variables, the processor may optimize the execution of a segment as if it were the only image in execution. [end NOTE] NOTE 8.22b The model upon which the interpretation of the standard is based is there is a permanent memory location for each co-array variable and that all images may access it. In practice, an image may make a copy of a non-volatile co-array variable (in cache or a register, for example) and, as an optimization, defer copying a changed value back to the permanent location while it still needs it. Since the variable is not volatile, it is safe to defer this transfer to the end of the current segment and thereafter to reload from permanent memory any co-array variable that was not defined within the segment. It would not be safe to defer these actions beyond the end of the current segment since another image might reference the variable then. [end NOTE] If an image I writes a record to a file that is opened for direct access with a TEAM= specifier and image J reads that record during the execution of segment S_J, then, (1) after image I writes the record, it shall execute a FLUSH statement (9.8) for the file during the execution of a segment S_I that precedes S_J, and (2) in segment S_J, before image J reads the record, it shall execute a FLUSH statement (9.8) for the file. 8.5.1 SYNC_ALL statement R851 <> SYNC_ALL [([])] R852 <> STAT = <> ERRMSG = C835 (R851) No specifier shall appear more than once in a given . C836 (R852) A scalar variable in a STAT= specifier, or ERRMSG= specifier shall not contain an (6.2a). The STAT= and ERRMSG= specifiers are described in 8.5.6. Execution of a performs a synchronization of all images. Execution on an image, M, of the segment following the is delayed until all other images have executed a as many times as has image M. The segments that executed before the on image M precede the segments that execute after the corresponding on another image T. NOTE 8.23 The processor may have special hardware or employ an optimized algorithm to make the execute efficiently. Here is a simple example of its use. Image 1 reads data and broadcasts it to other images: REAL :: P[*] ... SYNC_ALL IF (THIS_IMAGE()==1) THEN READ(6,*) P DO I = 2, NUM_IMAGES() P[I] = P END DO END IF SYNC_ALL [end NOTE] NOTE 8.23a If synchronization is required before all images commence execution, a SYNC_ALL statement should be the first executable statement of the main program. This is necessary if the code relies on the initialization of a co-array variable on another image. [end NOTE] 8.5.2 SYNC_TEAM statement R853 <> SYNC_TEAM ( [,]) R854 <> C837 (R854) shall be scalar of the derived type IMAGE_TEAM defined in the intrinsic module ISO_FORTRAN_ENV and shall not contain an . Execution of a performs a <>. Images in the team do not commence executing subsequent statements until all images in the team have executed a statement for the team an equal number of times. If images M and T are members of the team, the segments that executed before the statement on image M precede the segments that execute after the statement on image T. NOTE 8.24 Here is an example where the images are divided into two teams, one for an ocean calculation and one for an atmosphere calculation. USE, INTRINSIC :: ISO_FORTRAN_ENV TYPE(IMAGE_TEAM) :: TEAM INTEGER :: N2, STEP, NSTEPS LOGICAL :: OCEAN N2 = NUM_IMAGES()/2 OCEAN = (THIS_IMAGE<=N2) IF(OCEAN) THEN CALL FORM_TEAM(TEAM, (/ (I, I=1,N2) /) ) ELSE CALL FORM_TEAM(TEAM, (/ (N2+I, I=1,N2) /) ) END IF : ! Initial calculation SYNC_ALL DO STEP = 1, NSTEPS IF (OCEAN) THEN DO : ! Ocean calculation SYNC_TEAM(TEAM) IF ( ... ) EXIT ! Ready to swap data END DO ELSE DO : ! Atmosphere calculation SYNC_TEAM(TEAM) IF ( ... ) EXIT ! Ready to swap data END DO END IF SYNC_ALL : ! Swap data END DO In the inner loops, each set of images works entirely with its own data and each image synchronizes with the rest of its team. The number of synchronizations for the ocean team may differ from the number for the atmosphere team. The SYNC_ALL statement is needed to ensure that both teams have done their calculations before data is swapped. [end NOTE] 8.5.3 SYNC_IMAGES statement R855 <> SYNC_IMAGES ( [,]) R856 <> <> * C838 (R856) An that is an shall be scalar or of rank one and shall not contain an (6.2a). If is an array expression, the value of each element shall be in the range 1, 2, ..., NUM_IMAGES(), there shall be no repeated values, and at least one element shall have a value other than THIS_IMAGE(). If is a scalar expression, its value shall be in the range 1, 2, ..., NUM_IMAGES(). An that is an asterisk specifies all the images. Execution of a performs a synchronization of the image with each of the other images in the . Execution on an image, M, of the segment following the is delayed until all other images T in the have executed a with M in its as many times as image M has executed a with T in the . The segments that executed before the on image M precede the segments that execute after the corresponding on image T. NOTE 8.25 In NOTE 8.24, the images of each team all wait for each other. There may, however, be situations where this is not efficient. In the following example, each image synchronizes with its neighbor. INTEGER :: ME, NE, STEP, NSTEPS NE = NUM_IMAGES() ME = THIS_IMAGE() : ! Initial calculation SYNC_ALL DO STEP = 1, NSTEPS IF (ME > 1) SYNC_IMAGES(ME-1) : ! Perform calculation IF (ME < NE) SYNC_IMAGES(ME+1) END DO SYNC_ALL The calculation starts on image 1 since all the others will be waiting on SYNC_IMAGES(ME-1). When this is done, image 2 can start and image 1 can perform its second calculation. This continues until they are all executing different steps at the same time. Eventually, image 1 will finish and then the others will finish one by one. [end NOTE] 8.5.4 NOTIFY and QUERY statements R857 <> NOTIFY ( [,]) R858 <> QUERY ( [,]) R859 <> READY = <> STAT = <> ERRMSG = C839 (R858) No specifier shall appear more than once in a given . C840 (R859) A scalar variable that is a STAT= specifier, or ERRMSG= specifier shall not contain an (6.2a). The STAT= and ERRMSG= specifiers are described in 8.5.6. Execution on image M of a NOTIFY statement with image T in its increments by 1 a record of the number of times, n(M,T), image M executed such a NOTIFY statement. The execution of a QUERY statement on an image is called <> if the QUERY statement has no READY= specifier or if it has a READY= specifier whose value after the statement has been executed is .true. . Execution on image M of a blocking QUERY statement with image T included in its image set increases by 1 a record of the number of times, b(M,T), image M executed such a QUERY statement. This increase is made after its value has been compared with n(T,M). If a READY= specifier appears, execution on image M of a QUERY statement causes the to become defined. The value is true if, for each image T in the image set, n(T,M) > b(M,T). Otherwise, the value is false. If a READY= specifier does not appear, increasing b(M,T) and completing execution of the statement is delayed until, for each T in the image set, n(T,M) > b(M,T). For a blocking QUERY statement on image M with image T in its image set, the NOTIFY statement execution on image T for which n(T,M) = b(M,T) + 1 is said to correspond to it. Segments on image T executed before the execution of this NOTIFY statement precede the executions of the segments on image M that follow this QUERY statement. NOTE 8.26 The NOTIFY and QUERY statements may be used to order statement executions between a producer and consumer image. INTEGER,PARAMETER :: PRODUCER = 1, CONSUMER = 2 INTEGER :: VALUE[*] LOGICAL :: READY SELECT CASE (THIS_IMAGE()) CASE(PRODUCER) VALUE[CONSUMER] = 3 NOTIFY(CONSUMER) CASE(CONSUMER) WaitLoop: DO QUERY(PRODUCER,READY=READY) IF(READY) EXIT WaitLoop ! Statements not referencing, or causing to become defined or ! undefined VALUE[CONSUMER] END DO WaitLoop ! references to VALUE CASE DEFAULT ! Statements not referencing, or causing to become defined or ! undefined VALUE[CONSUMER] END SELECT Unlike SYNC_IMAGES statements, the number of notifications and corresponding queries may be unequal. A program can complete with an excess number of notifies. [end NOTE] NOTE 8.27 NOTIFY/QUERY pairs can be used in place of SYNC_ALL and SYNC_IMAGES to achieve better load balancing and allow one image to proceed with calculations while another image is catching up. Here is an example: : IF (THIS_IMAGE()==1) THEN DO I=1,100 : ! Primary processing of column I NOTIFY(2) ! Done with column i END DO SYNC_IMAGES(2) ELSE IF (THIS_IMAGE()==2) THEN DO I=1,100 QUERY(1) ! Wait till 1 is done with column I : ! Secondary processing of column I END DO SYNC_IMAGES(1) END IF [end NOTE] NOTE 8.28 The incorrect sequencing of image control statements can halt execution indefinitely. For example, one image may be executing a while another is executing an allocate statement for a co-array; or one image may be executing a blocking QUERY statement for which an image in its image set never executes the appropriate NOTIFY statement. [end NOTE] 8.5.5 CRITICAL construct A critical construct may be used to limit execution of a block to one image at a time. R860 <> R861 <> [:] CRITICAL R862 <> END CRITICAL [] C841 (R810) If the of a specifies a , the corresponding shall specify the same . If the of a does not specify a , the corresponding shall not specify a . The effect of the construct is to limit the execution of to a single image at any one time. The processor shall ensure that once an image has commenced executing , no other image shall commence executing it until the image has completed executing it. If image T is the next to execute the construct after image M, the segments that executed before the construct on image M precede the segments that execute after the construct on image T. NOTE 8.29 If more than one image executes the block of a critical construct, its execution by one image always either precedes or succeeds its execution by another image. Typically no other statement ordering is needed. Consider the following example: CRITICAL global_counter[1] = global_counter[1] + 1 END CRITICAL The definition of global_counter[1] by a particular image will always precede the reference to the same variable by the next image to execute the block. [end NOTE] NOTE 8.30 The following example permits a large number of jobs to be shared among the images: INTEGER :: NUM_JOBS[*], JOB IF (THIS_IMAGE() == 1) READ(*,*) NUM_JOBS SYNC_ALL DO CRITICAL JOB = NUM_JOBS[1] NUM_JOBS[1] = JOB - 1 END CRITICAL IF (JOB > 0) THEN ! Work on JOB ELSE EXIT END IF END DO SYNC_ALL [end NOTE] 8.5.6 User-defined ordering constructs User-defined constructs may be used to order the execution of segments on different images. If one image, Q, executes a construct, Cq, that does not complete until a corresponding construct, Cp, is executed on image P, then executions of segments before Cp on image P precede the executions of segments on image Q after Cq. R863 <> SYNC_MEMORY [([])] The provides a separator on an image between two adjacent segments. NOTE 8.31 The primary purpose of the is to provide segment separators for user-defined ordering constructs. A common and important example is the spin-wait loop. LOGICAL,VOLATILE :: LOCKED[*] = .TRUE. INTEGER :: IAM IAM = THIS_IMAGE() IF (IAM == P) THEN ! Preceding segment SYNC_MEMORY LOCKED[Q] = .FALSE. ! construct Cp SYNC_MEMORY ELSE IF (IAM == Q) THEN DO WHILE (LOCKED); END DO ! construct Cq SYNC_MEMORY ! Following segment END IF The spin-wait loop can be used to implement specialized ordering schemes. It can be more specific than a NOTIFY/QUERY pair because independent spin variables can be used for each loop. Because the spin loop references a local variable, this method for execution ordering may be very efficient and appropriate for performance sensitive sections of a program. [end NOTE] NOTE 8.32 A user-defined ordering construct may also be formed using a synchronization subroutine defined by the user or available in a separate library. For example: INTEGER :: IAM REAL :: X[*] IAM = THIS_IMAGE() IF (IAM == 1) X = 1.0 SYNC_MEMORY CALL EXTERNAL_SYNC() SYNC_MEMORY IF (IAM == 2) WRITE(*,*) X[1] where the subroutine EXTERNAL_SYNC has an effect similar to SYNC_ALL. In this example, the constructs Cp and Cq are both the CALL statement. [end NOTE] 8.5.7 STAT= and ERRMSG= specifiers in image execution control statements. If the STAT= specifier appears, successful execution of the SYNC_ALL, SYNC_TEAM, SYNC_IMAGES, SYNC_MEMORY, NOTIFY, or QUERY statement causes the to become defined with the value zero. If an error occurs during execution of one of these statement, the becomes defined with a processor-dependent positive integer value. If an error condition occurs during execution of a SYNC_ALL, SYNC_TEAM, SYNC_IMAGES, SYNC_MEMORY, NOTIFY, or QUERY statement that does not contain the STAT= specifier, the effect is processor dependent. If the ERRMSG= specifier appears and an error condition occurs during execution of the SYNC_ALL, SYNC_TEAM, SYNC_IMAGES, SYNC_MEMORY, NOTIFY, or QUERY statement, the processor shall assign an explanatory message to . If no such condition occurs, the processor shall not change the value of . NOTE 8.33 Which errors, if any, are diagnosed is processor dependent. The processor might check that a valid set of images has been provided, with no out-of-range or repeated values. It might test for network time-outs; while the overall program would probably not be able to recover from the failure of an image, it could perhaps provide information on what failed and be able to save some of the program state to a file. [end NOTE] ....................................................................... [171:21+] At the end of 9, add new paragraph All images reference a single file system. ....................................................................... [173:29] At the end of 9.2.2.1, after 'statements.', add 'Each record shall be read or written by a single image. The processor shall ensure that once an image commences transferring the data of a record to the file, no other image transfers data to the file until the whole record has been transferred.'. ....................................................................... [178:36] At the end of the second paragraph of 9.4, after the second 'program', add 'and on all images'. ....................................................................... [179:33] In the second line of 9.4.3, after 'external file', add ', and the connection is for a team of images'. ....................................................................... [181:39+] At the end of R905, add <> TEAM = ....................................................................... [181:41+] After R907, add R907a <> ....................................................................... [185:4+] Add new subsection at the end of 9.4.5: 9.4.5.16 TEAM= specifier in the OPEN statement The specifies the <> for the unit, which is the set of images that are associated with the unit. If there is no TEAM= specifier, the connect team consists of only the executing image. All images in the connect team, and no others, shall invoke the same OPEN statement with identical values for the s. The unit shall not be open on other images. There is an implicit team synchronization. The OPEN statement connects the file on the invoking images only, and the unit becomes unavailable on all other images. If the OPEN statement does not have a FILE= specifier and the unit is not connected to a file, the processor shall connect the same file to the unit on all images in the connect team. If the connect team contains more than one image, the OPEN statement shall (1) have an ACCESS= specifier that evaluates to DIRECT, or, (2) have an ACTION= specifier that evaluates to WRITE, and shall not specify an ACCESS= specifier or shall have an ACCESS= specifier that evaluates to SEQUENTIAL. NOTE 9.21a Writing to a sequential file from more than one image is permitted for situations in which the ordering of records is unimportant. An example is for diagnostic output that is labelled with the image index. [end NOTE] An OPEN statement on a unit connected to a file by a previously executed OPEN statement shall have the same connect team as currently in effect. NOTE 9.21b The rule in 9.4.3 that a unit shall not be connected to more than one file at the same time and a file shall not be connected to more than one unit at the same time applies even if the connect teams for the units have no images in common. [end NOTE] Pre-connected units have a connect team consisting of all the images. Input on pre-connected units shall be performed only on the image with index one. NOTE 9.21c The input unit identified by * is therefore only available on the image with index one. [end NOTE] ....................................................................... [185:19+] Add new paragraph at the end of 9.4.6: At the start of execution of a CLOSE statement, there is an implied FLUSH operation for the unit. There is also an implicit team synchronization. ....................................................................... [208:4+] Add new paragraph at the end of 9.7.1: A BACKSPACE statement shall not specify a that is connected with a connect team with more than one image. ....................................................................... [208:19+] Add new paragraph at the end of 9.7.2: An ENDFILE statement shall not specify a that is connected with a connect team with more than one image. ....................................................................... [208:23+] Add new paragraph at the end of 9.7.3: A REWIND statement shall not specify a that is connected with a connect team with more than one image. ....................................................................... [209:18-19] In the second paragraph of 9.8, change 'to other processes, or causes data placed in an external file' to 'to other images of the same team or to other processes, or causes data placed in an external file by other images of the same team or'. ....................................................................... [256:26] In the first line of 12.2.1.1, after 'its shape,' add 'its co-shape,'. ....................................................................... [257:25+] In the first list of 12.3.1.1, add (a1) With an argument that is a co-array subobject (2.4.5a), ....................................................................... [257:33+] In the second list of 12.3.1.1, add (b1) is a co-array, ....................................................................... [269:13+] In 12.4.1.2, Actual arguments associated with dummy data objects, add a new paragraph and note: If the actual argument is a co-array subobject, the dummy argument shall not be a co-array and shall have the INTENT(IN) or the VALUE attribute. A co-array subobject shall not be an actual argument if it has a pointer subcomponent. NOTE 12.21a If the actual argument is a co-array subobject and the corresponding dummy argument is not a co-array, it is expected that implementations will make a local copy of the actual argument, including copies of any allocated allocatable subcomponents, before execution of the procedure starts. [end NOTE] ....................................................................... [269:17+] After the sixth paragraph of 12.4.1.2, add new paragraphs: If a dummy argument is an allocatable co-array, the actual argument shall be an allocatable co-array of the same rank and co-rank. If the dummy argument is allocatable and is not a co-array, the actual argument shall not be an allocatable co-array. If a dummy argument is a co-array that is not allocatable, the co-rank and co-bounds are specified by the local declaration and are independent of those of the actual argument. The actual argument shall be the name of a co-array or a subobject of a co-array without any co-subscripts, vector-valued subscripts, allocatable component selection, or pointer component selection. NOTE 12.21b Only the local part of the co-array actual argument is specified and this is associated with the local part of the dummy co-array. It is assumed that the rest of the actual co-array consists of the corresponding parts on the other images. Here is an example INTERFACE SUBROUTINE SUB(X,Y) REAL :: X(:)[*], Y(:)[*] END SUBROUTINE SUB END INTERFACE : REAL, ALLOCATABLE :: A(:)[:], B(:,:)[:] : CALL SUB(A(:),B(1,:)) [end NOTE] Each image independently associates each non-allocatable co-array dummy argument with a co-array that is not a dummy argument (perhaps through several layers of procedure calls). Any reference to a co-array subobject of the dummy argument uses the local co-bounds and co-rank. NOTE 12.21c It is expected that a procedure with a co-array dummy argument will often be called simultaneously on all the images of a team with the same actual co-array and the same bounds and co-bounds. However, there are other possibilities; for example, a procedure might be called on a single image to perform a calculation that needs data from other images. [end NOTE] If an assumed-shape array or a subobject of an assumed-shape array appears as an actual argument corresponding to a dummy co-array, the dummy co-array shall be of assumed shape. A pointer component, an allocatable component that is not a co-array, or a subobject of such a component shall not be an actual argument that corresponds to a co-array dummy argument. If an array section appears as an actual argument corresponding to a dummy co-array that is not of assumed shape, the section shall be contiguous (5.1.2.4a). ***Note to editor: Section number 5.1.2.4a is specified in paper 05-273r3, for feature J3-043. NOTE 12.21d It is intended to ensure that copy-in or copy-out does not take place for a dummy co-array. [end NOTE] ....................................................................... [270:6] In the first full paragraph on page 270, replace "corresponding actual argument has the TARGET attribute" with "corresponding actual argument is not a co-array subobject and has the TARGET attribute". ....................................................................... [271:6] In constraint C1232, replace the beginning of the first sentence: "If an actual argument is an array section or an assumed-shape array," with If an actual argument is an array section or an assumed-shape array that is not a co-array subobject ....................................................................... [279:21+] After R1224, add C1233a (R1223) A function result shall not be a co-array. ....................................................................... ***Note to Editor: This text should be added before the Note edited by 05-237r3 [287:7+] After C1276, add C1276a A pure subprogram shall not contain a reference or definition of any part of a co-array. ....................................................................... [291:12] In the fifth line of 13.1, after 'described in 12.7' add "A <> is one that has a co-array or team argument and, on each image of a team, computes a result that has the shape of the co-array argument. Each element of the result is calculated from the values of the corresponding elements on all the images of the team. It shall be invoked on all images of the team with its co-array argument having the same shape. There is an implicit team synchronization (8.5.2)." ....................................................................... [291:14] In the seventh line of 13.1, change 'standard intrinsic' to 'inquiry, elemental, and transformational intrinsic'. ....................................................................... [297:25+] After 13.5.14, add new subsection: 13.5.14a Collective subroutines CO_ALL(MASK,RESULT[,TEAM]) True if all values are true CO_ANY(MASK,RESULT[,TEAM]) True if any value is true CO_COUNT(MASK,RESULT[,TEAM]) Numbers of true elements CO_MAXLOC(ARRAY,RESULT[,TEAM]) Image indices of maximum values CO_MAXVAL(ARRAY,RESULT[,TEAM]) Maximum values CO_MINLOC(ARRAY,RESULT[,TEAM]) Image indices of minimum values CO_MINVAL(ARRAY,RESULT[,TEAM]) Minimum values CO_PRODUCT(ARRAY,RESULT[,TEAM]) Products of elements CO_SUM(ARRAY,RESULT[,TEAM]) Sums of elements FORM_TEAM(TEAM,IMAGES) Forms a team of images NOTE 13.6a The subroutine FORM_TEAM forms a team of images. The other collective subroutines return an array of the same shape as the given co-array after having applied an operation across the images in a team. The simple case of wanting the operation to be applied to all the elements of the co-array can easily be programmed, for example REAL FUNCTION GLOBAL_SUM(ARRAY) REAL :: ARRAY(:,:)[*] REAL,SAVE :: TEMP[*] TEMP = SUM(ARRAY) ! Local sum CALL CO_SUM(TEMP,GLOBAL_SUM) END FUNCTION GLOBAL_SUM Versions with other arguments can be programmed similarly, see C.9a.2. [end NOTE] ....................................................................... [298:9+] Add to 13.5.18 IMAGE_INDEX(ARRAY, SUB) Index of an image ....................................................................... [298:11+] Add to 13.5.18 NUM_IMAGES() Number of images ....................................................................... [298:12+] Add to 13.5.18 THIS_IMAGE([ARRAY [,DIM]]) Index of the invoking image ....................................................................... [309:32+] After 13.7.25, add new subsections 13.7.25a CO_ALL(MASK,RESULT[,TEAM]) <> Determine whether all values are true across a team of images. <> Collective subroutine. <> MASK shall be a co-array of type logical. It is an INTENT(IN) argument. RESULT shall have the same type, kind type parameter, and shape as MASK. It is an INTENT(OUT) argument. An element has the value true if all the corresponding elements of MASK are true on all the images of TEAM, and the value false otherwise. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and MASK has the values (/.TRUE.,.FALSE.,.TRUE./) and (/.TRUE.,.TRUE.,.TRUE./) on the two images, the statement CALL CO_ALL(MASK,RESULT) gives RESULT the value [true, false, true]. 13.7.25b CO_ANY(MASK,RESULT[,TEAM]) <> Determine whether any value is true across a team of images. <> Collective subroutine. <> MASK shall be a co-array of type logical. It is an INTENT(IN) argument. RESULT shall have the same type, kind type parameter, and shape as MASK. It is an INTENT(OUT) argument. An element of RESULT has the value true if any of the corresponding elements of MASK are true on the images of TEAM, and the value false otherwise. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and MASK has the values (/.TRUE.,.FALSE.,. FALSE./) and (/.TRUE.,.TRUE.,.FALSE./) on the two images, the statement CALL CO_ANY(MASK,RESULT) gives RESULT the value [true, true, false]. 13.7.25c CO_COUNT(MASK,RESULT[,TEAM]) <> Count the numbers of true elements across a team of images. <> Collective subroutine. <> MASK shall be a co-array of type logical. It is an INTENT(IN) argument. RESULT shall have type integer and the shape of MASK. It is an INTENT(OUT) argument. An element of RESULT has a value equal to the number of corresponding elements of MASK on the images of TEAM that have the value true. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and MASK has the values (/.TRUE.,.FALSE.,. FALSE./) and (/.TRUE.,.TRUE.,.FALSE./) on the two images, the statement CALL CO_COUNT(MASK,RESULT) gives RESULT the value [2, 1, 0]. 13.7.25d CO_MAXLOC(ARRAY,RESULT[,TEAM]) <> Image indices of maximum values of the elements across a team of images. <> Collective subroutine. <> ARRAY shall be a co-array of type integer, real, or character. It is an INTENT(IN) argument. RESULT shall have type integer and the shape of ARRAY. It is an INTENT(OUT) argument. An element of RESULT has a value equal to the smallest image index of a maximum value of all the corresponding elements of ARRAY on the images of the team. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and ARRAY has the values (/1,5,6/) and (/4,1,6/) on the two images, the statement CALL CO_MAXLOC(ARRAY,RESULT) gives RESULT the value (/2,1,1/). 13.7.25e CO_MAXVAL(ARRAY,RESULT[,TEAM]) <> Maximum values of the elements across a team of images. <> Collective subroutine. <> ARRAY shall be a co-array of type integer, real, or character. It is an INTENT(IN) argument. RESULT shall have the same type, type parameters, and shape as ARRAY. It is an INTENT(OUT) argument. An element of RESULT has a value equal to the maximum value of all the corresponding elements of ARRAY on the images of the team. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and ARRAY has the values (/1,5,3/) and (/4,1,6/) on the two images, the statement CALL CO_MAXVAL(ARRAY,RESULT) gives RESULT the value (/4,5,6/). 13.7.25f CO_MINLOC(ARRAY,RESULT[,TEAM]) <> Image indices of minimum values of the elements across a team of images. <> Collective subroutine. <> ARRAY shall be a co-array of type integer, real, or character. It is an INTENT(IN) argument. RESULT shall have type integer and the shape of ARRAY. It is an INTENT(OUT) argument. An element of RESULT has a value equal to the smallest image index of a minimum value of all the corresponding elements of ARRAY on the images of the team. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and ARRAY has the values (/1,5,6/) and (/4,1,6/) on the two images, the statement CALL CO_MINLOC(ARRAY,RESULT) gives RESULT the value (/1,2,1/). 13.7.25g CO_MINVAL(ARRAY,RESULT[,TEAM]) <> Minimum values of the elements across a team of images. <> Collective subroutine. <> ARRAY shall be a co-array of type integer, real, or character. It is an INTENT(IN) argument. RESULT shall have the same type, type parameters, and shape as ARRAY. It is an INTENT(OUT) argument. An element of RESULT has a value equal to the minimum value of all the corresponding elements of ARRAY on the images of the team. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and ARRAY has the values (/1,5,3/) and (/4,1,6/) on the two images, the statement CALL CO_MINVAL(ARRAY,RESULT) gives RESULT the value (/1,1,3/). 13.7.25h CO_PRODUCT(ARRAY,RESULT[,TEAM]) <> Products of elements across a team of images. <> Collective subroutine. <> ARRAY shall be a co-array of type integer, real, or complex. It is an INTENT(IN) argument. RESULT shall have the same type, kind type parameter, and shape as ARRAY. It is an INTENT(OUT) argument. If the size of ARRAY is not zero, an element of RESULT has a value equal to a processor-dependent approximation to the product of all the corresponding elements of ARRAY on the images of the team. If ARRAY has size zero, RESULT has the value one. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and ARRAY has the values (/1,5,3/) and (/4,1,6/) on the two images, the statement CALL CO_PRODUCT(ARRAY,RESULT) gives RESULT the value (/4,5,18/). 13.7.25i CO_SUM(ARRAY,RESULT[,TEAM]) <> Sums of elements across a team of images. <> Collective subroutine. <> ARRAY shall be a co-array of type integer, real, or complex. It is an INTENT(IN) argument. RESULT shall have the same type, kind type parameter, and shape as ARRAY. It is an INTENT(OUT) argument. An element RESULT has a value equal to a processor-dependent approximation to the sum of all the corresponding elements of ARRAY on the images of the team. TEAM (optional) shall be a scalar of type IMAGE_TEAM (13.8.2). It is an INTENT(IN) argument. The case with TEAM absent is treated as if it were present with a team consisting of all images. <> If NUM_IMAGES() has the value 2 and ARRAY has the values (/1,5,3/) and (/4,1,6/) on the two images, the statement CALL CO_SUM(ARRAY,RESULT) gives RESULT the value (/5,6,9/). 13.7.25j FORM_TEAM(TEAM, IMAGES) <> Form a team of images. <> Collective subroutine. <> TEAM shall be a scalar of derived type IMAGE_TEAM, that is defined in the intrinsic module ISO_FORTRAN_ENV (13.8.2). It is an INTENT(IN) argument. IMAGES shall be of rank one and type integer. It is an INTENT(IN) argument. All the elements shall have values in the range 1<=IMAGES(I)<=NUM_IMAGES(); there shall be no repeated values and one element shall have the value THIS_IMAGE(). <> The following splits images into two equal groups and implicitly synchronizes each of the teams: USE, INTRINSIC :: ISO_FORTRAN_ENV INTEGER :: I TYPE(IMAGE_TEAM) :: TEAM IF(THIS_IMAGE()<=NUM_IMAGES()/2) THEN CALL FORM_TEAM(TEAM,[(I,I=1,NUM_IMAGES()/2)]) ELSE CALL FORM_TEAM(TEAM,[(I,I=NUM_IMAGES()/2+1,NUM_IMAGES())]) END IF ...................................................................... [322:23+] After 13.7.51, insert new subsection 13.7.51a IMAGE_INDEX(ARRAY, SUB) <> Return the index of the image corresponding to the set of co-subscripts SUB for ARRAY. <> Inquiry function. <> ARRAY shall be a co-array of any type. SUB shall be a rank-one array of size equal to the co-rank of ARRAY and type integer. <> Scalar of type default integer. <> If the value of SUB is a valid set of co-subscripts for ARRAY, the result is the index of the corresponding image. Otherwise, the result lies outside the range 1, 2, ... NUM_IMAGES(). <> If A is declared A[0:*], IMAGE_INDEX(A,(/0/)) has the value 1. NOTE 13.16a A module containing an implementation of this function with a slightly changed argument list is available in C9a.2 [end NOTE] ...................................................................... [341:30+] After 13.7.88, insert new subsection 13.7.88a NUM_IMAGES() <> Return the number of images. <> Inquiry function. <> None. <> Scalar of type default integer. <> The number of images. <> To employ image 1 to read data and broadcast it to other images: REAL :: P[*] ... IF (THIS_IMAGE()==1) THEN READ(6,*)P DO I = 2, NUM_IMAGES() P[I] = P END DO END IF SYNC_ALL NOTE 13.16b These two functions are useful in connection with NUM_IMAGES. INTEGER FUNCTION LOG2_IMAGES() ! Returns the base-2 logarithm of the number of images, ! truncated to an integer. LOG2_IMAGES = LOG(NUM_IMAGES()+0.5)/LOG(2.0) END FUNCTION LOG2_IMAGES INTEGER FUNCTION REM_IMAGES() ! Returns MOD(NUM_IMAGES(),2**LOG2_IMAGES()). REM_IMAGES = MOD(NUM_IMAGES(),2**LOG2_IMAGES()) END FUNCTION REM_IMAGES [end NOTE] ...................................................................... [356:1+] After 13.7.119, insert new subsection 13.7.119a THIS_IMAGE([ARRAY [,DIM]]) <> Return the index of the invoking image. <> Inquiry function. <> ARRAY (optional) shall be a co-array of any type. DIM (optional) shall be scalar and of type default integer. Its value shall be in the range 1<=DIM<=n where n is the co-rank of ARRAY. <> Type default integer, scalar or of rank one. <> Case (i): If ARRAY is absent, the result is a scalar with value equal to the index of the invoking image. It is in the range 1, ..., NUM_IMAGES(). Case (ii): If ARRAY is present and DIM is absent, the result is an array of size equal to the co-rank of ARRAY. Element K of the result has value equal to co-subscript K of ARRAY on the invoking image. Case (iii): If ARRAY and DIM are present, the result is a scalar with value equal to co-subscript DIM of of ARRAY on the invoking image. <> The following employs the first image to read data. The other images then copy the data. IF (THIS_IMAGE()==1) READ(*,*)P SYNC_ALL P = P[1] NOTE 13.17a A module containing an implementation of this function with a slightly changed argument list is available in C.9a.1. [end NOTE] ................................................... ***Note to Editor: This edit interacts with the edits proposed in paper 05-252. [360:2+] Add new sentence: This module shall define a derived type IMAGE_TEAM with private components. ................................................... [421:25+] In section 16.5.5, events that cause variables to become defined, add after list item (17): (17a) Execution of a SYNC_ALL, SYNC_TEAM, SYNC_IMAGES, SYNC_MEMORY, NOTIFY, or QUERY statement with a STAT= specifier causes the variable specified by the STAT= specifier to become defined. (17b) If an error condition occurs during execution of a SYNC_ALL, SYNC_TEAM, SYNC_IMAGES, SYNC_MEMORY, NOTIFY, or QUERY statement that has an ERRMSG= specifier, the variable specified by the ERRMSG specifier becomes defined. ...................................................................... Add to glossary: [427:1-] <> (2.4.5a) : A data entity declared with a in square brackets. The part of a co-array that resides on another image may be addressed by using co-subscripts in square brackets. <> (2.4.5a) : An object whose designator includes square brackets. <> (2.4.5a) : The bounds given by the data in square brackets in a co-array's declaration or allocation. <> (2.4.5a) : The number of co-subscripts of a co-array. <> (2.4.5a) : The part of a co-array stored in the memory of a specified image may be addressed by using co-subscripts in square brackets. Each co-subscript shall be a scalar integer expression. [427:2+] <> (13.1) : An intrinsic subroutine that has a co- array argument and, on each image of a team, returns a result that depends on data on all the images of the team. [427:18+] <> (9.4.5) : The set of images that are associated with an input-output unit. [430:26+] <> (1.2a) : A Fortran program executes as if it were replicated a number of times, the number of replications remaining fixed during execution of the program. Each copy is called an <> and each image executes asynchronously. <> (1.2a) An integer in the range 1, ..., NUM_IMAGES() that indexes an image. [436:1-] <> (1.2a) A set of images that are collaborating in a computation. <> (8.5.2) Execution of a SYNC_TEAM statement. ...................................................................... [484:9+] After C.9, add new section C.9a Section 13 notes C.9a.1 Module for THIS_IMAGE and IMAGE_INDEX The intrinsics THIS_IMAGE(ARRAY) and IMAGE_INDEX(ARRAY,SUB) cannot be coded in Fortran since ARRAY may be of any type and THIS_IMAGE(ARRAY) needs to know the index of the image on which the code is running. We therefore require the bounds to be specified as integer arrays and we require the image index for THIS_IMAGE(ARRAY). MODULE INDEX CONTAINS INTEGER FUNCTION IMAGE_INDEX(LBOUND,UBOUND,SUB) INTEGER,INTENT(IN) :: LBOUND(:),UBOUND(:),SUB(:) INTEGER :: I,N N = SIZE(SUB) IMAGE_INDEX = SUB(N) - LBOUND(N) DO I = N-1,1,-1 IMAGE_INDEX = IMAGE_INDEX*(UBOUND(I)-LBOUND(I)+1) + & SUB(I) - LBOUND(I) END DO IMAGE_INDEX = IMAGE_INDEX + 1 END FUNCTION IMAGE_INDEX INTEGER FUNCTION THIS_IMAGE(LBOUND,UBOUND,ME) RESULT(SUB) INTEGER,INTENT(IN) :: LBOUND(:),UBOUND(:),ME INTEGER :: SUB(SIZE(LBOUND)) INTEGER :: EXTENT,I,M,ML,N N = SIZE(SUB) M = ME - 1 DO I = 1,N-1 EXTENT = UBOUND(I)-LBOUND(I)+1 ML = M M = M/EXTENT SUB(I) = ML - M*EXTENT + LBOUND(I) END DO SUB(N) = M + LBOUND(N) END FUNCTION THIS_IMAGE END MODULE INDEX C.9a.2 Collective co-array subroutines The subroutines of 13.5.14a return an array of the same shape as the given co-array after having applied an operation across the images involved. A note there shows how to program the operation being applied to all the elements of the co-array to produce a scalar result. Versions with other arguments can similarly be programmed, for example REAL FUNCTION GLOBAL_SUM_MASK(ARRAY,MASK) REAL :: ARRAY(:,:)[*],MASK(:,:) REAL,SAVE :: TEMP[*] TEMP = SUM(ARRAY,MASK=MASK) ! Local sum CALL CO_SUM(TEMP,GLOBAL_SUM) END FUNCTION GLOBAL_SUM FUNCTION GLOBAL_SUM(ARRAY,DIM) REAL :: ARRAY(:,:)[*] INTEGER :: DIM REAL,ALLOCATABLE :: GLOBAL_SUM(:) REAL,ALLOCATABLE :: TEMP(:)[:] ALLOCATE(GLOBAL_SUM(SIZE(ARRAY,3-DIM)) ) ALLOCATE( TEMP(SIZE(ARRAY,3-DIM))[*] ) TEMP = SUM(ARRAY,DIM) ! Local sum CALL CO_SUM(TEMP,GLOBAL_SUM) END FUNCTION GLOBAL_SUM