To: J3 J3/13-336 From: Bill Long, John Reid Subject: Comments on CHANGE TEAM, image selectors, FORM SUBTEAM Date: 2013 September 30 References: N1983, N1989, 13-300, 13-333 Overview -------- This paper contains a portion of the responses to the comments in N1989, the results of the ballot on the TS 18508 draft N1983. The following identifiers are used to indicate the source of comments: Reinhold : Reinhold Bader Bill : Bill Long Tobias : Tobias Burnus Nick : Nick Maclaren Daniel : Daniel Chen Dan : Dan Nagel Malcolm : Malcolm Cohen John : John Reid Steve : Steve Lionel Van : Van Snyder along with {Ed} for the document editor. Discussion - Responses with edits --------------------------------- {Bill I14}: At [28:25,28,38] The new text in the edits for 8.5.7 includes the case of a STAT= specifier appearing on an END TEAM statement, but there is no syntax for that. Note that the text at [28:23-24] contains the "as an appearance both there and in the corresponding END TEAM". This text has various ambiguities and needs to be fixed, or specific syntax added for END TEAM. Response: It is confusing to have an implicit on the END TEAM statement. An edit is provided. {Van 2}: At [10:13-14] Values of variables are not "formed," they are defined. Response: Partly agree. Values are not defined; variables are defined. An edit with revised wording is provided. {John 1.1e}: The CHANGE TEAM statement should work in the presence of failed images. Response: Agreed. This is indirectly implied in [9:31] "executing image", which automatically excludes failed images. In [10:20-26] instances of "all images" need to be qualified. Five edits are provided. {Reinhold, A.2.2}: Need to specify that the coarray being referenced existed when the specified team was in effect. Response: Agree. Need sentence in para [10:30-32]. {Reinhold, A.2.3}: Need to specify what set of cobounds and corank apply to the reference. Response: the ones active at the time the specified team split into the team that the current team is descended from. An edit is provided. {Bill I3}: At [10:29] The new syntax for image selectors needs illustrative examples. A Note at the end of 5.4 [10:32+] would be good; at least an example in the Annex. Response: Agreed. An edit is provided. {John 1.2}: New syntax (R624) was added during the Delft meeting to allow a coarray to be addressed by the cosubscipts of a team other than the current team. It is not restricted to be the current team or an ancestor, but I think that was the intention. Because there is no means of specifying the mapping between cosubscripts when teams change, the new syntax should be restricted to refer to the team in which the coarray was declared. Alternatively, a mechanism for specifying the mapping should be added. I suggest that it should be as for the association of a dummy coarray with the corresponding actual coarray. A possibility is the statement new cosubscripts () where is [] Response: Partial overlap with {Reinhold A.2.2}. More that where a coarray is declared, the relevant question is where it is given its bounds, particularly for allocatable coarrays. This is preferred over the alternative mechanism suggested. Examples would be very helpful in guiding the discussion of this topic. Edits needed. {Steve, 1}: In R507 should be . Response: Agreed - this is a typo. {Van 4}: [Same as {Steve 1}.] {Bill I4}: At [11:1] The new syntax to allow a team member to choose an image index value needs illustrative examples. A Note at the end of 5.5 [11:17+] would be good; at least an example in the Annex. Response: Agreed. An edit is provided. {Nick 4}: 5.5 p11:4-16 (especially 4, 11-12 and 14). This is seriously inconsistent. FORM SUBTEAM cannot be an image control statement if it occurs within a segment, ordered or not (Fortran 2008 8.5.2) and, if it does, it has explicit synchronization. I think that the sentence on line 4 "It is an image control statement." is erroneous and should be removed. Response: [11:14] says there is a synchronization. An edit is provided to correct the statement about segments. {Van 7}: At [11:9] "a different value" -> "different values". Response: OK. Prefer "Each image" at the beginning of the sentence. An edit is provided. {John 2.1}: At [11:7-10] What happens if NEW_INDEX is absent? Is the mapping from parent image index to child image index processor dependent? Or is THIS_IMAGE(DISTANCE=1) monotonic increasing? Response: Subgroup discussion. The simplest option is to leave it unspecified which allows the processor to assign image index values based on the characteristics of the underlying hardware. An edit is provided for this option. {Nick 5}: 5.5 p11:12-13. I have no idea why the restriction on the subscripts is needed, and cannot see that it does anything useful. A single dummy argument can be associated with different variables on different images and, if it has value semantics, there is no reason to restrict it in this way. Indeed, different images can even pass the same variable in different elements of the same dummy array on different images. See [A]. Response: Subgroup discussion needed. The intention is to avoid the confusing situation of having different array elements on different images. Edits possibly needed. {Van 8}: At [11:13] "on" -> "in" or "these statements" -> "images of the current team". Response: Implement the second suggestion. {John 1.1d}: The FORM SUBTEAM statement should work in the presence of failed images. I am inclined to think that if a subteam has a nonfailed image, all its images should be nonfailed. Response: It is necessary that FORM SUBTEAM ignore failed images of the current team, to allow formation of a new "clean" team. Edits are provided. {Nick 6}: 5.5 p11:14-16. FORM SUBTEAM does not say what happens if an error occurs, unlike the collectives. Response: Partly addressed by {John 1.1d} above. For other failures, the team variable is undefined. An edit is provided for the other failure case. {Van 9}: At [11:17] A prohibition against changing the value of a team variable during execution of a CHANGE TEAM construct appears to be necessary. Otherwise, how does synchronization at the END TEAM statement work reliably? Does the processor keep a cached copy of the team variable, maybe on something like a "team stack," just in case it's changed? The restriction at [11:17] doesn't cover this problem. Maybe it's enough to insert "or the current team". Response: This is less simple than it appears. At a minimum, a team variable that should not be changed is only the one specified in any currently executing CHANGE TEAM construct. And even that might be too restrictive if we want to allow redefinition of the team variable describing the current team, followed by exiting the construct, in the event of failed images. Since the failed images could not participate in the END TEAM statement anyway, there is not a problem there. If you do not allow reuse of the same name, the syntax gets complicated. Separately, the current sentence at [11:17] is confusing. The reference to "The team variable" is to the one that appears in the FORM SUBTEAM statement. That is defined by that statement, so preventing a bad value is the task of the processor, not the programmer. If it were changed to "A team variable" there is still a problem, since this would prevent forming a single team consisting of all of the images of the current team. Subgroup discussion. But some edit appears necessary. {Nick 7}: 5.6 [5.5] p11:17+. There is nothing said about when resources may be released, and no mechanism for the user to free them. This is not reasonable, and there needs to be some defined way for a programmer to avoid memory leaks when using FORM SUBTEAM heavily. Response: We allow ALLOCATE but not DEALLOCATE. This irregularity is the root of the problem noted here. Edits are provided to allow deallocation. Discussion - Responses only --------------------------- {Nick 3}: 5.3, p10:22-24. Executing a common CHANGE TEAM statement the same number of times is not enough, because the variable could be a dummy argument associated with a different team on different images. There needs to be an explicit restriction (probably in lines 14-16) that all variables must have been created by the same execution of the same FORM SUBTEAM statement. See [A]. Response: This restriction is given by the text in [10:13-14]. {Reinhold, A.2.1}: Can image selector specify an image outside the current team? Response: Yes. {Malcolm Reason 1a}: Cross-team access and synchronization has made TEAMs too complicated. Response: For additional comments on synchronization, see 5.6. While the initial proposal for teams, prohibiting cross-team references, was adequate for library writers, it lacked significant functionality required by important real-world application programs. The addition of cross-team references was the compromise to make this useful to that audience. Perhaps clarifying examples would help make the feature less unwieldy. The cross-team reference facility was not invented "out of whole cloth". It was based on the long-existing facility in the Rice CAF compiler. {Van 3}: Image selectors don't solve the problem that motivated the discussion that led to them. In any case, they seem to be a fundamentally bad idea, somewhat akin to the idea that a subroutine can access a sequence-associated dummy argument array using subscripts relative to the actual argument. A proposal to solve the original problem is attached [to the original ballot]. Response: The problem that is being addressed by image selectors is different and was the most important point that was made by Reinhold Bader in his ballot (see N1971). Reinhold was concerned about the synchronization overheads involved in frequent changes of team. Such changes will be needed infrequently if there is a mechanism for accessing data outside the current team. Infrequent changes of team will also mean that coarrays allocated during the execution of a CHANGE TEAM construct have a long life, thereby avoiding the overheads of archiving and restoring data. The data remapping problem that Van Snyder is addressing may be avoided by making more use of allocated coarrays and by calling a procedure during execution of execution of a CHANGE TEAM construct. Note that the Rice CAF has had a similar mechanism for some time (though the syntax works for only one codimension). {Reinhold, A.1}: Request to add DOMAINS. Response: This proposal is the topic of paper 13-300. {Reinhold, A.3}: Remove image control statement and synchronization requirements for FORM SUBTEAM. Response: FORM SUBTEAM is a collective activity The executing image needs to know which images are its mates before it can complete the work. Since you cannot predict which team the "last" executer will select, the synchronization is needed. {Van 5}: One could calculate a team variable describing the initial team by executing a FORM SUBTEAM statement in which has the same value on all images. If this would be a common thing to do, it would be useful to have a team variable describing the initial team, maybe in ISO_FORTRAN_ENV. Response: An example showing the formation of a team for all images is included as part of Note 5.1a. This is so simple that adding an extra feature seems unnecessary. A minor problem with the proposal is that it causes a team variable to be defined by some mechanism other that by executing FORM SUBTEAM. Note the alternative THIS_TEAM (see paper 13-333) from Tobias. That intrinsic could be used for this purpose, along with purposes that provide stronger justification for its inclusion. {Van 6}: At [11:6] Restricting to be positive seems pointless. Can it at least be non-negative? One might reasonably want to use mod(this_image(),2), instead of 1+mod(this_image(),2) or 2-mod(this_image(),2). Response: This would preclude a future use of 0 for "not joining any team". {Nick 8}: Note 5.2 and the example at 7.4.13 p22:16-18. I can find no guarantee that the subteam id. is assigned in a defined order, and hope that is not the case. The example comments should say "Code for half of the images in the current team" and "Code for the other half of the images in the current team". Response: The subteam id values are the ones specified by the user for each image. There does not appear to be a ordering issue involved. Edits to N1983 -------------- [9:24] {Nick 7} Before "or as a" add "as an in a DEALLOCATE statement," [9:29+] {Nick 7} Add paragraph: "The team variable for the current team or an ancestor team shall not be deallocated." [9:37] {Bill I14} Add at the end of R503: "[,]". [10-13-14] {Van 2} Replace the first two sentences in the paragraph following C507 by "The shall have been defined by execution of a FORM SUBTEAM statement in the team that executes the ." [10:21-22] {John 1.1e} Change "all images of the team ... of this team." to "all nonfailed images of the team containing the executing image that is identified by ." [10:22] {John 1.1e} Change "on each image" to "on each nonfailed image". [10:23] {John 1.1e} Change "other images" to "other nonfailed images". [10:25] {John 1.1e} Change "images in its team" to "nonfailed images in the current team". [10:26] {John 1.1e} Change "images" to "nonfailed images". [10:30-32] [Missing edit for {Reinhold, A.2.2}.] [10:32] {Reinhold A.2.3} After "." add "If the team distance between the teams is , the statement shall lie within nested CHANGE TEAM constructs." [10:32] [Missing edit for {John 1.2}.] [10:32+] {Bill I3} Add NOTE 5.1a In the following code, the vector of length N*P is distributed over P images. Each has an array A(0,N+1) holding its own values of and halo values from its two neighbors. The images are divided into two teams that execute independently but periodically exchange halo data. Before the data exchange, all the images (of the initial team) must be synchronized and for the data exchange the coindices of the initial team are needed. USE, INTRINSIC :: ISO_FORTRAN_ENV TYPE(TEAM_TYPE) :: INITIAL, BLOCK REAL :: A(0:N+1)[*] INTEGER :: ME, P2 FORM SUBTEAM(1,INITIAL) ME = THIS_IMAGE() P2 = NUM_IMAGES()/2 FORM SUBTEAM(1+(ME-1)/P2,BLOCK) CHANGE TEAM(BLOCK) DO ! Iterate : ! Halo exchange SYNC TEAM(INITIAL) IF(ME=P2) A(N+1) = A(1)[INITIAL::ME+1] IF(ME=P2+1) A(0) = A(N)[INITIAL::ME-1] END DO END CHANGE TEAM [11:1] {Steve 1, Van 4} In rule R507 replace "" by "". [11:9] {Van 7} Replace "Images with the same value" by "Each image with the same value". [11:12] {Nick 4} Delete ", in execution segments ... each other". [11:12-13] [Missing edit for {Nick 5}.] [11:13] {Van 8} At the end of the sentence, replace "these statements" by "images of the current team". [11:12-13] {John 1.1d} change "all images of the current team" to "all nonfailed images of the current team". [11:14] {John 1.1d} Change "images" to "nonfailed images". [11:15] {John 1.1d} Change "images" to "nonfailed images". [11:15] {Nick 6} At the end of the paragraph add a new sentence "If an error condition other than detection of a failed image occurs the team variable becomes undefined." [11:17] [Missing edit for {Van 9}.] [11:17+] Bill I4} Add NOTE 5.2a When executing on P^2 images with coarrays regarded as spread over a P by P square, the following code establishes teams for the rows with image indices equal to the column indices. USE, INTRINSIC :: ISO_FORTRAN_ENV TYPE(TEAM_TYPE) :: ROW REAL :: A[P,*] INTEGER :: ME(2) ME(:) = THIS_IMAGE(A) FORM SUBTEAM(ME(1),ROW,NEW_INDEX=ME(2)) [31:29+] {John 2.1} At the end of 8.9: In line 29 replace "." by ";" and add a new dependency "the image index value assigned by a FORM SUBTEAM statement without a NEW_INDEX= specifier."