To: J3 J3/13-336r1 From: Bill Long & John Reid & Lorri Menard Subject: Comments on CHANGE TEAM, image selectors, FORM SUBTEAM Date: 2013 October 17 References: N1983, N1989, 13-300, 13-333 Discussion - Responses with edits --------------------------------- {Bill I14}: At [28:25,28,38] The new text in the edits for 8.5.7 includes the case of a STAT= specifier appearing on an END TEAM statement, but there is no syntax for that. Note that the text at [28:23-24] contains the "as an appearance both there and in the corresponding END TEAM". This text has various ambiguities and needs to be fixed, or specific syntax added for END TEAM. Response: It is confusing to have an implicit on the END TEAM statement. An edit is provided. {Van 2}: At [10:13-14] Values of variables are not "formed," they are defined. Response: Partly agree. Values are not defined; variables are defined. An edit with revised wording is provided. {John 1.1e}: The CHANGE TEAM statement should work in the presence of failed images. Response: Agreed. This is indirectly implied in [9:31] "executing image", which automatically excludes failed images. In [10:20-26] instances of "all images" need to be qualified. Five edits are provided. {Reinhold, A.2.2}: Need to specify that the coarray being referenced existed when the specified team was in effect. Response: Agree. Need sentence in para [10:30-32]. {Reinhold, A.2.3}: Need to specify what set of cobounds and corank apply to the reference. Response: the ones active at the time the specified team split into the team that the current team is descended from. An edit is provided. {Bill I3}: At [10:29] The new syntax for image selectors needs illustrative examples. A Note at the end of 5.4 [10:32+] would be good; at least an example in the Annex. Response: Agreed. An edit is provided. {John 1.2}: New syntax (R624) was added during the Delft meeting to allow a coarray to be addressed by the cosubscipts of a team other than the current team. It is not restricted to be the current team or an ancestor, but I think that was the intention. Because there is no means of specifying the mapping between cosubscripts when teams change, the new syntax should be restricted to refer to the team in which the coarray was declared. Alternatively, a mechanism for specifying the mapping should be added. I suggest that it should be as for the association of a dummy coarray with the corresponding actual coarray. A possibility is the statement new cosubscripts () where is [] Response: Partial overlap with {Reinhold A.2.2}. More that where a coarray is declared, the relevant question is where it is given its bounds, particularly for allocatable coarrays. This is preferred over the alternative mechanism suggested. Examples would be very helpful in guiding the discussion of this topic. Edits needed. {Steve, 1}: In R507 should be . {Van 4}: [Same as {Steve 1}.] Response: While it is true that "team" became "subteam" for N1983, paper 13-337 reverses that change, and so this suggested edit is no longer necessary {Bill I4}: At [11:1] The new syntax to allow a team member to choose an image index value needs illustrative examples. A Note at the end of 5.5 [11:17+] would be good; at least an example in the Annex. Response: Agreed. An edit is provided. {Nick 4}: 5.5 p11:4-16 (especially 4, 11-12 and 14). This is seriously inconsistent. FORM SUBTEAM cannot be an image control statement if it occurs within a segment, ordered or not (Fortran 2008 8.5.2) and, if it does, it has explicit synchronization. I think that the sentence on line 4 "It is an image control statement." is erroneous and should be removed. Response: [11:14] says there is a synchronization. An edit is provided to correct the statement about segments. {Van 7}: At [11:9] "a different value" -> "different values". Response: OK. Prefer "Each image" at the beginning of the sentence. An edit is provided. {John 2.1}: At [11:7-10] What happens if NEW_INDEX is absent? Is the mapping from parent image index to child image index processor dependent? Or is THIS_IMAGE(DISTANCE=1) monotonic increasing? Response: The simplest option is to leave it unspecified which allows the processor to assign image index values based on the characteristics of the underlying hardware. An edit is provided for this option. {Nick 5}: 5.5 p11:12-13. I have no idea why the restriction on the subscripts is needed, and cannot see that it does anything useful. A single dummy argument can be associated with different variables on different images and, if it has value semantics, there is no reason to restrict it in this way. Indeed, different images can even pass the same variable in different elements of the same dummy array on different images. Response: The intention is to avoid the confusing situation of having different array elements on different images. However, there is not an actual problem here. An edit is provided to delete the sentence. {Van 8}: At [11:13] "on" -> "in" or "these statements" -> "images of the current team". Response: The sentence is deleted by the {Nick 5} edit. {John 1.1d}: The FORM SUBTEAM statement should work in the presence of failed images. I am inclined to think that if a subteam has a nonfailed image, all its images should be nonfailed. Response: It is necessary that FORM SUBTEAM ignore failed images of the current team, to allow formation of a new "clean" team. Edits are provided. {Nick 6}: 5.5 p11:14-16. FORM SUBTEAM does not say what happens if an error occurs, unlike the collectives. Response: Partly addressed by {John 1.1d} above. For other failures, the team variable is undefined. An edit is provided for the other failure case. {Van 9}: At [11:17] A prohibition against changing the value of a team variable during execution of a CHANGE TEAM construct appears to be necessary. Otherwise, how does synchronization at the END TEAM statement work reliably? Does the processor keep a cached copy of the team variable, maybe on something like a "team stack," just in case it's changed? The restriction at [11:17] doesn't cover this problem. Maybe it's enough to insert "or the current team". Response: This is less simple than it appears. A team variable that should not be changed is only the one specified in any currently executing CHANGE TEAM construct. An edit is provided to state this limitation. {Nick 7}: 5.6 [5.5] p11:17+. There is nothing said about when resources may be released, and no mechanism for the user to free them. This is not reasonable, and there needs to be some defined way for a programmer to avoid memory leaks when using FORM SUBTEAM heavily. Response: We allow ALLOCATE but not DEALLOCATE. This irregularity is the root of the problem noted here. Edits are provided to allow deallocation. {Van 10}: At [11:21] "FORM SUBTEAM" -> "a FORM SUBTEAM statement". Response: Agreed. The edit is included below, also changing SUBTEAM to TEAM. (13-337r1). Discussion - Responses only --------------------------- {Nick 3}: 5.3, p10:22-24. Executing a common CHANGE TEAM statement the same number of times is not enough, because the variable could be a dummy argument associated with a different team on different images. There needs to be an explicit restriction (probably in lines 14-16) that all variables must have been created by the same execution of the same FORM SUBTEAM statement. See [A]. Response: This restriction is given by the text in [10:13-14]. {Reinhold, A.2.1}: Can image selector specify an image outside the current team? Response: Yes. {Malcolm Reason 1a}: Cross-team access and synchronization has made TEAMs too complicated. Response: For additional comments on synchronization, see 5.6. While the initial proposal for teams, prohibiting cross-team references, was adequate for library writers, it lacked significant functionality required by important real-world application programs. The addition of cross-team references was the compromise to make this useful to that audience. Perhaps clarifying examples would help make the feature less unwieldy. The cross-team reference facility was not invented "out of whole cloth". It was based on the facility in the Rice CAF compiler. {Van 3}: Image selectors don't solve the problem that motivated the discussion that led to them. In any case, they seem to be a fundamentally bad idea, somewhat akin to the idea that a subroutine can access a sequence-associated dummy argument array using subscripts relative to the actual argument. A proposal to solve the original problem is attached [to the original ballot]. Response: The problem that is being addressed by image selectors is different and was the most important point that was made by Reinhold Bader in his ballot (see N1971). Reinhold was concerned about the synchronization overheads involved in frequent changes of team. Such changes will be needed infrequently if there is a mechanism for accessing data outside the current team. Infrequent changes of team will also mean that coarrays allocated during the execution of a CHANGE TEAM construct have a long life, thereby avoiding the overheads of archiving and restoring data. The data remapping problem that Van Snyder is addressing may be avoided by making more use of allocated coarrays and by calling a procedure during execution of execution of a CHANGE TEAM construct. Note that the Rice CAF has had a similar mechanism for some time (though the syntax works for only one codimension). {Reinhold, A.1}: Request to add DOMAINS. Response: This proposal is the topic of paper 13-300. {Reinhold, A.3}: Remove image control statement and synchronization requirements for FORM SUBTEAM. Response: FORM SUBTEAM is a collective activity The executing image needs to know which images are its mates before it can complete the work. Since you cannot predict which team the "last" executer will select, the synchronization is needed. {Van 5}: One could calculate a team variable describing the initial team by executing a FORM SUBTEAM statement in which has the same value on all images. If this would be a common thing to do, it would be useful to have a team variable describing the initial team, maybe in ISO_FORTRAN_ENV. Response: An example showing the formation of a team for all images is included as part of Note 5.1a. This is so simple that adding an extra feature seems unnecessary. A minor problem with the proposal is that it causes a team variable to be defined by some mechanism other that by executing FORM SUBTEAM. Note the alternative THIS_TEAM (see paper 13-333) from Tobias. That intrinsic could be used for this purpose, along with purposes that provide stronger justification for its inclusion. {Van 6}: At [11:6] Restricting to be positive seems pointless. Can it at least be non-negative? One might reasonably want to use mod(this_image(),2), instead of 1+mod(this_image(),2) or 2-mod(this_image(),2). Response: This would preclude a future use of 0 for "not joining any team". {Nick 8}: Note 5.2 and the example at 7.4.13 p22:16-18. I can find no guarantee that the subteam id is assigned in a defined order, and hope that is not the case. The example comments should say "Code for half of the images in the current team" and "Code for the other half of the images in the current team". Response: The subteam id values are the ones specified by the user for each image. There does not appear to be a ordering issue involved. Edits to N1983 -------------- [9:24] {Nick 7} Before "or as a" add "as an in a DEALLOCATE statement," [9:29+] {Nick 7} Add paragraph: "The team variable for the current team or an ancestor team shall not be deallocated." [9:37] {Bill I14} Add at the end of R503: "[,]". [10:13-14] {Van 2} Replace the first two sentences in the paragraph following C507 by "The shall have been defined by execution of a FORM TEAM statement in the team that executes the ." [10:16+] {Ed, Van 9} Insert a new paragraph: "The team variable specified in a CHANGE TEAM statement shall not be redefined or become undefined during execution of that CHANGE TEAM construct." [10:21-22] {John 1.1e} Change "all images of the team ... of this team." to "all nonfailed images of the team containing the executing image that is identified by ." [10:22] {John 1.1e} Change "on each image" to "on each nonfailed image". [10:23] {John 1.1e} Change "other images" to "other nonfailed images". [10:25] {John 1.1e} Change "images in its team" to "nonfailed images in the current team". [10:26] {John 1.1e} Change "images" to "nonfailed images". [10:26+2] {Van m11} In Note 5.1 change "The deallocation" to "Deallocation" [10:26+3] {Van m12} In Note 5.1 change "was allocated at the end of the construct" to "is allocated at the end of execution of the construct". [10:32] {Reinhold A.2.3, John 1.2} After "." add "If the team distance between the teams is , the statement shall lie within nested CHANGE TEAM constructs." [10:32+] {Reinhold A.2.2} At end of paragraph add "If is specified, the coindexed object referenced by the image selector shall have been established by the team that formed or one of its ancestors." [10:32+] {Bill I3} Add NOTE 5.1a In the following code, the vector of length N*P is distributed over P images. Each has an array A(0,N+1) holding its own values of and halo values from its two neighbors. The images are divided into two teams that execute independently but periodically exchange halo data. Before the data exchange, all the images (of the initial team) must be synchronized and for the data exchange the coindices of the initial team are needed. USE, INTRINSIC :: ISO_FORTRAN_ENV TYPE(TEAM_TYPE) :: INITIAL, BLOCK REAL :: A(0:N+1)[*] INTEGER :: ME, P2 CALL GET_TEAM(INITIAL) ME = THIS_IMAGE() P2 = NUM_IMAGES()/2 FORM TEAM(1+(ME-1)/P2,BLOCK) CHANGE TEAM(BLOCK) DO ! Iterate : ! Halo exchange SYNC TEAM(INITIAL) IF(ME=P2) A(N+1) = A(1)[INITIAL::ME+1] IF(ME=P2+1) A(0) = A(N)[INITIAL::ME-1] END DO END CHANGE TEAM [11:3] {Van m13} In C508 delete "given". [11:9] {Van 7} Replace "Images with the same value" by "Each image with the same value". [11:10] {Van m14} At the of para add a new sentence "If the NEW_INDEX= specifier does not appear, the image index that the executing image will have in the new team specified by is assigned by the processor. [11:11-12] {John 1.1d} change "all images of the current team" to "all nonfailed images of the current team". [11:12] {Nick 4} Delete ", in execution segments ... each other". [11:12-13] {Nick 5} Delete the sentence "If contains ... all these statements." [11:14] {John 1.1d} Change "images" to "nonfailed images". [11:15] {John 1.1d} Change "images" to "nonfailed images". [11:15] {Nick 6} At the end of the paragraph add a new sentence "If an error condition other than detection of a failed image occurs the team variable becomes undefined." [11:17] {Ed, Van 9} Delete the paragraph/sentence at [11:17] {The edit at [10:16+] makes this paragraph unnecessary.} [11:17+] Bill I4} Add NOTE 5.2a When executing on P^2 images with coarrays regarded as spread over a P by P square, the following code establishes teams for the rows with image indices equal to the column indices. USE, INTRINSIC :: ISO_FORTRAN_ENV TYPE(TEAM_TYPE) :: ROW REAL :: A[P,*] INTEGER :: ME(2) ME(:) = THIS_IMAGE(A) FORM TEAM(ME(1),ROW,NEW_INDEX=ME(2)) [11:21] {Van m18} In "by an execution" delete the "an". [11:21] {Van 10} "FORM SUBTEAM" -> "a FORM TEAM statement". [31:29+] {John 2.1} At the end of 8.9: In line 29 replace "." by ";" and add a new dependency "the image index value assigned by a FORM TEAM statement without a NEW_INDEX= specifier."