To: J3 J3/19-251 From: Bill Long, Jon Steidel, Kiran Chandramohan, Zach Jibben Subject: Put with Notify Date: 2019-October-16 Reference: 18-277r1, 18-007r1 I Introduction A capability called "put with notify" is popular in the SHMEM community as a very efficient, narrowly focused, synchronization technique for data transfers between images. The basic idea is to combine a "put" (definition of a variable on a different image) with a notification mechanism that allows the receiving image to know that the data has arrived. While long requested for coarrays in Fortran, there was previously no simple syntax to allow for two things (the put and the notify) on the left side of an assignment. Fortran 2018 introduced specifiers as part of an image selector. The addition of a notify-specifier to an image selector, and a NOTIFY WAIT statement could solve this problem. For example, x[10, NOTIFY=nx] = y where nx is a coarray declared of type(NOTIFY_TYPE). The NOTIFY_TYPE is defined in the intrinsic module ISO_FORTRAN_ENV and has properties similar to type EVENT_TYPE. The effective notify operation is to the nx variable on the same image as specified by the image selector. On image 10, NOTIFY WAIT (nx) [here it is safe to reference x and get the new value] II Use-cases It is common for an image, i, to wait until another image, j, has updated a coarray on image i before image i continues execution. This can currently be implemented using events. For example, ! On some image other than 10 x[10] = y EVENT POST (nx [10]) ! On image 10 EVENT WAIT (nx) z = x It could be implemented more efficiently using a put with notify mechanism: ! On some image other than 10 x[10, NOTIFY=nx] = y ! On image 10 NOTIFY WAIT (nx) z = x The put with notify version differs from the EVENT version in that the notify can be incorporated into the same data packet as the value of y. Also the synchronization is only on this particular transfer, and not all outstanding memory operations on this image. The put with notify operation does not constitute segment ordering, and the NOTIFY WAIT statement is not an image control statement. The notify variable semantics are like those of atomic operations with regard to segment ordering. If x is an array "waiting" for data to be transferred to separate elements from one or more images, the NOTIFY WAIT could have an optional UNTIL_COUNT= clause similar to EVENT WAIT. ! On images other than image 10 x(i)[10, NOTIFY=nx] = y ! On image 10 NOTIFY WAIT (nx, UNTIL_COUNT=size(x)) z(:) = x(:) III Requirements A new type NOTIFY_TYPE, similar to EVENT_TYPE would be defined in the intrinsic module ISO_FORTRAN_ENV. Like an EVENT_TYPE, it would have an integer count component which is atomically incremented. Variables of type NOTIFY_TYPE would be coarrays. Alternative: Consider using type EVENT_TYPE for the type of notify variables, instead of inventing a new type NOTIFY_TYPE. The internal structure of the EVENT_TYPE and NOTIFY_TYPE are the same. Subgroup rejected this, as they felt mixing EVENT POST and NOTIFY= specifiers involving the same EVENT_TYPE variable was undesirable. An optional NOTIFY= specifier would be added to the image selector. This form of image selector is allowed only as part of the designator for the variable of an assignment statement. The effective notify operation is to the nx variable on the same image as specified by the image selector. A notify operation atomically increments the notify variable count on the specified image by one. A notify operation does not constitute segment ordering. A NOTIFY WAIT statement with syntax similar to an EVENT WAIT statement would be added with an optional UNTIL_COUNT= specifier. Successful execution of a NOTIFY WAIT statement decrements the count of the notify variable by one, or by the specified UNTIL_COUNT= count, if specified. A NOTIFY WAIT statement would have optional STAT= and ERRMSG= specifiers with the semantics of those on an EVENT WAIT statement. A NOTIFY WAIT statement is not an image control statement. IV Specification An image selector can use a NOTIFY specifier to notify another image that it can proceed to work on tasks that use common resources. An image can wait for notification from one or more images that modify a coarray on the waiting image before it proceeds. NOTIFY_TYPE is a derived type with private components. It is an extens- ible type with no type parameters. All components have default initial- ization. The type is defined in the ISO_FORTRAN_ENV intrinsic module. A scalar variable of type NOTIFY_TYPE is notify variable. A notify variable includes a count of the difference between the number of successful notifies and successful waits for the notify variable. The initial value of the count of a notify variable is zero. It is essentially the same as an EVENT variable, but can be modified only by a NOTIFY= specifier in an image selector or by a NOTIFY WAIT statement. A must be a coarray, but it shall not be coindexed. An optional NOTIFY= specifier is added to an image selector. The NOTIFY= specifier is permitted only in the designator of the variable of an assignment statement. Successful execution of an assignment statement with a NOTIFY= specifier atomically increments the count of the specified on the image specified by the image selector. The value of the is processor dependent if the execution of an assignment statement with a NOTIFY= specifier is unsuccessful. A new execution control statement, NOTIFY WAIT, is introduced. NOTIFY WAIT ( [, ]) The is the same as for an EVENT WAIT statement, containing an optional and an optional . A threshold value is computed as for an EVENT WAIT statement. The executing image shall wait until the count equals or exceeds the threshold value. A successful wait for a notify variable decrements the count of by the threshold value. If execution of the NOTIFY WAIT statement is unsuccessful, the value of is processor dependent. If the STAT= specifier appears in and the exec- ution of the statement is successful, is assigned the value zero. If the STAT= specifier appears in and an error condition occurs, is assigned a processor-dependent positive value that is different from the value STAT_FAILED_IMAGE and STAT_STOPPED_IMAGE from the intrinsic module ISO_FORTRAN_ENV. If an ERRMSG= specifier appears and an error condition occurs, is assigned an explanatory message, truncated or padded according to the rules of intrinsic assignment. If no such condition occurs, the definition status and value of is unchanged. The set of error conditions that can occur during execution of a NOTIFY WAIT statement is processor-dependent. The NOTIFY WAIT statement is not an image control statement. A shall not appear in a variable definition context except as the in an NOTIFY= specifier of an image selector or in a NOTIFY WAIT statement, as an in an ALLOCATE statement without a SOURCE= , or as an actual argument in a reference to a procedure with an explicit interface where the corresponding dummy argument has INTENT(INOUT). A variable with a subobject of type NOTIFY_TYPE shall not appear in a variable definition context except as an in an ALLOCATE statement without a SOURCE= or as an actual argument in a reference to a procedure with an explicit interface where the corresponding dummy argument has INTENT(INOUT). A notify variable may be referenced or defined during the execution of a segment that is unordered relative to the execution of another segment in which the notify variable is defined. A coarray defined by an assignment statement with a NOTIFY=specifier cannot be referenced until after the successful execution of a corresponding NOTIFY WAIT statement whose is the same as the specified in the NOTIFY= specifier, unless the coarray definition and the coarray reference are in ordered segments.